# Deep suite: the plan and every change to it, as logged

Each entry was written before the step it describes ran. Notes about setup failures say what failed and what was done; those runs are kept and never counted.

## HM1, hardmix: hard public tasks in our own harness, SWE-bench Pro pilot (logged 2026-10-13, before any agent runs it)
- Reason: H1 and R1 were too easy (every model scored full marks on both). Bren asked for a harder test covering several kinds of work: building features, fixing bugs, security and performance. Plan agreed with Bren: hard subsets of public benchmarks that frontier models have not saturated, run in this bench's own harness with the same CLIs and settings as the suite. SWE-bench Pro's hard subset (HARD-51) for features and bugs comes first; GSO for performance (graded by measured speed-up against the expert's commit) and PatchBench for security (graded by the crash input, replayed crash and benign inputs, unit tests and output checks) follow, each preregistered here before it runs. The current suite is unchanged.
- Tasks (tasks.json; task folders copied unchanged from github.com/scaleapi/SWE-bench_Pro-os v2/tasks at commit 66f92766): ansible (build a feature, Python), flipt-524f2773 (build a feature, Go), protonmail webclients (fix a bug, TypeScript), navidrome (fix a bug, Go). Picked from HARD-51 with seed 7001, one feature and one bug fix each in a different language. Swap, made before any run: the first Go feature drawn (flipt-c1728053) needs the cuelang.org/go library, which its image does not have, and runs have no internet; it was replaced by the only HARD-51 Go task labeled core_feat whose fix adds no library.
- Agent run (hardmix.py): the task's own image with the repository at /app; Claude Code 2.1.284 (the musl build of the same release on the Alpine image) and Codex CLI 0.159.0 mounted read-only. The container's only network is an internal one whose single exit is proxy.py, which opens tunnels only to the vendors' model and login hosts (api.anthropic.com, claude.ai, platform.claude.com, console.anthropic.com, chatgpt.com, auth.openai.com, api.openai.com) and logs every refusal. Checked before any run: no DNS, no direct connection, GitHub and PyPI refused through the proxy. Claude Code: WebFetch and WebSearch denied, no MCP servers. Codex: web search off, apps and plugins off. The prompt is the task's instruction.md unchanged. The run never sees the tests or the reference fix. Effort medium for all four models; 1800 s cap (SWE-bench Pro allows 3000); 4 CPUs and 6 GB.
- Grading: the run's diff (every change against the base commit, files present before the run excluded) is applied to a fresh container of the same image, then the task's own tests/test.sh runs: it resets every test file to the base, adds the hidden tests and scores 1 only if every listed fail-to-pass and pass-to-pass test passes, else 0. Editing tests cannot change the score.
- Key checks, done: on all four tasks the reference fix scores 1 and the untouched repository 0; each image's git history ends at the base commit (no later commits to read the fix from) with a clean tree.
- Pilot, not counted: 1 run of each task by each of Claude Opus 5.5, Claude Haiku 5.5, GPT-6.1 Sol and GPT-6 Luna (16 runs), one stream per model with its own seeded task order (seeds 7101 to 7104), the four streams side by side, into runs-pilot. Reported per run: score, minutes, tokens, whether it hit the cap. If all 16 score 1, this slice is not hard enough and the selection changes before anything is counted. Counted rounds use 3 runs per task and model (the reviewers' condition).
- Noted during the HM1 pilot, before any GSO or PatchBench run: on ansible, the two failing runs so far lost the three manager tests because their ModuleManager has no get_manager(), which the hidden tests stub. The task's text describes the dispatcher but does not name that method; 22 of the repository's 135 F5 modules use it. The benchmark's own score stands; the report will flag the task as borderline, and whether it stays in counted rounds goes to Bren and the reviewers.


## HM2, hardmix: performance (GSO) and security (PatchBench) tasks, pilot (logged 2026-10-13, before any agent runs them)
- Same harness, CLIs, settings, models, network lock and 1800 s cap as HM1. Grading is each benchmark's own code.
- Performance, GSO (huggingface.co/datasets/gso-bench/gso; task folders generated by Harbor's GSO adapter, github.com/harbor-framework/harbor adapters/gso, which reuses GSO's own grading): two tasks drawn with seed 7002 from the 17 whose expert fix changes Python files only (so a rebuild fits a 30-minute run), excluding the two repositories GSO marks high-resource: pandas BaseMaskedArray._validate_setitem_value (pandas-71c94af) and numpy np.isin (numpy-cedba62). The prompt is the adapter's instruction (the task's performance script, "make it faster", the rebuild commands). The agent image adds a filled uv package cache so the agent can rebuild with no internet; the grading image is the adapter's own. GSO images keep the expert commit as an unreachable git object; the agent's copy drops it before the run, and each run records that it is gone. Grading (GSO's eval script, 5 timed runs of every performance test before and after, then the expert commit): score 1 if the patched build's outputs match and its speed-up over the base is at least 1.1x and at least 95% of the expert commit's (harmonic mean over tests), else 0. Grading runs alone on 8 pinned cores. GSO allows agents 3 hours; this pilot gives 30 minutes, so a 0 here can mean out of time, and the report says which.
- Security, PatchBench (github.com/ai-sec-lab/PatchBench at b991b295; images b4drequest/vulpatch:{id}-vul): two tasks, the first two from different projects in seed-7003 order: wolfssl 29185 (CWE-122 heap overflow) and gpac 44766 (CWE-416 use after free). The prompt is PatchBench's own (the sanitizer report, the crash input at /tmp/poc, vulpatch compile and vulpatch run) with its no-browsing rule. As in PatchBench's harnesses, the agent's copy of the repository starts from a fresh git history, and only C/C++ source changes count. Grading is PatchBench's own four stages: the crash input no longer crashes (poc), no input in the task's crashing or benign fuzz corpus triggers a sanitizer error (replay), every unit test that passes with the reference fix still passes (unittest), and output on benign inputs matches the reference fix byte for byte (verify). Score 1 only if all four pass. The two tasks' corpus files were read out of PatchBench's published archives by byte range.
- Key checks before any agent run: for each task the reference fix scores 1 and an empty change 0, under the same grading.
- Pilot, not counted: 1 run of each of the four tasks by each of the four models (16 runs), seeds 7111 to 7114, into runs-pilot-2. Performance tasks run and grade one at a time, with nothing else heavy on the machine.
- Harness failure, logged 2026-10-13: the first start of the four wolfssl runs never reached the agent. On the task's Ubuntu 16.04 image, bash 4.3 stops on an empty list under set -u, so the setup script exited before the CLI was called (no agent output). Grading was stopped, the four folders moved to runs-harness-failures (not model results, never counted), the script fixed and checked with a stand-in agent on that image, and the four runs started again. hardmix.py now records any run whose agent never started as a harness failure and does not grade it.
- Second harness failure, logged 2026-10-13: on the restarted wolfssl runs, the end-of-run capture staged nothing, because git add stops when any one of the eight C/C++ patterns matches no file (wolfssl has no .cxx files). GPT-6 Luna's and GPT-6.1 Sol's patches came out empty though both had edited sp_int.c; Claude Opus 5.5 and Claude Haiku 5.5 were stopped mid-run. All four moved to runs-harness-failures/second-start, never counted. The capture now adds each pattern separately, checked with stand-in agents on the wolfssl image (Luna's exact edit is captured, a stray compile_error.txt is not) and on the pandas GSO image (the source edit is captured; .venv and compiled files are not; the expert commit is absent from the agent's copy). All four wolfssl runs start again.
- gpac 44766 failed validation, logged 2026-10-13 before any agent ran it: the reference fix passes the crash, replay and output checks but not the unit-test check. Under load 3 of the 871 tests in PatchBench's answer key failed (dash-read-seek and two multithreaded filter tests); rerun once alone, 6 failed (dash and http push tests). gpac's suite does not hold steady here, so by the rule above it is replaced by the next task in seed-7003 order from a new project: libxml2 65409 (CWE-416 use after free). libxml2 validated: the reference fix passes all four checks, an empty change fails the crash input.
- Noted 2026-10-13 after the libxml2 runs, before any grading change: Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Luna all stop the original crash (poc pass) and all score 0. PatchBench injected this bug by reverting a 1,803-line upstream change, and its output check (byte-for-byte against the reference on benign inputs) and one unit test ("Error cases stream regression tests") effectively require restoring that change; Opus's and Sol's one-line fixes miss only those, plus a crash-variant replay that PatchBench could not classify (counted as a fail by its rules). The task passed validation, so it stays and is scored by PatchBench's rules; the report flags it, and keeping it for counted rounds goes to Bren and the reviewers. One Claude Haiku 5.5 libxml2 run failed on a sign-in error before the model produced anything; it was set aside, never counted, and rerun.
- GSO runs: one stream per model on its own 8 pinned cores (Opus 0-7, Haiku 8-15, Sol 16-23, Luna 24-31), seeds 7121 to 7124, grading deferred; the 8 runs are then graded one at a time with nothing else running.


## HM3, hardmix counted round (logged 2026-10-13, after the pilot and its review, before any counted run)
- Pilot result (1 run each, not counted): GPT-6.1 Sol 5/8, Claude Opus 5.5 5/8, Claude Haiku 5.5 3/8, GPT-6 Luna 1/8 (runs-pilot, runs-pilot-2; table built by report.py). GSO ratios against the expert commit: numpy Opus 1.012 (pass), Haiku 0.949, Sol 0.904, Luna 0.761; pandas Opus 0.736, Sol 0.438, Haiku 0.434, Luna 0.428.
- Review of the pilot by Fable 5.1, Grok 4.7 and GPT-6 Astra (Bren's request), decisions by majority:
  - No retained score shown to be caused by the harness (all three). GPT-6 Luna's empty wolfssl patch was checked in its transcript: no file edits; it concluded the bug was in the fuzz harness's caller.
  - libxml2 65409 replaced by the next seed-7003 task from a new project, libredwg 31591 (all three): every model's one-line fix in valid.c stopped the crash, but the output and unit-test checks require the reference's unrelated parser rewrite. Its pilot runs stay in the pilot record.
  - ansible kept but reported separately (Fable, Grok; Astra would replace): its manager tests hinge on get_manager(), which the task text never names. Fable noted Claude Opus 5.5's passing patch reproduces the upstream module almost verbatim, so a pass there may be recall; the report says so.
  - Claude Opus 5.5's numpy pass is a legitimate optimization by GSO's HackDetector criteria (all three).
  - Claude Haiku 5.5's numpy 0.949 against the 0.95 bar stays a 0 and is printed with its ratio, marked within timing noise (about 1%), never re-graded.
  - Pilot runs do not count as run 1 (Fable, Astra; Grok would allow it): the pilot was preregistered as not counted.
- Counted round: the 8 tasks (ansible, flipt, protonmail, navidrome, pandas-perf, numpy-perf, wolfssl-sec, libredwg-sec once validated) x 4 models x 3 new runs = 96, same settings. To avoid the shared-login refresh that revoked a token in the pilot, Claude Code runs one at a time (Opus and Haiku in one stream); GPT-6.1 Sol and GPT-6 Luna each get a stream. Run order within a stream is seeded (7201 Claude, 7203 Sol, 7204 Luna). GSO agent runs are pinned to 8 cores per stream and graded afterward, one at a time, with nothing else running; other runs are graded as they finish. Reported per task and model: runs solved of 3, every run's grader detail (GSO ratios, PatchBench checks), minutes, output tokens, and whether any run hit the cap. ansible is reported in its own row.
- libredwg 31591 runs: once it validates, queued behind each stream (seeds 7211 Claude, 7213 Sol, 7214 Luna) so Claude Code still runs one at a time. PatchBench grading runs as runs finish; GSO grading runs last, alone.


## Two suites, and deep suite v1 (logged 2026-10-13, Bren's naming)
- Launch suite: the existing tests (speed, TPS, skills, real app work, ASD-STE100), run on each model launch; they show cost, tokens and time, since every model is near full marks on their correctness checks. Deep suite: the hardmix tasks, run on big launches or when a model is said to have improved; it shows ability. They are reported on separate pages and data keys and never mixed.
- Deep suite v1 frozen in deep-v1.json: the 8 counted tasks, 3 runs each, medium effort, 30-minute cap, Claude Code 2.1.284 and Codex CLI 0.159.0, the proxy allowlist, and the exact image IDs (registry digests for SWE-bench Pro, PatchBench and GSO bases; local IDs for the two built GSO images, which are kept, not rebuilt). `make deep MODEL=<id>` checks all of it before any run and stops on a difference, runs only the missing runs (harness failures do not count), keeps Claude Code runs one at a time and timed GSO grading alone through machine-wide locks, and grades. Changing any task, setting or image makes a new version, whose scores are not compared with v1's. The baseline round's runs move to runs-deep-v1 when it finishes.
- Logged 2026-10-13: `make deep` grades each run in the background (two at a time) while the next run works; GSO's timed grading still waits for the machine lock alone.