Claude Opus 5.5, GPT-6.1 Sol and Claude Haiku 5.5 were within one run of each other on hard coding tasks.

Why a deep suite: my launch suite, on the other pages, has every model near full marks, so it can show a model getting worse but not better. The tasks: building features and fixing bugs, speeding up Python code, and fixing memory-safety bugs in C, taken from three public task sets (credited below). Within each set the tasks were drawn in a fixed random order before any model ran, then checked: the task's own reference fix must score as solved and an unchanged repository must not. Two drawn tasks were replaced by the next in the same order: one whose reference fix failed that check here, and one that the pilot's reviewers found could only be passed by rewriting unrelated code. One drawn Go task was swapped before that because its fix needs a library the task's image does not have, and runs have no internet.

Deep suite v1, frozen 13 October 2026: these eight tasks, three runs each, with the same settings and the same container images, checked before every run. A new model runs the same set and is compared with the runs here. Changing a task, a setting or a container image makes a new version, and its scores are not compared with these.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, each run capped at 30 minutes (1 run reached it). The original tasks allow longer. Each run worked inside the task's own container, whose only way out reached the two vendors' model servers: no web search, no package downloads and no way to look up the original fix. The run never saw the hidden tests or the reference fix; the speed-up tasks' text includes their timing script, as intended. Its change was then applied to a fresh copy of the container and graded by the task's own grader, which resets any test file the run edited. Claude Opus 5.5 and Claude Haiku 5.5 shared one login, so their runs went one at a time.

Limits: this suite cannot confirm that a model is improving week to week. It has a handful of tasks and three runs each, so a single run moves a model's count by a lot, and it measures each model inside one tool at one effort. These are public tasks, so a model may have seen the original fix in training; the run table marks runs that reproduced most of it. The speed-up tasks were picked from the ones whose expert fix changes Python only, so a rebuild fits in a run.

A pilot ran first, one run per task, and is not counted. It found bugs in my setup, each logged, with the affected runs set aside and rerun, and Fable 5.1, Grok 4.7 and GPT-6 Astra reviewed its results before these runs were made. Its runs are in the data.

The run records are in results.json, under deep_runs, deep_checks, deep_pilot_runs and deep_suite. The plan and every change to it, as logged, are in deep-suite-plan.txt.

Credit: the tasks and their graders come from SWE-bench Pro's public hard set (Scale AI), GSO and PatchBench.