Claude Opus 5.5 led on speed-up and scheduling, GPT-6.1 Sol on forecasting, and both averaged 0.98 or more on every correctness task.

In version 2 the two strongest models passed every try, so its scores couldn't tell them apart. For version 3 I wrote six tasks in six languages: two in real open-source code as released, four in applications I wrote for the suite. Three are open-ended: they measure how much a try improved the code, on a log2 scale, and aren't capped at my own fix's score; the other three score the share of hidden checks a try gets right. Each task keeps its own score, so there is no overall number or rank. Tasks, prompts and graders are private so later runs stay fair.

The hidden checks and hidden data never enter the model's container, and every model got the same seeds. A try whose code doesn't build or breaks the project's own tests is invalid and scores 0. Fable 5.1 and GPT-6 Astra reviewed the plan before the counted runs began; trial runs aren't counted.

This is version 3, frozen 15 October 2026: 6 tasks, 2 tries each, the same settings and the same containers for every model. Each version tests something different, so scores don't carry over from earlier versions.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, 20 minutes per try (10 for the job service). The code was saved at the halfway point without stopping the model, and that copy was graded too. 11 tries hit the limit and were graded on the code at that point. Each try ran in its own container with no internet except the model's API, and was graded in a fresh container. Tasks that measure time were graded one try at a time with nothing else running. Runs were paced to stay under a fixed share of each subscription's usage.

Costs are estimates at each vendor's standard API prices, from the tokens each try used. Claude Code logs no usage for a try stopped at the time limit, so for those 11 Claude tries I counted the tokens from the try's own requests, the same way Claude Code counts them on tries that finished. Claude Haiku 5.5 is a range because Anthropic charges more for prompts over 100k tokens and prompt sizes weren't logged; GPT costs are a range because Codex doesn't log cache writes, which OpenAI bills at 1.25 times the input price. The tries ran on subscriptions, so these are not what was spent.

Limits: 6 tasks and 2 tries each, one tool and one effort setting per model. Two tries can't show which gaps would hold up over more runs, so none is called significant here.

Every try's task, score, time and cost is in results.json, under deep_v3_runs and deep_v3_suite.