GPT-6.1 Sol and Claude Opus 5.5 passed every try; Claude Haiku 5.5 passed 6 of 9 and GPT-6 Luna none.

Version 1 used tasks from public task sets. For version 2 I wrote my own tasks in one real open-source web app: an audit for data people shouldn't see, a feature that shipped broken, and a data export that has to get fast without changing its output. Each task has hidden checks the model never sees. Security and Broken feature score a try by the share of their graded parts it passes, and a try passes when every part does. Slow export scores a try half by how much of my own fix's time savings it matches and half by its query savings, capped at 1, and a try passes at 0.8 or more within the task's time and query limits. A try that breaks any of the app's own tests scores 0. Tasks, prompts and graders are private so later runs stay fair.

Before counted runs began, each task was checked the same way: my own fix scores 1, the untouched code scores 0, and the tested wrong fixes scored below 1. Three outside reviewers (GPT-6 Astra, Grok 4.7 and Gemini 4 Argon) checked every task for fairness, and I made the tasks harder twice after trial rounds where too many tries passed. The trial rounds aren't counted.

This is version 2, frozen 15 October 2026: 3 areas, 3 tries each, the same settings and the same containers. Each version tests something different, so scores don't carry over from version 1.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, 20 minutes per try (2 tries hit the limit and were graded on the code at that point). Each try ran in its own container with no internet except the model's API. Its change was then copied into a fresh container and graded there, after the app's own tests were put back. The two tools ran side by side; the export task, which measures time, was graded afterwards one try at a time with nothing else running.

Costs are estimates at each vendor's standard API prices, from the tokens each try logged. Claude Haiku 5.5 is a range because Anthropic charges more for prompts over 100k tokens and prompt sizes weren't logged; GPT costs are a range because Codex doesn't log cache writes, which OpenAI bills at 1.25 times the input price. 2 tries stopped at the time limit before logging tokens, so they have no cost estimate and aren't in the totals. The tries ran on subscriptions, so these are not what was spent.

Limits: 3 areas and 3 tries each, one app, one tool and one effort setting per model. Two models passed every try, so these scores can't tell their ability apart on this suite, only which was faster and cheaper here.

Every try's area, score, time and cost is in results.json, under deep_v2_runs and deep_v2_suite.