Claude Opus 5.5, GPT-6.1 Sol and Claude Haiku 5.5 passed almost the same number of hard coding tasks.

Why this test: on the other pages, my launch tests give every model near full marks, so they can't show a model getting better. The 8 tasks come from three public task sets, credited below: features and bug fixes, speeding up Python code, and memory-safety fixes in C. 7 of them count in the scores above; the other is shown on its own. I picked the tasks at random before any model ran, then checked each one: the original fix must pass and untouched code must fail. I replaced 2 picked tasks: in one the original fix failed here, and the other could only be passed by rewriting unrelated code. One more was replaced before that, because its fix needs a library that isn't installed and tries have no internet.

This is version 1, frozen 13 October 2026: these 8 tasks, 3 tries each, the same settings and the same containers. A new model gets the same set. If I change a task or a setting, that's a new version, and it isn't compared with this one.

Claude Opus 5.5 and Claude Haiku 5.5 ran in Claude Code 2.1.284, and GPT-6.1 Sol and GPT-6 Luna in Codex CLI 0.159.0, all at medium effort, 30 minutes per try (1 try hit the limit). Each try ran in the task's own container with no internet except the model's API: no web search, no downloads, no way to look up the original fix. The model never saw the hidden tests or the original fix; the speed-up tasks include their timing script, as intended. Its change was then copied into a fresh container and graded by the task's own grader, which undoes any edit to existing tests. Claude Opus 5.5 and Claude Haiku 5.5 shared one login, so their tries went one at a time.

Costs use each vendor's standard API prices. Claude Haiku 5.5 is a range because Anthropic charges more for prompts over 100k tokens, and prompt sizes weren't logged. GPT costs are a range because Codex doesn't log cache writes, which OpenAI bills at 1.25 times the input price. Claude Opus 5.5's try 1 on wolfSSL hit the time limit before logging its tokens, so 20 of its tries have a cost. Minutes are start to finish, tests included.

The security tasks have 4 checks, each on a fresh build: the crash input no longer crashes; none of the task's other crash inputs or normal inputs trips the memory checker; every library test that passes with the original fix still passes; and output on normal inputs matches the original fix exactly.

Limits: this test can't confirm that a model improves week to week. It has 7 counted tasks and 3 tries each, so one try moves a model's count a lot, and each model ran in one tool at one effort. These are public tasks, so a model may have seen the original fix in training; the try table marks tries that repeated most of it. The speed-up tasks were picked from those whose expert fix changes Python only, so a rebuild fits in one try.

A trial round ran first, 1 try per task, and isn't counted. It found bugs in my setup, each logged; the affected tries were set aside and rerun. Fable 5.1, Grok 4.7 and GPT-6 Astra reviewed it before these tries.

Every try's record is in results.json, under deep_runs, deep_checks, deep_pilot_runs and deep_suite. The plan and every change to it are in deep-suite-plan.txt.

Credit: the tasks and their graders come from SWE-bench Pro's public hard set (Scale AI), GSO and PatchBench.