Token-saving skills hit their claim in 1 of 29 checks.
The one claim that held was caveman's shorter answers on Claude Sonnet 5. In round 13, the latest, it cut 72% against a claimed 65%, and every one of its answers skipped the impossible-date case, a date like 2024‑02‑30. I ran caveman, ponytail and karpathy-skills against the same task with no skill, three runs each way. That made 372 runs on six models and four small tasks, in Claude Code and Codex CLI. The Models page compares the models themselves.
On average across the models, caveman and ponytail missed all 5 claimed numbers, and 3 averaged the other way.
Each bar is the models' average cut against no skill, on the task closest to each claim. The skill's own claim is at the right.
Source: skill-evals.brenhq.com, updated 16 October 2026. 3 runs per model, task and skill, up to 6 models, in Claude Code and Codex CLI. The Claude runs on the explain task are from round 13, with the account extras off. The others had them on.
Show each model's cut as a table
| Claim | Claude Sonnet 5Sonnet 5 | Claude Opus 5.5Opus 5.5 | Claude Sonnet 5.5Sonnet 5.5 | GPT‑5.6 Luna5.6 Luna | GPT‑6 Luna6 Luna | GPT‑6.1 Sol6.1 Sol |
|---|---|---|---|---|---|---|
| cavemanshorter answers, 65% claimed | 72% fewer | 26% fewer | 28% fewer | 1% fewer | not compared | 36% fewer |
| ponytailless code, 54% claimed | 24% less | 26% less | 37% less | 14% less | 11% less | 14% less |
| ponytailfewer tokens, 22% claimed | 34% more | 32% more | 6% more | 5% more | 23% more | 8% fewer |
| ponytailcheaper, 20% claimed | 16% pricier | 6% pricier | 10% pricier | 10% cheaper | 15% pricier | 9% pricier |
| ponytailfaster, 27% claimed | 11% slower | 12% faster | 6% faster | 27% slower | 11% faster | 9% slower |
Bold on green: a cut that held its claim.
caveman107,335 GitHub stars
Held on 1 of 5 checks
I found
It held on Claude Sonnet 5 and no other model. caveman's earlier sets of runs on Claude Sonnet 5 cut 66%, 62% and 64%. The verdict uses round 13, the latest. GPT-6 Luna is left out because some of its runs on that task failed.
It claims
cuts 65% of tokens
Its README dropped the 65% on 8 September 2026, before my runs. The description still says it.
From its GitHub description.
I measured visible output tokens, which count what the model writes minus hidden thinking, on the explain task. That task asks for one written answer, the closest of my tasks to a single reply.
How it measured that
The 65% came from a results table in the old README. That table counted output tokens per reply on ten coding prompts, against no prompt at all. The README now reports 50% fewer output tokens against an "Answer concisely." prompt, which I do not test. I tested the skill file only, not the proxy that caveman now ships.
File tested: SKILL.md at commit 880114a.
ponytail144,278 GitHub stars
Did not hold
I found
It held on no model.
It claims
~54% less code (up to 94%) · ~20% cheaper · ~27% faster · 100% safe
Its results table also lists −22% tokens.
From its README.
I measured lines of code, total tokens, cost at API prices and time on the build task. That task is my biggest coding task. I don't test safety. With no skill, the build task's median ran from 49 to 62 lines, depending on the model.
How it measured that
It ran 12 feature tickets in Claude Code on Haiku 4.5, four runs each, and counted lines added in the diff, 191 a ticket without the skill. Its token count covers the whole session, input included, about 349,000 a ticket without the skill. It says cost and time can rise on some reasoning models.
File tested: AGENTS.md at commit b6c0448.
karpathy-skills214,632 GitHub stars
No number claimed
I found
Less code on 6 of 6 models, every run on 5.
It says
"Minimum code that solves the problem" and "fewer unnecessary changes in diffs"
From its CLAUDE.md.
I measured lines of code added, on the build task.
The four tasks
- Explain taskExplain in prose how the code computes monthly totals and what happens with an invalid date, without changing any file.
- Feature taskA small Python expense ledger gets a failing test for a new function.
- Bug fix taskA failing test exposes a rounding bug in the same ledger.
- Build taskThe agent adds a command-line report to the ledger.
Want a skill tested next? Ask on X.
Where each claim fell short#
caveman: shorter answers#
In round 13, every caveman run on Claude Sonnet 5 came in under the shortest no-skill run. All three of its answers skipped the impossible-date case, which all three no-skill answers covered. A date like 2024‑02‑30 has the right format, but the code fails when it builds the date.
caveman held its claimed 65% only on Claude Sonnet 5.
Visible output tokens on the explain task, the midpoint of 3 runs. Right of No skill means the skill used less. Visible output tokens: what the model writes, minus its hidden thinking.
- What the skill claims
- Outlined: skill and no-skill runs overlapped, so it could be chance
GPT-6 Luna left out: 9 of its 15 runs on this task failed.
Source: skill-evals.brenhq.com, updated 16 October 2026. 3 runs per model, task and skill, in Claude Code and Codex CLI. Each change is against the same task with no skill. The Claude runs are from round 13, with the account extras off. The others had them on.
Karpathy's ASD-STE100 tip, tested as a one-line instruction on the same explain task, has its own page.
Show each model and each set of runs
- No skill
- A skill
- Hollow: some runs failed, so the bar is not compared
caveman made Claude Opus 5.5's explanations 26% shorter, and all 15 of Opus 5.5's explain answers, with and without a skill, still covered the impossible-date case. It cut GPT-6.1 Sol's explanations 36% at medium effort, every run below every no-skill run. GPT-6 Luna failed 9 of 15 explain runs because the model answered without reading the file, often saying it had no file tools. Its panel shows those failures and says nothing about the skills.
| Runs | No skill | caveman |
|---|---|---|
| Claude Sonnet 5, rounds 1 and 2Claude Code 2.1.278, high effort, 3 runs a side | 875 | 29466% fewer |
| Claude Sonnet 5, round 5Claude Code 2.1.278, high effort, 3 runs a side | 763 | 29262% fewer |
| Claude Sonnet 5, round 9Claude Code 2.1.284, high effort, 2 runs a side | 815 | 296*64% fewer |
| Claude Sonnet 5, round 13Claude Code 2.1.284, high effort, 3 runs a side | 956 | 27272% fewer |
| Claude Opus 5.5, round 6Claude Code 2.1.280, medium effort, 3 runs a side | 1,203* | 93922% fewer |
| Claude Opus 5.5, round 13Claude Code 2.1.284, medium effort, 3 runs a side | 988 | 72826% fewer |
| Claude Sonnet 5.5, round 9Claude Code 2.1.284, medium effort, 3 runs a side | 858 | 74613% fewer |
| Claude Sonnet 5.5, round 13Claude Code 2.1.284, medium effort, 3 runs a side | 864 | 62128% fewer |
| GPT-5.6 Luna, rounds 1 and 2Codex CLI 0.153.3, low effort, 3 runs a side | 226 | 2231% fewer |
| GPT-5.6 Luna, round 5Codex CLI 0.153.3, low effort, 3 runs a side | 216 | 2181% more |
| GPT-5.6 Luna, round 8Codex CLI 0.157.1, medium effort, 2 runs a side | 230 | 2109% fewer |
| GPT-6 Sol, round 8Codex CLI 0.157.1, medium effort, 2 runs a side | 228 | 15632% fewer |
| GPT-6 Sol, round 10Codex CLI 0.159.0, medium effort, 2 runs a side | 199 | 14428% fewer |
* This median comes from a light run, a session's first, before Claude Code loaded the account extras.
Each row is one set of runs, with its rounds, release, effort and runs a side under the model's name. For the Claude models the charts use round 13, run with the account extras off. In round 13 caveman cut more on every Claude model than in any earlier set. It made no clear difference on GPT-5.6 Luna in any of its three sets.
ponytail and karpathy-skills: less code#
Both skills wrote less code on every model.
ponytail cut code by 11% to 37%, short of its claimed 54%, and karpathy-skills by 8% to 24%, clearly on 5 of 6.
Lines of code on the build task, the midpoint of 3 runs. Right of No skill means the skill used less. Lines of code: lines added in the diff the run leaves behind.
- What the skill claims
- Outlined: skill and no-skill runs overlapped, so it could be chance
Code size is compared on the build task only. The explain task writes no code, and the feature and bug fix edits added 1 to 8 lines with no skill, too few to compare.
ponytail
karpathy-skills
Source: skill-evals.brenhq.com, updated 16 October 2026. 3 runs per model, task and skill, in Claude Code and Codex CLI. Each change is against the same task with no skill. Every run had the account extras on.
Show each model's code on the build task
- No skill
- A skill
My control file, generic project guidelines that ask for a review pass, cut code only on GPT-6 Luna, 18%, the same as karpathy-skills there. On the other five models it wrote as much code as no skill or more, up to 32% more on GPT-6.1 Sol.
Cost at API prices#
Every skill's median cost more than no skill's on every model except ponytail on GPT-5.6 Luna.
Cost at API prices for all four tasks, the midpoint of 3 runs per task added up. The no-skill range runs from each task's lowest no-skill run, added up, to each task's highest, added up. Right of No skill means the skill used less. Cost at API prices: what the run's tokens cost at the model's published API prices, cached tokens at the cache price.
- Outlined: the skill's total fell inside the no-skill range, so it could be chance
caveman
ponytail
karpathy-skills
† Leaves out GPT‑6 Luna's explain task, where runs failed.
Source: skill-evals.brenhq.com, updated 16 October 2026. 3 runs per model, task and skill, in Claude Code and Codex CLI. Each change is against the same task with no skill. The Claude runs on the explain task are from round 13, with the account extras off. The others had them on.
ponytail claims to be 20% cheaper. The chart adds up all four tasks, but I check that claim on the build task alone. There it ranged from 10% cheaper on GPT-5.6 Luna to 16% pricier on Claude Sonnet 5, and averaged 8% pricier. caveman made Claude Sonnet 5's explain answers 72% shorter but only 2% cheaper. Output was about 15% of that task's cost, and caveman's own file added input to every run.
Show each model's cost and how it is priced
- No skill
- A skill
For Claude, these dollar figures are Claude Code's own estimate. It prices the tokens each run sent at Anthropic's API prices, with cache writes at the one-hour rate a subscription login uses. With an API key's default five-minute cache, the same runs price 21% lower on Sonnet 5, 27% lower on Sonnet 5.5 and 29% lower on Opus 5.5. Each skill's gap to no skill moves by up to 3 points. The estimate is not a bill, and a task built directly on the API would send a different prompt and tools.
Every measure and task#
Pick any measure and task to see every skill against no skill on every model.
Across all four tasks added up, caveman clearly cut output on 0 of 6 models, ponytail on 0 of 6 and karpathy-skills on 0 of 6.
Visible output tokens for all four tasks, each task's midpoint of 3 runs, added up. The no-skill range runs from each task's lowest no-skill run, added up, to each task's highest, added up. Right of No skill means the skill used less. A claimed number shows only when you pick the task I check it on. Visible output tokens: what the model writes, minus its hidden thinking.
- Outlined: the skill's total fell inside the no-skill range, so it could be chance
caveman
ponytail
karpathy-skills
† Leaves out GPT‑6 Luna's explain task, where runs failed.
Source: skill-evals.brenhq.com, updated 16 October 2026. 3 runs per model, task and skill, in Claude Code and Codex CLI. Each change is against the same task with no skill. The Claude runs on the explain task are from round 13, with the account extras off. The others had them on.
Show every model and task, and where the runs overlapped
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | 72% fewer, every run | 4% more, runs overlapped | 10% more, runs overlapped | 8% fewer, runs overlapped |
| ponytail | 23% fewer, runs overlapped | 34% more, runs overlapped | 28% more, runs overlapped | 6% more, runs overlapped |
| karpathy-skills | 20% fewer, runs overlapped | 20% more, every run | 19% more, runs overlapped | 3% more, runs overlapped |
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | 26% fewer, every run | 8% fewer, runs overlapped | 9% fewer, every run | same, runs overlapped |
| ponytail | 2% fewer, runs overlapped | 4% fewer, runs overlapped | 12% fewer, runs overlapped | 11% fewer, runs overlapped |
| karpathy-skills | 6% more, runs overlapped | 14% more, every run | 14% fewer, every run | 5% fewer, every run |
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | 28% fewer, every run | 5% more, runs overlapped | 16% fewer, runs overlapped | 8% more, runs overlapped |
| ponytail | 10% fewer, runs overlapped | 37% more, runs overlapped | 12% more, runs overlapped | 14% fewer, every run |
| karpathy-skills | 1% fewer, runs overlapped | 46% more, every run | 5% fewer, runs overlapped | 1% more, runs overlapped |
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | 1% fewer, runs overlapped | 11% fewer, every run | 9% more, runs overlapped | 6% fewer, runs overlapped |
| ponytail | 17% more, every run | 5% fewer, runs overlapped | 18% more, runs overlapped | 4% fewer, runs overlapped |
| karpathy-skills | 10% more, every run | 11% more, runs overlapped | 58% more, runs overlapped | 12% fewer, runs overlapped |
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | not compared: a no-skill run failed | 15% fewer, runs overlapped | 24% fewer, every run | 27% fewer, runs overlapped |
| ponytail | failed: 3 of 3 runs failed | 6% more, runs overlapped | 9% more, runs overlapped | 8% fewer, runs overlapped |
| karpathy-skills | failed: 2 of 3 runs failed | 1% fewer, runs overlapped | 7% more, runs overlapped | 16% fewer, runs overlapped |
| Skill | Explain | Feature | Bug fix | Build |
|---|---|---|---|---|
| caveman | 36% fewer, every run | 18% fewer, every run | 11% fewer, every run | 7% fewer, runs overlapped |
| ponytail | 26% fewer, every run | 11% fewer, runs overlapped | same, runs overlapped | 9% fewer, every run |
| karpathy-skills | 12% fewer, runs overlapped | 15% fewer, runs overlapped | 14% more, runs overlapped | 3% fewer, runs overlapped |
- Every run used fewer tokens than every no-skill run
- Every run used more
- Runs overlapped, so it could be chance
- Some runs failed
- Not compared: a no-skill run on that task failed
In 19 of 69 cells, every skill run landed on the same side of every no-skill run. Chance alone would give about 7.
Method and data#
Limits
- Each skill ran three times per task and model. That shows which way a number moved, but three runs are too few to prove it.
- The four tasks are small and share one Python project. Only the build task is big enough to show a difference in lines of code, and lines include blanks and comments.
- The charts cover six models, one setting each. Claude Sonnet 5, Claude Opus 5.5 and Claude Sonnet 5.5 ran at Claude Code's default effort, which is high for Sonnet 5 and medium for the other two. GPT-5.6 Luna and GPT-6 Luna ran at low effort, and GPT-6.1 Sol at medium on a newer Codex CLI release. Outside the charts, round 8 ran Claude Fable 5.1, GPT-5.6 Sol, GPT-6 Sol and GPT-6 Astra on the explain task and reran both Lunas at medium effort.
- Visible output leaves out hidden thinking, a median 51% of output across Sonnet 5's build-task runs. Total tokens and cost still count it.
- I ran on subscriptions, with no API keys. For Claude, cost is Claude Code's own estimate from each model's published rates per million tokens, and it includes Claude Code's own system prompt and tools. Claude Sonnet 5 and Claude Sonnet 5.5 each list $2 for input, $4 for one-hour cache writes, $0.20 for cache reads and $10 for output. Claude Opus 5.5 lists $4, $8, $0.20 and $20. Codex CLI reports no cost. A GPT run's cost is its tokens at OpenAI's published rates, cached input at the cache rate, with Codex CLI's hidden first request added.
13 more limits
- Time moves with the service's latency. Small time differences mean little.
- An explain answer passes if it names ValueError and month. That shows it is on topic, not that it is complete.
- My control file is a short set of generic project guidelines I wrote after the first two rounds. It asks for a review pass and is not a neutral placebo. It stays out of the charts and appears in the full tables.
- Rounds 1 to 4 did not block runs from editing the tests or keep each run's diff. Round 5 and every round from 6 on did both.
- Skills added in later rounds ran after the no-skill runs I compare them with.
- I logged each round's plan before running it. The plan itself is private.
- My scripts and earlier run records were readable inside the container, except in rounds 6 and 7. No Codex CLI run log shows an agent reading them, and the Claude runs can't be checked.
- Whether an explain answer covered the impossible-date case comes from a text search of each answer, which I checked by reading them.
- Until round 13, each CLI also loaded the account extras a subscription login brings by default. For Claude Code these are the plugins and skills enabled on the account's claude.ai, synced after a session's first run, and the account's claude.ai connectors. For Codex CLI they are its connected apps and plugins, built-in skills included. Round 13 turned the account extras off: Claude Code with no MCP servers and no claude.ai sync, Codex CLI with its apps and plugins off. No Codex CLI run log shows any of them used. Claude Code's output does not list tool calls, so its runs can't be checked.
- GPT-6 Luna's tools worked. In checks outside the benchmark it ran shell commands whenever asked to. It still often answered the explain task without opening the file. Turning off Codex CLI's apps and code-mode host, raising effort to medium, and adding a note that it can run commands did not stop that. I report those runs as failures rather than change its task. In round 8, on a newer Codex CLI release, GPT-6 Luna read the file on every explain run. The failures happened only on the older release.
- Codex CLI leaves its hidden first request out of its own token count. I measured it once for each model and CLI release, not in each run, and add it to every Codex CLI run. Its size did not change with the skill file or the task prompt.
- Sonnet 5.5's runs went in three parallel streams on one Claude Code release, which can shift its times slightly. The launch-day comparison on the Models page uses its first two explain runs with no skill.
- Some Claude Code runs started with thousands fewer tokens of instructions and tools than the same model's other runs. I call them light runs. On the explain task every run takes two or three steps. There I mark a run light when its last request is more than 4,000 tokens smaller than the largest from the same model with the same number of steps. Each light run was a session's first, before Claude Code loaded the account extras. Of the 18 light explain runs across all rounds, 7 ended by saying the account's connectors needed authorization, and no other run said so. I removed that remark from the published answers, but their token counts still include it. Round 13 reran every Claude model's explain runs, the Models page's explain runs and the real app work with the account extras off, and none of those is light. The explain runs still light all ran before round 13. They appear in round 8's table, the Models page's launch-day runs, the set-by-set table's rows from before round 13, and the run list. The coding tasks take many more steps. There I mark a run light when it wrote over 4,000 fewer tokens to the cache than the top run with the same model, task and skill. That marks 12 Claude coding runs, and no median has half its runs light. A number where at least half the runs behind it are light carries a star. The coding tasks had the account extras on both with and without a skill, which keeps the comparison within a model fair.
How a run works
Each run is a fresh session on a clean copy of the same small Python project. The container holds Python, pytest and the model's CLI at the release listed with its results. The only instruction file in a run is the skill or control file under test. Both CLIs sign in with subscriptions, not API keys, and load that login's account extras, which the Limits define. Round 13 turned them off for the Claude explain runs.
To test a skill, I copy its instruction file, pinned to a commit, into the project as CLAUDE.md and AGENTS.md. No-skill runs have neither file, and control runs use the control file described in the Limits. I shuffle the runs and drop none. Token counts come from what the CLIs report, plus Codex CLI's hidden first request, which I measured.
I tested every skill with more than 100,000 GitHub stars that claims to cut tokens or code and ships as an instruction file an agent reads. I read the star counts from GitHub on 22 September 2026. I considered these and left them out.
- superpowers, with 290,109 stars, is a development method with no token or code savings claim.
- ECC, with 265,217 stars, is a full agent setup with its own hooks and memory, not one instruction file.
- gstack, with 133,921 stars, is a set of role tools with no savings claim.
- graphify, with 120,473 stars, is a codebase knowledge-graph tool, not an instruction file.
- Explain task
- Explain in prose how the code computes monthly totals and what happens with an invalid date, without changing any file. An answer passes if it names ValueError and month.
- Feature task
- A small Python expense ledger gets a failing test for a new function. The agent has to make the whole suite pass without touching the tests.
- Bug fix task
- A failing test exposes a rounding bug in the same ledger. The agent again has to make the whole suite pass without touching the tests.
- Build task
- The agent adds a command-line report to the ledger. It reads a CSV file and prints each category's total for one month, largest first, then the grand total. Four failing tests define the output and the error cases. It is the one task where the number of lines of code can vary a lot.
Visible output tokens are what the model wrote minus its hidden thinking. They count its answer and its tool calls. Total tokens count everything it read and wrote, including text served from the cache. Every run records whether it passed its task's test, and 363 of 372 did. Claude Code and Codex CLI count cached text differently. Each CLI also adds its own instructions to what the model reads. Token counts compare best within one model. I keep the test code and exact prompts private so skill authors can't tune their skills to them.
For the coding tasks, the tests are the only quality measure. After the runs I wrote extra tests with new inputs for the same behavior. I ran them against the kept changes from all 180 coding runs of Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Luna and GPT-6.1 Sol. The main rounds with Claude Sonnet 5 and GPT-5.6 Luna kept no changes to test. All 180 passed. These tasks can't separate those models on correctness.
All the numbers
The full tables show every skill on every task, with hidden thinking and time. The Models page puts every model's speed, tokens and cache use side by side, with no skill.
Show the results by task
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 976 | 744 to 984 | 0 | 2% | 3 of 3 | |||
| my control file | −24% | 743 | 724 to 963 | 0 | +3% | 3% | −9% | 3 of 3 |
| caveman | −70% | 289 | 239 to 291 | 0 | −2% | 5% | −45% | 3 of 3 |
| ponytail | −21% | 767 | 749 to 917 | 0 | +5% | 3% | +9% | 3 of 3 |
| karpathy-skills | −20% | 785 | 745 to 907 | 0 | −5% | 3% | −18% | 3 of 3 |
| Feature task | ||||||||
| No skill | 1,145 | 1,047 to 1,197 | 5 | 9% | 3 of 3 | |||
| my control file | +22% | 1,396 | 1,187 to 1,540 | 6 | +10% | 7% | +21% | 3 of 3 |
| caveman | −10% | 1,026 | 836 to 1,156 | 5 | +8% | 3% | +4% | 3 of 3 |
| ponytail | +20% | 1,369 | 948 to 1,483 | 7 | +11% | 10% | +5% | 3 of 3 |
| karpathy-skills | +5% | 1,197 | 1,117 to 1,279 | 5 | +6% | 4% | +6% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 1,223 | 1,111 to 1,277 | 2 | 11% | 3 of 3 | |||
| my control file | +68% | 2,050 | 1,596 to 2,054 | 2 | +6% | 36% | 0% | 3 of 3 |
| caveman | +3% | 1,256 | 816 to 1,337 | 2 | +1% | 5% | −26% | 3 of 3 |
| ponytail | +40% | 1,718 | 1,552 to 1,824 | 4 | +12% | 27% | −7% | 3 of 3 |
| karpathy-skills | +36% | 1,666 | 1,317 to 1,791 | 2 | +15% | 29% | −2% | 3 of 3 |
| Build task | ||||||||
| No skill | 3,822 | 3,161 to 3,891 | 62 | 50% | 3 of 3 | |||
| my control file | +28% | 4,895 | 3,077 to 5,249 | 63 | +4% | 57% | +39% | 3 of 3 |
| caveman | −17% | 3,191 | 3,124 to 3,258 | 57 | +5% | 49% | −7% | 3 of 3 |
| ponytail | −2% | 3,761 | 3,267 to 4,661 | 47 | +16% | 51% | +11% | 3 of 3 |
| karpathy-skills | −5% | 3,636 | 3,566 to 4,310 | 56 | −4% | 54% | +7% | 3 of 3 |
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 1,167 | 1,066 to 1,429 | 0 | 17% | 3 of 3 | |||
| my control file | +4% | 1,210 | 1,066 to 1,227 | 0 | +9% | 17% | +2% | 3 of 3 |
| caveman | −22% | 907 | 880 to 990 | 0 | +20% | 20% | −7% | 3 of 3 |
| ponytail | +9% | 1,272 | 1,010 to 1,294 | 0 | +13% | 19% | +9% | 3 of 3 |
| karpathy-skills | +5% | 1,223 | 1,176 to 1,420 | 0 | +12% | 20% | +14% | 3 of 3 |
| Feature task | ||||||||
| No skill | 652 | 574 to 693 | 6 | 0% | 3 of 3 | |||
| my control file | +43% | 933 | 896 to 956 | 8 | +14% | 4% | +49% | 3 of 3 |
| caveman | −5% | 619 | 570 to 803 | 6 | +12% | 3% | +6% | 3 of 3 |
| ponytail | 0% | 651 | 616 to 689 | 6 | +5% | 4% | +3% | 3 of 3 |
| karpathy-skills | +18% | 772 | 751 to 786 | 6 | +7% | 4% | +28% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 1,139 | 1,109 to 1,247 | 12 | 9% | 3 of 3 | |||
| my control file | +17% | 1,333 | 1,252 to 1,630 | 7 | +12% | 18% | +33% | 3 of 3 |
| caveman | −13% | 994 | 993 to 1,041 | 7 | +25% | 7% | 0% | 3 of 3 |
| ponytail | +9% | 1,241 | 1,126 to 1,417 | 4 | +10% | 28% | +33% | 3 of 3 |
| karpathy-skills | −8% | 1,045 | 903 to 1,053 | 2 | +1% | 14% | +4% | 3 of 3 |
| Build task | ||||||||
| No skill | 1,659 | 1,496 to 1,681 | 54 | 7% | 3 of 3 | |||
| my control file | +22% | 2,028 | 1,874 to 2,048 | 66 | +7% | 12% | +12% | 3 of 3 |
| caveman | −3% | 1,601 | 1,534 to 1,617 | 56 | +24% | 8% | −5% | 3 of 3 |
| ponytail | −10% | 1,485 | 1,358 to 1,674 | 40 | +6% | 14% | −12% | 3 of 3 |
| karpathy-skills | −6% | 1,554 | 1,539 to 1,639 | 41 | +6% | 14% | −10% | 3 of 3 |
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 864 | 841 to 909 | 0 | 0% | 3 of 3 | |||
| my control file | −9% | 788 | 732 to 848 | 0 | +6% | 0% | +14% | 3 of 3 |
| caveman | −28% | 621 | 603 to 694 | 0 | +22% | 0% | −15% | 3 of 3 |
| ponytail | −10% | 774 | 718 to 842 | 0 | +9% | 0% | +28% | 3 of 3 |
| karpathy-skills | −1% | 857 | 845 to 900 | 0 | +11% | 0% | +14% | 3 of 3 |
| Feature task | ||||||||
| No skill | 506 | 498 to 652 | 6 | 0% | 3 of 3 | |||
| my control file | +67% | 844 | 830 to 858 | 8 | +11% | 0% | +7% | 3 of 3 |
| caveman | +5% | 530 | 510 to 625 | 8 | +26% | 0% | −14% | 3 of 3 |
| ponytail | +37% | 691 | 574 to 784 | 6 | +11% | 0% | +6% | 3 of 3 |
| karpathy-skills | +52% | 768 | 687 to 792 | 5 | +12% | 0% | −7% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 628 | 505 to 771 | 6 | 0% | 3 of 3 | |||
| my control file | +41% | 884 | 772 to 955 | 3 | +6% | 7% | +30% | 3 of 3 |
| caveman | −2% | 616 | 554 to 634 | 2 | +14% | 17% | −19% | 3 of 3 |
| ponytail | +41% | 886 | 728 to 1,316 | 4 | +5% | 16% | +32% | 3 of 3 |
| karpathy-skills | +1% | 634 | 500 to 660 | 2 | 0% | 6% | −5% | 3 of 3 |
| Build task | ||||||||
| No skill | 1,187 | 1,108 to 1,319 | 49 | 0% | 3 of 3 | |||
| my control file | +32% | 1,571 | 1,444 to 1,629 | 59 | +12% | 6% | +16% | 3 of 3 |
| caveman | +14% | 1,349 | 1,298 to 1,358 | 55 | +29% | 5% | +2% | 3 of 3 |
| ponytail | −2% | 1,159 | 1,101 to 1,248 | 31 | +10% | 12% | −6% | 3 of 3 |
| karpathy-skills | +14% | 1,349 | 1,275 to 1,405 | 43 | +14% | 7% | +4% | 3 of 3 |
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 240 | 234 to 247 | 0 | 6% | 3 of 3 | |||
| my control file | −1% | 237 | 232 to 252 | 0 | +17% | 6% | +46% | 3 of 3 |
| caveman | −3% | 232 | 217 to 246 | 0 | +15% | 4% | +11% | 3 of 3 |
| ponytail | +16% | 279 | 254 to 290 | 0 | +20% | 4% | +13% | 3 of 3 |
| karpathy-skills | +9% | 262 | 250 to 270 | 0 | +19% | 6% | +47% | 3 of 3 |
| Feature task | ||||||||
| No skill | 1,045 | 1,031 to 1,577 | 5 | 17% | 3 of 3 | |||
| my control file | +4% | 1,087 | 834 to 1,146 | 6 | −9% | 13% | 0% | 3 of 3 |
| caveman | −8% | 958 | 696 to 1,025 | 5 | −12% | 17% | −12% | 3 of 3 |
| ponytail | −3% | 1,009 | 985 to 1,135 | 5 | −11% | 17% | −14% | 3 of 3 |
| karpathy-skills | +15% | 1,199 | 986 to 1,285 | 5 | −12% | 17% | −19% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 849 | 779 to 1,123 | 2 | 15% | 3 of 3 | |||
| my control file | +49% | 1,265 | 847 to 1,432 | 2 | +10% | 22% | +1% | 3 of 3 |
| caveman | +2% | 870 | 725 to 1,002 | 2 | +13% | 18% | −18% | 3 of 3 |
| ponytail | +19% | 1,009 | 1,004 to 1,125 | 2 | +7% | 19% | +4% | 3 of 3 |
| karpathy-skills | +60% | 1,360 | 958 to 1,391 | 2 | +22% | 18% | +17% | 3 of 3 |
| Build task | ||||||||
| No skill | 1,858 | 1,567 to 2,343 | 51 | 16% | 3 of 3 | |||
| my control file | +9% | 2,031 | 1,569 to 2,060 | 51 | −15% | 17% | −12% | 3 of 3 |
| caveman | −9% | 1,691 | 1,336 to 2,072 | 50 | +12% | 15% | +6% | 3 of 3 |
| ponytail | +1% | 1,868 | 1,772 to 2,091 | 44 | −10% | 19% | +27% | 3 of 3 |
| karpathy-skills | −6% | 1,753 | 1,740 to 2,003 | 45 | −4% | 21% | −19% | 3 of 3 |
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 180 | 47 to 181 | 0 | 0% | 2 of 3 | |||
| my control file | −82% | 32 | 31 to 54 | 0 | −24% | 0% | −27% | 0 of 3 |
| caveman | −34% | 119 | 107 to 131 | 0 | +6% | 0% | −8% | 3 of 3 |
| ponytail | −73% | 48 | 48 to 51 | 0 | −22% | 0% | −45% | 0 of 3 |
| karpathy-skills | −67% | 60 | 49 to 177 | 0 | −4% | 0% | −55% | 1 of 3 |
| Feature task | ||||||||
| No skill | 699 | 528 to 821 | 5 | 0% | 3 of 3 | |||
| my control file | +14% | 799 | 709 to 884 | 5 | +17% | 0% | +40% | 3 of 3 |
| caveman | −15% | 596 | 549 to 653 | 6 | +21% | 0% | −8% | 3 of 3 |
| ponytail | +6% | 740 | 695 to 790 | 5 | +17% | 0% | +35% | 3 of 3 |
| karpathy-skills | −1% | 692 | 582 to 773 | 5 | +11% | 0% | −7% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 684 | 653 to 711 | 5 | 0% | 3 of 3 | |||
| my control file | 0% | 684 | 662 to 717 | 5 | +3% | 0% | +12% | 3 of 3 |
| caveman | −24% | 520 | 482 to 538 | 2 | +1% | 0% | −17% | 3 of 3 |
| ponytail | +9% | 744 | 710 to 744 | 5 | +13% | 0% | −3% | 3 of 3 |
| karpathy-skills | +7% | 733 | 646 to 910 | 2 | +19% | 0% | −4% | 3 of 3 |
| Build task | ||||||||
| No skill | 1,267 | 1,115 to 1,357 | 56 | 0% | 3 of 3 | |||
| my control file | +5% | 1,336 | 1,099 to 1,371 | 46 | +12% | 0% | −9% | 3 of 3 |
| caveman | −27% | 929 | 863 to 1,215 | 55 | 0% | 0% | −28% | 3 of 3 |
| ponytail | −8% | 1,170 | 1,145 to 1,264 | 50 | +15% | 0% | −11% | 3 of 3 |
| karpathy-skills | −16% | 1,070 | 1,054 to 1,138 | 46 | −11% | 0% | −29% | 3 of 3 |
| Skill | Output vs no skill | Output tokens | Output range | Lines changed | Cost vs no skill | Thinking | Time vs no skill | Passed |
|---|---|---|---|---|---|---|---|---|
| Explain task | ||||||||
| No skill | 286 | 232 to 302 | 0 | 0% | 3 of 3 | |||
| my control file | −21% | 225 | 216 to 226 | 0 | −4% | 0% | −21% | 3 of 3 |
| caveman | −36% | 184 | 173 to 186 | 0 | +2% | 0% | −33% | 3 of 3 |
| ponytail | −26% | 211 | 211 to 223 | 0 | −4% | 0% | −30% | 3 of 3 |
| karpathy-skills | −12% | 253 | 237 to 262 | 0 | −3% | 0% | −21% | 3 of 3 |
| Feature task | ||||||||
| No skill | 669 | 569 to 706 | 8 | 2% | 3 of 3 | |||
| my control file | −16% | 561 | 443 to 634 | 6 | +4% | 0% | −9% | 3 of 3 |
| caveman | −19% | 542 | 398 to 555 | 6 | +8% | 0% | −9% | 3 of 3 |
| ponytail | −8% | 615 | 536 to 658 | 5 | +7% | 4% | −6% | 3 of 3 |
| karpathy-skills | −15% | 571 | 556 to 620 | 5 | +9% | 2% | −6% | 3 of 3 |
| Bug fix task | ||||||||
| No skill | 537 | 498 to 652 | 2 | 7% | 3 of 3 | |||
| my control file | +11% | 597 | 583 to 630 | 5 | +6% | 8% | +8% | 3 of 3 |
| caveman | −12% | 471 | 381 to 486 | 2 | +12% | 4% | −1% | 3 of 3 |
| ponytail | +3% | 555 | 542 to 604 | 2 | +2% | 13% | +4% | 3 of 3 |
| karpathy-skills | +8% | 580 | 561 to 625 | 2 | +4% | 3% | +3% | 3 of 3 |
| Build task | ||||||||
| No skill | 1,026 | 978 to 1,077 | 50 | 2% | 3 of 3 | |||
| my control file | +14% | 1,168 | 1,144 to 1,318 | 66 | +13% | 3% | +9% | 3 of 3 |
| caveman | −9% | 936 | 896 to 1,082 | 57 | +17% | 0% | −4% | 3 of 3 |
| ponytail | +10% | 1,124 | 1,044 to 1,180 | 43 | +9% | 19% | +9% | 3 of 3 |
| karpathy-skills | +2% | 1,042 | 1,035 to 1,060 | 46 | +11% | 8% | +7% | 3 of 3 |
Output tokens include hidden thinking, unlike the charts, which count visible output. Each is the median of 3 runs, and the range runs from the lowest run to the highest. Lines changed are lines added plus lines deleted. Thinking is the share of output that was hidden thinking. Cost is at API prices.
Show every run on this page
Scroll sideways for every column.
| Round | Model | Task | Skill | Run | CLI | Account extras | Result | Output | Thinking | Claude input | Codex CLI input | Cache write | Cache read | Cost | Lines changed | Seconds |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Sonnet 5 | Feature | caveman | 1 | Claude Code 2.1.278 | on | passed | 836 | 20 | 12 | 24,526 | 227,140 | $0.152 | 5 | 12.1 | |
| 1 | Sonnet 5 | Feature | No skill | 1 | Claude Code 2.1.278 | on | passed | 1,145 | 104 | 14 | 22,853 | 255,498 | $0.154 | 5 | 13.4 | |
| 1 | Sonnet 5 | Feature | ponytail | 1 | Claude Code 2.1.278 | on | passed | 1,369 | 76 | 16 | 24,147 | 303,708 | $0.171 | 7 | 15.2 | |
| 1 | Sonnet 5 | Bug fix | caveman | 1 | Claude Code 2.1.278 | on | passed | 1,256 | 57 | 18 | 18,459 | 296,583 | $0.146* | 2 | 16.2 | |
| 1 | Sonnet 5 | Bug fix | No skill | 1 | Claude Code 2.1.278 | on | passed | 1,223 | 332 | 12 | 22,553 | 216,164 | $0.146 | 2 | 22 | |
| 1 | Sonnet 5 | Bug fix | ponytail | 1 | Claude Code 2.1.278 | on | passed | 1,824 | 503 | 16 | 24,556 | 306,856 | $0.178 | 4 | 29.9 | |
| 1 | 5.6 Luna | Feature | caveman | 1 | Codex CLI 0.153.3 | on | passed | 696 | 115 | 85,497 | 20,217 | 65,280 | $0.00618 to $0.00720 | 5 | 22.8 | |
| 1 | 5.6 Luna | Feature | No skill | 1 | Codex CLI 0.153.3 | on | passed | 1,045 | 180 | 86,466 | 27,586 | 58,880 | $0.00795 to $0.00933 | 5 | 39.8 | |
| 1 | 5.6 Luna | Feature | ponytail | 1 | Codex CLI 0.153.3 | on | passed | 985 | 170 | 75,398 | 24,454 | 50,944 | $0.00709 to $0.00831 | 5 | 28.7 | |
| 1 | 5.6 Luna | Bug fix | caveman | 1 | Codex CLI 0.153.3 | on | passed | 1,002 | 216 | 135,817 | 33,417 | 102,400 | $0.00993 to $0.0116 | 2 | 44.9 | |
| 1 | 5.6 Luna | Bug fix | No skill | 1 | Codex CLI 0.153.3 | on | passed | 849 | 131 | 85,267 | 22,291 | 62,976 | $0.00674 to $0.00785 | 2 | 39.7 | |
| 1 | 5.6 Luna | Bug fix | ponytail | 1 | Codex CLI 0.153.3 | on | passed | 1,125 | 211 | 92,307 | 24,211 | 68,096 | $0.00755 to $0.00877 | 2 | 30.9 | |
| 1 | 5.6 Luna | Explain | caveman | 1 | Codex CLI 0.153.3 | on | passed | 232 | 9 | 35,949 | 14,957 | 20,992 | $0.00369 to $0.00444 | 0 | 8.2 | |
| 1 | 5.6 Luna | Explain | No skill | 1 | Codex CLI 0.153.3 | on | passed | 247 | 18 | 32,596 | 12,628 | 19,968 | $0.00322 to $0.00385 | 0 | 9.1 | |
| 1 | 5.6 Luna | Explain | ponytail | 1 | Codex CLI 0.153.3 | on | passed | 279 | 14 | 33,818 | 15,898 | 17,920 | $0.00387 to $0.00467 | 0 | 10.3 | |
| 1‡ | Sonnet 5 | Explain | caveman | 1 | Claude Code 2.1.278 | on | passed | 294 | 0 | 4 | 22,769 | 59,226 | $0.106 | 0 | 7 | |
| 1‡ | Sonnet 5 | Explain | No skill | 1 | Claude Code 2.1.278 | on | passed | 985 | 16 | 6 | 20,272 | 94,669 | $0.110 | 0 | 14.4 | |
| 1‡ | Sonnet 5 | Explain | ponytail | 1 | Claude Code 2.1.278 | on | passed | 537 | 0 | 4 | 21,145 | 57,603 | $0.101 | 0 | 12.9 | |
| 2 | Sonnet 5 | Feature | caveman | 2 | Claude Code 2.1.278 | on | passed | 1,156 | 44 | 14 | 25,169 | 271,861 | $0.167 | 5 | 15.1 | |
| 2 | Sonnet 5 | Feature | caveman | 3 | Claude Code 2.1.278 | on | passed | 1,026 | 28 | 16 | 27,447 | 323,210 | $0.185 | 5 | 17.1 | |
| 2 | Sonnet 5 | Feature | No skill | 2 | Claude Code 2.1.278 | on | passed | 1,047 | 83 | 14 | 22,241 | 256,134 | $0.151 | 5 | 14.5 | |
| 2 | Sonnet 5 | Feature | No skill | 3 | Claude Code 2.1.278 | on | passed | 1,197 | 301 | 16 | 23,585 | 299,504 | $0.166 | 5 | 24.4 | |
| 2 | Sonnet 5 | Bug fix | caveman | 2 | Claude Code 2.1.278 | on | passed | 1,337 | 242 | 14 | 28,234 | 284,612 | $0.183 | 2 | 25.5 | |
| 2 | Sonnet 5 | Bug fix | caveman | 3 | Claude Code 2.1.278 | on | passed | 816 | 34 | 14 | 24,407 | 269,459 | $0.160 | 2 | 14.4 | |
| 2 | Sonnet 5 | Bug fix | No skill | 2 | Claude Code 2.1.278 | on | passed | 1,111 | 120 | 16 | 22,279 | 292,902 | $0.159 | 2 | 14.3 | |
| 2 | Sonnet 5 | Bug fix | No skill | 3 | Claude Code 2.1.278 | on | passed | 1,277 | 136 | 16 | 23,166 | 299,967 | $0.165 | 2 | 24.1 | |
| 2 | 5.6 Luna | Feature | caveman | 2 | Codex CLI 0.153.3 | on | passed | 958 | 159 | 119,713 | 29,345 | 90,368 | $0.00883 to $0.0103 | 7 | 37.1 | |
| 2 | 5.6 Luna | Feature | caveman | 3 | Codex CLI 0.153.3 | on | passed | 1,025 | 197 | 111,045 | 19,653 | 91,392 | $0.00699 to $0.00797 | 5 | 35.1 | |
| 2 | 5.6 Luna | Feature | No skill | 2 | Codex CLI 0.153.3 | on | passed | 1,031 | 136 | 99,202 | 23,170 | 76,032 | $0.00739 to $0.00855 | 5 | 29.3 | |
| 2 | 5.6 Luna | Feature | No skill | 3 | Codex CLI 0.153.3 | on | passed | 1,577 | 295 | 149,421 | 23,725 | 125,696 | $0.00915 to $0.0103 | 7 | 40.4 | |
| 2 | 5.6 Luna | Bug fix | caveman | 2 | Codex CLI 0.153.3 | on | passed | 870 | 84 | 132,376 | 22,808 | 109,568 | $0.00780 to $0.00894 | 2 | 29 | |
| 2 | 5.6 Luna | Bug fix | caveman | 3 | Codex CLI 0.153.3 | on | passed | 725 | 134 | 95,028 | 18,740 | 76,288 | $0.00614 to $0.00708 | 1 | 31.7 | |
| 2 | 5.6 Luna | Bug fix | No skill | 2 | Codex CLI 0.153.3 | on | passed | 1,123 | 158 | 114,379 | 25,291 | 89,088 | $0.00819 to $0.00945 | 2 | 31.4 | |
| 2 | 5.6 Luna | Bug fix | No skill | 3 | Codex CLI 0.153.3 | on | passed | 779 | 148 | 85,527 | 23,575 | 61,952 | $0.00689 to $0.00807 | 2 | 38.5 | |
| 2 | 5.6 Luna | Explain | caveman | 2 | Codex CLI 0.153.3 | on | passed | 217 | 20 | 35,750 | 17,830 | 17,920 | $0.00419 to $0.00508 | 0 | 11.8 | |
| 2 | 5.6 Luna | Explain | caveman | 3 | Codex CLI 0.153.3 | on | passed | 246 | 8 | 35,949 | 14,957 | 20,992 | $0.00371 to $0.00445 | 0 | 10.1 | |
| 2 | 5.6 Luna | Explain | No skill | 2 | Codex CLI 0.153.3 | on | passed | 234 | 13 | 32,578 | 12,610 | 19,968 | $0.00320 to $0.00383 | 0 | 8.8 | |
| 2 | 5.6 Luna | Explain | No skill | 3 | Codex CLI 0.153.3 | on | passed | 240 | 14 | 32,584 | 12,616 | 19,968 | $0.00321 to $0.00384 | 0 | 9.5 | |
| 2‡ | Sonnet 5 | Explain | caveman | 2 | Claude Code 2.1.278 | on | passed | 375 | 6 | 6 | 22,981 | 100,019 | $0.116 | 0 | 12.3 | |
| 2‡ | Sonnet 5 | Explain | caveman | 3 | Claude Code 2.1.278 | on | passed | 289 | 12 | 4 | 22,779 | 59,223 | $0.106 | 0 | 5.9 | |
| 2‡ | Sonnet 5 | Explain | No skill | 2 | Claude Code 2.1.278 | on | passed | 839 | 37 | 6 | 20,305 | 94,702 | $0.109 | 0 | 11.3 | |
| 2‡ | Sonnet 5 | Explain | No skill | 3 | Claude Code 2.1.278 | on | passed | 891 | 16 | 6 | 20,280 | 94,672 | $0.109 | 0 | 11 | |
| 3 | Sonnet 5 | Feature | karpathy-skills | 1 | Claude Code 2.1.278 | on | passed | 1,117 | 30 | 14 | 17,535 | 218,950 | $0.125* | 5 | 13.7 | |
| 3 | Sonnet 5 | Feature | karpathy-skills | 2 | Claude Code 2.1.278 | on | passed | 1,279 | 103 | 16 | 26,005 | 311,651 | $0.179 | 5 | 17.9 | |
| 3 | Sonnet 5 | Feature | karpathy-skills | 3 | Claude Code 2.1.278 | on | passed | 1,197 | 42 | 14 | 24,492 | 267,110 | $0.163 | 5 | 15.4 | |
| 3 | Sonnet 5 | Feature | ponytail | 2 | Claude Code 2.1.278 | on | passed | 1,483 | 169 | 18 | 24,277 | 343,739 | $0.181 | 7 | 16.6 | |
| 3 | Sonnet 5 | Feature | ponytail | 3 | Claude Code 2.1.278 | on | passed | 948 | 96 | 12 | 24,050 | 222,153 | $0.150 | 5 | 15.1 | |
| 3 | Sonnet 5 | Bug fix | karpathy-skills | 1 | Claude Code 2.1.278 | on | passed | 1,317 | 269 | 18 | 24,956 | 346,230 | $0.182 | 2 | 26.4 | |
| 3 | Sonnet 5 | Bug fix | karpathy-skills | 2 | Claude Code 2.1.278 | on | passed | 1,791 | 587 | 16 | 25,938 | 309,438 | $0.184 | 2 | 21.6 | |
| 3 | Sonnet 5 | Bug fix | karpathy-skills | 3 | Claude Code 2.1.278 | on | passed | 1,666 | 485 | 14 | 25,219 | 265,256 | $0.171 | 2 | 17.5 | |
| 3 | Sonnet 5 | Bug fix | ponytail | 2 | Claude Code 2.1.278 | on | passed | 1,718 | 446 | 16 | 24,956 | 308,564 | $0.179 | 4 | 20.5 | |
| 3 | Sonnet 5 | Bug fix | ponytail | 3 | Claude Code 2.1.278 | on | passed | 1,552 | 415 | 18 | 17,695 | 286,548 | $0.144* | 4 | 19.3 | |
| 3 | Sonnet 5 | Build | caveman | 1 | Claude Code 2.1.278 | on | passed | 3,191 | 1,573 | 14 | 30,429 | 290,224 | $0.212 | 59 | 41.6 | |
| 3 | Sonnet 5 | Build | caveman | 2 | Claude Code 2.1.278 | on | passed | 3,258 | 1,734 | 14 | 30,529 | 290,465 | $0.213 | 50 | 32.4 | |
| 3 | Sonnet 5 | Build | caveman | 3 | Claude Code 2.1.278 | on | passed | 3,124 | 1,379 | 18 | 22,550 | 315,519 | $0.185* | 57 | 28.4 | |
| 3 | Sonnet 5 | Build | No skill | 1 | Claude Code 2.1.278 | on | passed | 3,822 | 2,066 | 14 | 27,290 | 270,318 | $0.201 | 65 | 34.7 | |
| 3 | Sonnet 5 | Build | No skill | 2 | Claude Code 2.1.278 | on | passed | 3,891 | 1,938 | 16 | 27,975 | 314,305 | $0.214 | 59 | 35.3 | |
| 3 | Sonnet 5 | Build | No skill | 3 | Claude Code 2.1.278 | on | passed | 3,161 | 1,560 | 12 | 24,810 | 221,888 | $0.175 | 62 | 28.4 | |
| 3 | Sonnet 5 | Build | karpathy-skills | 1 | Claude Code 2.1.278 | on | passed | 3,636 | 1,971 | 12 | 27,773 | 231,205 | $0.194 | 56 | 39.5 | |
| 3 | Sonnet 5 | Build | karpathy-skills | 2 | Claude Code 2.1.278 | on | passed | 4,310 | 2,510 | 14 | 20,316 | 224,947 | $0.169* | 57 | 37.3 | |
| 3 | Sonnet 5 | Build | karpathy-skills | 3 | Claude Code 2.1.278 | on | passed | 3,566 | 1,632 | 18 | 29,444 | 368,655 | $0.227 | 55 | 32.9 | |
| 3 | Sonnet 5 | Build | ponytail | 1 | Claude Code 2.1.278 | on | passed | 3,761 | 1,933 | 16 | 29,940 | 326,607 | $0.223 | 48 | 38.4 | |
| 3 | Sonnet 5 | Build | ponytail | 2 | Claude Code 2.1.278 | on | passed | 3,267 | 1,097 | 20 | 29,397 | 413,165 | $0.233 | 47 | 34.9 | |
| 3 | Sonnet 5 | Build | ponytail | 3 | Claude Code 2.1.278 | on | passed | 4,661 | 2,799 | 18 | 29,653 | 371,077 | $0.239 | 41 | 45.2 | |
| 3 | 5.6 Luna | Feature | karpathy-skills | 1 | Codex CLI 0.153.3 | on | passed | 1,285 | 225 | 105,254 | 23,078 | 82,176 | $0.00780 to $0.00895 | 5 | 32.6 | |
| 3 | 5.6 Luna | Feature | karpathy-skills | 2 | Codex CLI 0.153.3 | on | passed | 1,199 | 203 | 90,841 | 20,697 | 70,144 | $0.00698 to $0.00802 | 5 | 32.2 | |
| 3 | 5.6 Luna | Feature | karpathy-skills | 3 | Codex CLI 0.153.3 | on | passed | 986 | 115 | 90,509 | 22,413 | 68,096 | $0.00703 to $0.00815 | 5 | 28 | |
| 3 | 5.6 Luna | Feature | ponytail | 2 | Codex CLI 0.153.3 | on | passed | 1,135 | 193 | 90,307 | 18,115 | 72,192 | $0.00643 to $0.00734 | 5 | 34.2 | |
| 3 | 5.6 Luna | Feature | ponytail | 3 | Codex CLI 0.153.3 | on | passed | 1,009 | 159 | 118,941 | 26,781 | 92,160 | $0.00841 to $0.00975 | 5 | 36.2 | |
| 3 | 5.6 Luna | Bug fix | karpathy-skills | 1 | Codex CLI 0.153.3 | on | passed | 1,391 | 256 | 133,598 | 24,286 | 109,312 | $0.00871 to $0.00993 | 2 | 44.9 | |
| 3 | 5.6 Luna | Bug fix | karpathy-skills | 2 | Codex CLI 0.153.3 | on | passed | 1,360 | 186 | 104,225 | 26,145 | 78,080 | $0.00842 to $0.00973 | 2 | 49.8 | |
| 3 | 5.6 Luna | Bug fix | karpathy-skills | 3 | Codex CLI 0.153.3 | on | passed | 958 | 191 | 75,174 | 17,062 | 58,112 | $0.00572 to $0.00658 | 2 | 27.1 | |
| 3 | 5.6 Luna | Bug fix | ponytail | 2 | Codex CLI 0.153.3 | on | passed | 1,009 | 252 | 89,331 | 22,259 | 67,072 | $0.00700 to $0.00812 | 2 | 40.1 | |
| 3 | 5.6 Luna | Bug fix | ponytail | 3 | Codex CLI 0.153.3 | on | passed | 1,004 | 157 | 89,326 | 24,302 | 65,024 | $0.00737 to $0.00858 | 2 | 94 | |
| 3 | 5.6 Luna | Explain | karpathy-skills | 1 | Codex CLI 0.153.3 | on | passed | 262 | 14 | 33,715 | 15,795 | 17,920 | $0.00383 to $0.00462 | 0 | 13.4 | |
| 3 | 5.6 Luna | Explain | karpathy-skills | 2 | Codex CLI 0.153.3 | on | passed | 250 | 14 | 33,717 | 15,797 | 17,920 | $0.00382 to $0.00461 | 0 | 13.8 | |
| 3 | 5.6 Luna | Explain | karpathy-skills | 3 | Codex CLI 0.153.3 | on | passed | 270 | 15 | 33,725 | 15,805 | 17,920 | $0.00384 to $0.00463 | 0 | 13.1 | |
| 3 | 5.6 Luna | Explain | ponytail | 2 | Codex CLI 0.153.3 | on | passed | 254 | 8 | 33,826 | 15,906 | 17,920 | $0.00384 to $0.00464 | 0 | 8.4 | |
| 3 | 5.6 Luna | Explain | ponytail | 3 | Codex CLI 0.153.3 | on | passed | 290 | 13 | 33,819 | 13,851 | 19,968 | $0.00352 to $0.00421 | 0 | 10.8 | |
| 3 | 5.6 Luna | Build | caveman | 1 | Codex CLI 0.153.3 | on | passed | 1,691 | 220 | 169,358 | 37,518 | 131,840 | $0.0122 to $0.0140 | 48 | 59.2 | |
| 3 | 5.6 Luna | Build | caveman | 2 | Codex CLI 0.153.3 | on | passed | 2,072 | 345 | 169,484 | 38,668 | 130,816 | $0.0128 to $0.0148 | 66 | 66.4 | |
| 3 | 5.6 Luna | Build | caveman | 3 | Codex CLI 0.153.3 | on | passed | 1,336 | 207 | 98,709 | 26,517 | 72,192 | $0.00835 to $0.00968 | 50 | 35.7 | |
| 3 | 5.6 Luna | Build | No skill | 1 | Codex CLI 0.153.3 | on | passed | 1,567 | 169 | 104,663 | 29,655 | 75,008 | $0.00931 to $0.0108 | 51 | 48.7 | |
| 3 | 5.6 Luna | Build | No skill | 2 | Codex CLI 0.153.3 | on | passed | 1,858 | 289 | 134,213 | 33,093 | 101,120 | $0.0109 to $0.0125 | 50 | 58.7 | |
| 3 | 5.6 Luna | Build | No skill | 3 | Codex CLI 0.153.3 | on | passed | 2,343 | 403 | 152,446 | 36,990 | 115,456 | $0.0125 to $0.0144 | 51 | 56 | |
| 3 | 5.6 Luna | Build | karpathy-skills | 1 | Codex CLI 0.153.3 | on | passed | 2,003 | 373 | 153,035 | 30,667 | 122,368 | $0.0110 to $0.0125 | 39 | 65.7 | |
| 3 | 5.6 Luna | Build | karpathy-skills | 2 | Codex CLI 0.153.3 | on | passed | 1,740 | 372 | 123,047 | 32,935 | 90,112 | $0.0105 to $0.0121 | 47 | 43.2 | |
| 3 | 5.6 Luna | Build | karpathy-skills | 3 | Codex CLI 0.153.3 | on | passed | 1,753 | 373 | 119,911 | 21,607 | 98,304 | $0.00839 to $0.00947 | 45 | 45.5 | |
| 3 | 5.6 Luna | Build | ponytail | 1 | Codex CLI 0.153.3 | on | passed | 1,772 | 298 | 142,121 | 26,665 | 115,456 | $0.00977 to $0.0111 | 44 | 224.9 | |
| 3 | 5.6 Luna | Build | ponytail | 2 | Codex CLI 0.153.3 | on | passed | 1,868 | 362 | 124,084 | 25,780 | 98,304 | $0.00936 to $0.0107 | 44 | 63.5 | |
| 3 | 5.6 Luna | Build | ponytail | 3 | Codex CLI 0.153.3 | on | passed | 2,091 | 423 | 141,265 | 29,905 | 111,360 | $0.0107 to $0.0122 | 43 | 71.3 | |
| 3‡ | Sonnet 5 | Explain | karpathy-skills | 1 | Claude Code 2.1.278 | on | passed | 877 | 53 | 4 | 21,146 | 57,587 | $0.105 | 0 | 10.8 | |
| 3‡ | Sonnet 5 | Explain | karpathy-skills | 2 | Claude Code 2.1.278 | on | passed | 635 | 16 | 6 | 21,360 | 96,763 | $0.111 | 0 | 8.5 | |
| 3‡ | Sonnet 5 | Explain | karpathy-skills | 3 | Claude Code 2.1.278 | on | passed | 911 | 16 | 6 | 21,355 | 96,757 | $0.114 | 0 | 11 | |
| 3‡ | Sonnet 5 | Explain | ponytail | 2 | Claude Code 2.1.278 | on | passed | 737 | 46 | 4 | 12,250 | 47,468 | $0.0659* | 0 | 8.2 | |
| 3‡ | Sonnet 5 | Explain | ponytail | 3 | Claude Code 2.1.278 | on | passed | 743 | 21 | 4 | 21,143 | 57,579 | $0.104 | 0 | 16.6 | |
| 4 | Sonnet 5 | Feature | my control file | 1 | Claude Code 2.1.278 | on | passed | 1,396 | 97 | 14 | 25,393 | 268,183 | $0.169 | 6 | 16.6 | |
| 4 | Sonnet 5 | Feature | my control file | 2 | Claude Code 2.1.278 | on | passed | 1,540 | 121 | 18 | 24,560 | 347,439 | $0.183 | 8 | 18.6 | |
| 4 | Sonnet 5 | Feature | my control file | 3 | Claude Code 2.1.278 | on | passed | 1,187 | 12 | 14 | 24,229 | 264,296 | $0.162 | 5 | 17.5 | |
| 4 | Sonnet 5 | Bug fix | my control file | 1 | Claude Code 2.1.278 | on | passed | 2,050 | 967 | 12 | 24,100 | 221,012 | $0.161 | 2 | 22.1 | |
| 4 | Sonnet 5 | Bug fix | my control file | 2 | Claude Code 2.1.278 | on | passed | 1,596 | 527 | 14 | 24,947 | 262,873 | $0.168 | 2 | 17.4 | |
| 4 | Sonnet 5 | Bug fix | my control file | 3 | Claude Code 2.1.278 | on | passed | 2,054 | 735 | 14 | 25,437 | 263,772 | $0.175 | 2 | 22.9 | |
| 4 | Sonnet 5 | Build | my control file | 1 | Claude Code 2.1.278 | on | passed | 3,077 | 671 | 18 | 21,960 | 309,811 | $0.181* | 71 | 30.3 | |
| 4 | Sonnet 5 | Build | my control file | 2 | Claude Code 2.1.278 | on | passed | 5,249 | 3,016 | 18 | 30,955 | 374,415 | $0.251 | 63 | 48.4 | |
| 4 | Sonnet 5 | Build | my control file | 3 | Claude Code 2.1.278 | on | passed | 4,895 | 3,068 | 12 | 28,500 | 231,568 | $0.209 | 63 | 54.2 | |
| 4 | 5.6 Luna | Feature | my control file | 1 | Codex CLI 0.153.3 | on | passed | 1,087 | 145 | 88,344 | 23,320 | 65,024 | $0.00727 to $0.00843 | 7 | 39.9 | |
| 4 | 5.6 Luna | Feature | my control file | 2 | Codex CLI 0.153.3 | on | passed | 1,146 | 180 | 89,689 | 22,617 | 67,072 | $0.00724 to $0.00837 | 6 | 40.4 | |
| 4 | 5.6 Luna | Feature | my control file | 3 | Codex CLI 0.153.3 | on | passed | 834 | 108 | 75,687 | 20,647 | 55,040 | $0.00623 to $0.00726 | 6 | 25.1 | |
| 4 | 5.6 Luna | Bug fix | my control file | 1 | Codex CLI 0.153.3 | on | passed | 1,432 | 318 | 132,125 | 28,957 | 103,168 | $0.00957 to $0.0110 | 2 | 53.7 | |
| 4 | 5.6 Luna | Bug fix | my control file | 2 | Codex CLI 0.153.3 | on | passed | 847 | 139 | 88,279 | 26,327 | 61,952 | $0.00752 to $0.00884 | 2 | 27.4 | |
| 4 | 5.6 Luna | Bug fix | my control file | 3 | Codex CLI 0.153.3 | on | passed | 1,265 | 300 | 118,831 | 20,527 | 98,304 | $0.00759 to $0.00861 | 4 | 38.9 | |
| 4 | 5.6 Luna | Explain | my control file | 1 | Codex CLI 0.153.3 | on | passed | 252 | 15 | 33,526 | 15,606 | 17,920 | $0.00378 to $0.00456 | 0 | 11.3 | |
| 4 | 5.6 Luna | Explain | my control file | 2 | Codex CLI 0.153.3 | on | passed | 237 | 17 | 33,527 | 13,559 | 19,968 | $0.00340 to $0.00407 | 0 | 13.3 | |
| 4 | 5.6 Luna | Explain | my control file | 3 | Codex CLI 0.153.3 | on | passed | 232 | 13 | 33,515 | 15,595 | 17,920 | $0.00376 to $0.00454 | 0 | 16.5 | |
| 4 | 5.6 Luna | Build | my control file | 1 | Codex CLI 0.153.3 | on | passed | 1,569 | 226 | 107,377 | 22,129 | 85,248 | $0.00801 to $0.00912 | 51 | 39.1 | |
| 4 | 5.6 Luna | Build | my control file | 2 | Codex CLI 0.153.3 | on | passed | 2,031 | 389 | 134,960 | 22,576 | 112,384 | $0.00920 to $0.0103 | 72 | 49.5 | |
| 4 | 5.6 Luna | Build | my control file | 3 | Codex CLI 0.153.3 | on | passed | 2,060 | 353 | 136,494 | 28,206 | 108,288 | $0.0103 to $0.0117 | 51 | 72.8 | |
| 4‡ | Sonnet 5 | Explain | my control file | 1 | Claude Code 2.1.278 | on | passed | 956 | 20 | 6 | 21,083 | 96,229 | $0.113 | 0 | 12.8 | |
| 4‡ | Sonnet 5 | Explain | my control file | 2 | Claude Code 2.1.278 | on | passed | 731 | 0 | 6 | 21,092 | 96,244 | $0.111 | 0 | 9.7 | |
| 4‡ | Sonnet 5 | Explain | my control file | 3 | Claude Code 2.1.278 | on | passed | 835 | 0 | 6 | 21,084 | 96,232 | $0.112 | 0 | 10.6 | |
| 5 | Sonnet 5 | Explain | caveman | 4 | Claude Code 2.1.278 | on | passed | 292 | 0 | 4 | 22,752 | 59,211 | $0.106 | 0 | 5.5 | |
| 5 | Sonnet 5 | Explain | caveman | 5 | Claude Code 2.1.278 | on | passed | 291 | 12 | 4 | 12,981 | 49,087 | $0.0647* | 0 | 4.5 | |
| 5 | Sonnet 5 | Explain | caveman | 6 | Claude Code 2.1.278 | on | passed | 324 | 0 | 4 | 22,744 | 59,205 | $0.106 | 0 | 7.6 | |
| 5 | Sonnet 5 | Explain | No skill | 4 | Claude Code 2.1.278 | on | passed | 858 | 25 | 6 | 20,294 | 94,681 | $0.109 | 0 | 10.2 | |
| 5 | Sonnet 5 | Explain | No skill | 5 | Claude Code 2.1.278 | on | passed | 793 | 30 | 6 | 20,331 | 94,719 | $0.108 | 0 | 10.2 | |
| 5 | Sonnet 5 | Explain | No skill | 6 | Claude Code 2.1.278 | on | passed | 793 | 37 | 6 | 20,288 | 94,676 | $0.108 | 0 | 11.7 | |
| 5 | 5.6 Luna | Explain | caveman | 4 | Codex CLI 0.153.3 | on | passed | 218 | 0 | 35,938 | 14,946 | 20,992 | $0.00367 to $0.00442 | 0 | 10.7 | |
| 5 | 5.6 Luna | Explain | caveman | 5 | Codex CLI 0.153.3 | on | passed | 230 | 22 | 35,962 | 14,970 | 20,992 | $0.00369 to $0.00444 | 0 | 11.8 | |
| 5 | 5.6 Luna | Explain | caveman | 6 | Codex CLI 0.153.3 | on | passed | 231 | 9 | 35,949 | 14,957 | 20,992 | $0.00369 to $0.00444 | 0 | 28.2 | |
| 5 | 5.6 Luna | Explain | No skill | 4 | Codex CLI 0.153.3 | on | passed | 253 | 14 | 32,413 | 13,469 | 18,944 | $0.00338 to $0.00405 | 0 | 12.6 | |
| 5 | 5.6 Luna | Explain | No skill | 5 | Codex CLI 0.153.3 | on | passed | 217 | 9 | 32,573 | 14,653 | 17,920 | $0.00355 to $0.00428 | 0 | 14.2 | |
| 5 | 5.6 Luna | Explain | No skill | 6 | Codex CLI 0.153.3 | on | passed | 232 | 16 | 32,591 | 14,671 | 17,920 | $0.00357 to $0.00430 | 0 | 8.1 | |
| 6 | 6 Luna | Explain | caveman | 1 | Codex CLI 0.156.0 | on | passed | 131 | 0 | 42,740 | 16,628 | 26,112 | $0.00199 to $0.00241 | 0 | 10.8 | |
| 6 | 6 Luna | Explain | caveman | 2 | Codex CLI 0.156.0 | on | passed | 107 | 0 | 42,702 | 16,590 | 26,112 | $0.00197 to $0.00239 | 0 | 7.9 | |
| 6 | 6 Luna | Explain | caveman | 3 | Codex CLI 0.156.0 | on | passed | 119 | 0 | 42,726 | 14,566 | 28,160 | $0.00180 to $0.00216 | 0 | 6.8 | |
| 6 | 6 Luna | Explain | No skill | 1 | Codex CLI 0.156.0 | on | passed | 180 | 0 | 39,360 | 15,296 | 24,064 | $0.00186 to $0.00224 | 0 | 8.6 | |
| 6 | 6 Luna | Explain | No skill | 2 | Codex CLI 0.156.0 | on | passed | 181 | 0 | 39,358 | 15,294 | 24,064 | $0.00186 to $0.00224 | 0 | 9.2 | |
| 6 | 6 Luna | Explain | No skill | 3 | Codex CLI 0.156.0 | on | failed | 47 | 0 | 25,315 | 14,307 | 11,008 | $0.00156 to $0.00192 | 0 | 3.7 | |
| 6‡ | Opus 5.5 | Explain | caveman | 1 | Claude Code 2.1.280 | on | passed | 1,200 | 298 | 4 | 19,603 | 39,035 | $0.189 | 0 | 15.3 | |
| 6‡ | Opus 5.5 | Explain | caveman | 2 | Claude Code 2.1.280 | on | passed | 1,364 | 412 | 4 | 10,847 | 28,329 | $0.120* | 0 | 14.4 | |
| 6‡ | Opus 5.5 | Explain | caveman | 3 | Claude Code 2.1.280 | on | passed | 1,384 | 445 | 4 | 19,632 | 39,064 | $0.193 | 0 | 17 | |
| 6‡ | Opus 5.5 | Explain | No skill | 1 | Claude Code 2.1.280 | on | passed | 1,666 | 463 | 4 | 10,297 | 25,776 | $0.121* | 0 | 16.9 | |
| 6‡ | Opus 5.5 | Explain | No skill | 2 | Claude Code 2.1.280 | on | passed | 1,645 | 269 | 4 | 16,950 | 36,384 | $0.176 | 0 | 18.5 | |
| 6‡ | Opus 5.5 | Explain | No skill | 3 | Claude Code 2.1.280 | on | passed | 1,576 | 407 | 4 | 16,952 | 36,386 | $0.174 | 0 | 17.2 | |
| 7 | Opus 5.5 | Feature | caveman | 1 | Claude Code 2.1.280 | on | passed | 803 | 35 | 8 | 15,201 | 73,550 | $0.152* | 6 | 11.6 | |
| 7 | Opus 5.5 | Feature | caveman | 2 | Claude Code 2.1.280 | on | passed | 619 | 20 | 8 | 20,663 | 98,736 | $0.197 | 6 | 9.3 | |
| 7 | Opus 5.5 | Feature | caveman | 3 | Claude Code 2.1.280 | on | passed | 570 | 0 | 8 | 23,294 | 101,380 | $0.218 | 6 | 9.3 | |
| 7 | Opus 5.5 | Feature | No skill | 1 | Claude Code 2.1.280 | on | passed | 574 | 0 | 8 | 17,986 | 90,745 | $0.174 | 6 | 8.7 | |
| 7 | Opus 5.5 | Feature | No skill | 2 | Claude Code 2.1.280 | on | passed | 693 | 0 | 8 | 18,212 | 90,964 | $0.178 | 6 | 11.1 | |
| 7 | Opus 5.5 | Feature | No skill | 3 | Claude Code 2.1.280 | on | passed | 652 | 0 | 8 | 18,121 | 90,798 | $0.176 | 6 | 8.8 | |
| 7 | Opus 5.5 | Feature | karpathy-skills | 1 | Claude Code 2.1.280 | on | passed | 751 | 51 | 10 | 19,294 | 123,075 | $0.194 | 6 | 13.5 | |
| 7 | Opus 5.5 | Feature | karpathy-skills | 2 | Claude Code 2.1.280 | on | passed | 772 | 28 | 8 | 19,332 | 93,857 | $0.189 | 6 | 11.3 | |
| 7 | Opus 5.5 | Feature | karpathy-skills | 3 | Claude Code 2.1.280 | on | passed | 786 | 22 | 8 | 19,280 | 93,812 | $0.189 | 6 | 9.9 | |
| 7 | Opus 5.5 | Feature | my control file | 1 | Claude Code 2.1.280 | on | passed | 896 | 31 | 10 | 19,763 | 123,492 | $0.201 | 6 | 13.1 | |
| 7 | Opus 5.5 | Feature | my control file | 2 | Claude Code 2.1.280 | on | passed | 956 | 39 | 10 | 20,106 | 123,135 | $0.205 | 8 | 13.9 | |
| 7 | Opus 5.5 | Feature | my control file | 3 | Claude Code 2.1.280 | on | passed | 933 | 38 | 8 | 20,201 | 94,203 | $0.199 | 8 | 11.2 | |
| 7 | Opus 5.5 | Feature | ponytail | 1 | Claude Code 2.1.280 | on | passed | 689 | 0 | 8 | 19,151 | 93,994 | $0.186 | 6 | 9.1 | |
| 7 | Opus 5.5 | Feature | ponytail | 2 | Claude Code 2.1.280 | on | passed | 616 | 33 | 8 | 19,075 | 93,893 | $0.184 | 6 | 8.8 | |
| 7 | Opus 5.5 | Feature | ponytail | 3 | Claude Code 2.1.280 | on | passed | 651 | 23 | 8 | 19,119 | 93,868 | $0.185 | 6 | 11.1 | |
| 7 | Opus 5.5 | Bug fix | caveman | 1 | Claude Code 2.1.280 | on | passed | 994 | 72 | 10 | 23,906 | 134,611 | $0.238 | 7 | 12.1 | |
| 7 | Opus 5.5 | Bug fix | caveman | 2 | Claude Code 2.1.280 | on | passed | 1,041 | 106 | 10 | 23,667 | 134,372 | $0.237 | 7 | 12.5 | |
| 7 | Opus 5.5 | Bug fix | caveman | 3 | Claude Code 2.1.280 | on | passed | 993 | 69 | 10 | 23,874 | 134,592 | $0.238 | 7 | 14.5 | |
| 7 | Opus 5.5 | Bug fix | No skill | 1 | Claude Code 2.1.280 | on | passed | 1,109 | 97 | 8 | 18,563 | 91,022 | $0.189 | 12 | 12.5 | |
| 7 | Opus 5.5 | Bug fix | No skill | 2 | Claude Code 2.1.280 | on | passed | 1,139 | 148 | 8 | 18,635 | 91,003 | $0.190 | 13 | 12.5 | |
| 7 | Opus 5.5 | Bug fix | No skill | 3 | Claude Code 2.1.280 | on | passed | 1,247 | 99 | 10 | 18,729 | 118,876 | $0.199 | 7 | 16.1 | |
| 7 | Opus 5.5 | Bug fix | karpathy-skills | 1 | Claude Code 2.1.280 | on | passed | 1,053 | 178 | 10 | 20,559 | 123,715 | $0.210 | 2 | 17.8 | |
| 7 | Opus 5.5 | Bug fix | karpathy-skills | 2 | Claude Code 2.1.280 | on | passed | 1,045 | 126 | 8 | 19,158 | 93,678 | $0.193 | 2 | 12.4 | |
| 7 | Opus 5.5 | Bug fix | karpathy-skills | 3 | Claude Code 2.1.280 | on | passed | 903 | 128 | 8 | 19,171 | 93,599 | $0.190 | 2 | 13 | |
| 7 | Opus 5.5 | Bug fix | my control file | 1 | Claude Code 2.1.280 | on | passed | 1,252 | 235 | 8 | 21,020 | 95,884 | $0.212 | 7 | 20.4 | |
| 7 | Opus 5.5 | Bug fix | my control file | 2 | Claude Code 2.1.280 | on | passed | 1,630 | 292 | 10 | 20,384 | 123,571 | $0.220 | 17 | 16.6 | |
| 7 | Opus 5.5 | Bug fix | my control file | 3 | Claude Code 2.1.280 | on | passed | 1,333 | 115 | 10 | 20,173 | 123,187 | $0.213 | 7 | 14.8 | |
| 7 | Opus 5.5 | Bug fix | ponytail | 1 | Claude Code 2.1.280 | on | passed | 1,126 | 235 | 8 | 19,287 | 93,705 | $0.196 | 4 | 14.6 | |
| 7 | Opus 5.5 | Bug fix | ponytail | 2 | Claude Code 2.1.280 | on | passed | 1,241 | 357 | 10 | 20,019 | 123,102 | $0.210 | 4 | 16.6 | |
| 7 | Opus 5.5 | Bug fix | ponytail | 3 | Claude Code 2.1.280 | on | passed | 1,417 | 393 | 10 | 20,865 | 124,138 | $0.220 | 4 | 17.2 | |
| 7 | Opus 5.5 | Build | caveman | 1 | Claude Code 2.1.280 | on | passed | 1,617 | 167 | 10 | 25,525 | 138,121 | $0.264 | 56 | 16.4 | |
| 7 | Opus 5.5 | Build | caveman | 2 | Claude Code 2.1.280 | on | passed | 1,601 | 101 | 10 | 25,479 | 138,052 | $0.264 | 56 | 18 | |
| 7 | Opus 5.5 | Build | caveman | 3 | Claude Code 2.1.280 | on | passed | 1,534 | 116 | 8 | 25,473 | 102,601 | $0.255 | 58 | 16.9 | |
| 7 | Opus 5.5 | Build | No skill | 1 | Claude Code 2.1.280 | on | passed | 1,681 | 234 | 8 | 20,245 | 92,040 | $0.214 | 54 | 18.1 | |
| 7 | Opus 5.5 | Build | No skill | 2 | Claude Code 2.1.280 | on | passed | 1,496 | 98 | 8 | 20,067 | 92,043 | $0.209 | 53 | 14.9 | |
| 7 | Opus 5.5 | Build | No skill | 3 | Claude Code 2.1.280 | on | passed | 1,659 | 83 | 10 | 19,452 | 121,519 | $0.213 | 56 | 17.8 | |
| 7 | Opus 5.5 | Build | karpathy-skills | 1 | Claude Code 2.1.280 | on | passed | 1,539 | 169 | 8 | 21,977 | 95,960 | $0.226 | 41 | 15.4 | |
| 7 | Opus 5.5 | Build | karpathy-skills | 2 | Claude Code 2.1.280 | on | passed | 1,554 | 218 | 8 | 22,016 | 96,021 | $0.226 | 43 | 16 | |
| 7 | Opus 5.5 | Build | karpathy-skills | 3 | Claude Code 2.1.280 | on | passed | 1,639 | 242 | 10 | 21,989 | 127,916 | $0.234 | 40 | 18.8 | |
| 7 | Opus 5.5 | Build | my control file | 1 | Claude Code 2.1.280 | on | passed | 2,048 | 194 | 10 | 21,081 | 125,076 | $0.235 | 66 | 20 | |
| 7 | Opus 5.5 | Build | my control file | 2 | Claude Code 2.1.280 | on | passed | 2,028 | 236 | 8 | 21,030 | 94,103 | $0.228 | 66 | 20.8 | |
| 7 | Opus 5.5 | Build | my control file | 3 | Claude Code 2.1.280 | on | passed | 1,874 | 260 | 8 | 20,941 | 94,094 | $0.224 | 63 | 19.6 | |
| 7 | Opus 5.5 | Build | ponytail | 1 | Claude Code 2.1.280 | on | passed | 1,674 | 229 | 10 | 21,384 | 127,441 | $0.230 | 44 | 19.6 | |
| 7 | Opus 5.5 | Build | ponytail | 2 | Claude Code 2.1.280 | on | passed | 1,485 | 201 | 10 | 21,265 | 127,327 | $0.225 | 40 | 15.7 | |
| 7 | Opus 5.5 | Build | ponytail | 3 | Claude Code 2.1.280 | on | passed | 1,358 | 188 | 8 | 14,421 | 72,199 | $0.157* | 38 | 12.8 | |
| 7 | 6 Luna | Feature | caveman | 1 | Codex CLI 0.156.0 | on | passed | 549 | 0 | 153,956 | 28,004 | 125,952 | $0.00433 to $0.00503 | 6 | 20.9 | |
| 7 | 6 Luna | Feature | caveman | 2 | Codex CLI 0.156.0 | on | passed | 596 | 0 | 142,476 | 22,412 | 120,064 | $0.00374 to $0.00430 | 6 | 23.1 | |
| 7 | 6 Luna | Feature | caveman | 3 | Codex CLI 0.156.0 | on | passed | 653 | 0 | 139,137 | 24,193 | 114,944 | $0.00390 to $0.00450 | 8 | 22.2 | |
| 7 | 6 Luna | Feature | No skill | 1 | Codex CLI 0.156.0 | on | passed | 699 | 0 | 114,536 | 19,048 | 95,488 | $0.00321 to $0.00368 | 5 | 29.4 | |
| 7 | 6 Luna | Feature | No skill | 2 | Codex CLI 0.156.0 | on | passed | 528 | 0 | 99,179 | 18,795 | 80,384 | $0.00295 to $0.00342 | 6 | 18.7 | |
| 7 | 6 Luna | Feature | No skill | 3 | Codex CLI 0.156.0 | on | passed | 821 | 0 | 127,969 | 18,401 | 109,568 | $0.00335 to $0.00381 | 5 | 24.2 | |
| 7 | 6 Luna | Feature | karpathy-skills | 1 | Codex CLI 0.156.0 | on | passed | 692 | 0 | 139,119 | 20,335 | 118,784 | $0.00357 to $0.00407 | 5 | 22.5 | |
| 7 | 6 Luna | Feature | karpathy-skills | 2 | Codex CLI 0.156.0 | on | passed | 773 | 0 | 136,873 | 23,209 | 113,664 | $0.00384 to $0.00442 | 7 | 24.8 | |
| 7 | 6 Luna | Feature | karpathy-skills | 3 | Codex CLI 0.156.0 | on | passed | 582 | 0 | 104,781 | 18,253 | 86,528 | $0.00298 to $0.00344 | 5 | 22.6 | |
| 7 | 6 Luna | Feature | my control file | 1 | Codex CLI 0.156.0 | on | passed | 799 | 0 | 150,673 | 23,953 | 126,720 | $0.00406 to $0.00466 | 5 | 33.9 | |
| 7 | 6 Luna | Feature | my control file | 2 | Codex CLI 0.156.0 | on | passed | 884 | 0 | 120,874 | 23,338 | 97,536 | $0.00375 to $0.00433 | 7 | 50.4 | |
| 7 | 6 Luna | Feature | my control file | 3 | Codex CLI 0.156.0 | on | passed | 709 | 0 | 136,245 | 19,509 | 116,736 | $0.00347 to $0.00396 | 5 | 24.3 | |
| 7 | 6 Luna | Feature | ponytail | 1 | Codex CLI 0.156.0 | on | passed | 790 | 0 | 152,228 | 20,388 | 131,840 | $0.00375 to $0.00426 | 5 | 32.6 | |
| 7 | 6 Luna | Feature | ponytail | 2 | Codex CLI 0.156.0 | on | passed | 695 | 0 | 136,815 | 18,031 | 118,784 | $0.00334 to $0.00379 | 5 | 28.7 | |
| 7 | 6 Luna | Feature | ponytail | 3 | Codex CLI 0.156.0 | on | passed | 740 | 0 | 158,362 | 24,474 | 133,888 | $0.00416 to $0.00477 | 5 | 34.2 | |
| 7 | 6 Luna | Bug fix | caveman | 1 | Codex CLI 0.156.0 | on | passed | 482 | 0 | 167,660 | 20,460 | 147,200 | $0.00376 to $0.00427 | 2 | 19.8 | |
| 7 | 6 Luna | Bug fix | caveman | 2 | Codex CLI 0.156.0 | on | passed | 520 | 0 | 114,233 | 19,513 | 94,720 | $0.00316 to $0.00365 | 2 | 18.8 | |
| 7 | 6 Luna | Bug fix | caveman | 3 | Codex CLI 0.156.0 | on | passed | 538 | 0 | 133,407 | 19,487 | 113,920 | $0.00336 to $0.00384 | 5 | 19.6 | |
| 7 | 6 Luna | Bug fix | No skill | 1 | Codex CLI 0.156.0 | on | passed | 684 | 0 | 126,912 | 18,368 | 108,544 | $0.00326 to $0.00372 | 5 | 25.6 | |
| 7 | 6 Luna | Bug fix | No skill | 2 | Codex CLI 0.156.0 | on | passed | 653 | 0 | 127,556 | 19,012 | 108,544 | $0.00331 to $0.00379 | 4 | 22.5 | |
| 7 | 6 Luna | Bug fix | No skill | 3 | Codex CLI 0.156.0 | on | passed | 711 | 0 | 129,490 | 18,898 | 110,592 | $0.00335 to $0.00382 | 7 | 23.6 | |
| 7 | 6 Luna | Bug fix | karpathy-skills | 1 | Codex CLI 0.156.0 | on | passed | 910 | 0 | 140,051 | 23,315 | 116,736 | $0.00395 to $0.00454 | 2 | 26.3 | |
| 7 | 6 Luna | Bug fix | karpathy-skills | 2 | Codex CLI 0.156.0 | on | passed | 646 | 0 | 124,380 | 22,748 | 101,632 | $0.00361 to $0.00418 | 2 | 22.6 | |
| 7 | 6 Luna | Bug fix | karpathy-skills | 3 | Codex CLI 0.156.0 | on | passed | 733 | 0 | 157,376 | 23,488 | 133,888 | $0.00405 to $0.00464 | 2 | 22.4 | |
| 7 | 6 Luna | Bug fix | my control file | 1 | Codex CLI 0.156.0 | on | passed | 717 | 0 | 122,089 | 20,457 | 101,632 | $0.00342 to $0.00393 | 4 | 26.5 | |
| 7 | 6 Luna | Bug fix | my control file | 2 | Codex CLI 0.156.0 | on | passed | 662 | 0 | 136,629 | 19,893 | 116,736 | $0.00349 to $0.00398 | 5 | 28.2 | |
| 7 | 6 Luna | Bug fix | my control file | 3 | Codex CLI 0.156.0 | on | passed | 684 | 0 | 120,541 | 19,933 | 100,608 | $0.00334 to $0.00384 | 7 | 21.8 | |
| 7 | 6 Luna | Bug fix | ponytail | 1 | Codex CLI 0.156.0 | on | passed | 744 | 0 | 136,778 | 22,090 | 114,688 | $0.00373 to $0.00428 | 5 | 28.9 | |
| 7 | 6 Luna | Bug fix | ponytail | 2 | Codex CLI 0.156.0 | on | passed | 744 | 0 | 133,416 | 25,896 | 107,520 | $0.00404 to $0.00468 | 2 | 22.6 | |
| 7 | 6 Luna | Bug fix | ponytail | 3 | Codex CLI 0.156.0 | on | passed | 710 | 0 | 106,001 | 17,425 | 88,576 | $0.00298 to $0.00342 | 5 | 22.8 | |
| 7 | 6 Luna | Explain | karpathy-skills | 1 | Codex CLI 0.156.0 | on | failed | 49 | 0 | 25,883 | 12,827 | 13,056 | $0.00144 to $0.00176 | 0 | 3.8 | |
| 7 | 6 Luna | Explain | karpathy-skills | 2 | Codex CLI 0.156.0 | on | failed | 60 | 0 | 25,877 | 25,877 | 0 | $0.00262 to $0.00327 | 0 | 3.9 | |
| 7 | 6 Luna | Explain | karpathy-skills | 3 | Codex CLI 0.156.0 | on | passed | 177 | 0 | 40,472 | 14,360 | 26,112 | $0.00179 to $0.00215 | 0 | 8.1 | |
| 7 | 6 Luna | Explain | my control file | 1 | Codex CLI 0.156.0 | on | failed | 31 | 0 | 25,781 | 12,725 | 13,056 | $0.00142 to $0.00174 | 0 | 6.8 | |
| 7 | 6 Luna | Explain | my control file | 2 | Codex CLI 0.156.0 | on | failed | 54 | 0 | 25,784 | 12,728 | 13,056 | $0.00143 to $0.00175 | 0 | 6.3 | |
| 7 | 6 Luna | Explain | my control file | 3 | Codex CLI 0.156.0 | on | failed | 32 | 0 | 25,781 | 12,725 | 13,056 | $0.00142 to $0.00174 | 0 | 3.9 | |
| 7 | 6 Luna | Explain | ponytail | 1 | Codex CLI 0.156.0 | on | failed | 48 | 0 | 25,927 | 12,871 | 13,056 | $0.00144 to $0.00176 | 0 | 4.7 | |
| 7 | 6 Luna | Explain | ponytail | 2 | Codex CLI 0.156.0 | on | failed | 51 | 0 | 25,924 | 12,868 | 13,056 | $0.00144 to $0.00177 | 0 | 7.7 | |
| 7 | 6 Luna | Explain | ponytail | 3 | Codex CLI 0.156.0 | on | failed | 48 | 0 | 25,933 | 12,877 | 13,056 | $0.00144 to $0.00176 | 0 | 3.5 | |
| 7 | 6 Luna | Build | caveman | 1 | Codex CLI 0.156.0 | on | passed | 1,215 | 0 | 200,596 | 24,980 | 175,616 | $0.00486 to $0.00549 | 63 | 37.7 | |
| 7 | 6 Luna | Build | caveman | 2 | Codex CLI 0.156.0 | on | passed | 863 | 0 | 115,829 | 22,133 | 93,696 | $0.00358 to $0.00413 | 55 | 27.1 | |
| 7 | 6 Luna | Build | caveman | 3 | Codex CLI 0.156.0 | on | passed | 929 | 0 | 141,180 | 24,188 | 116,992 | $0.00405 to $0.00466 | 50 | 29.4 | |
| 7 | 6 Luna | Build | No skill | 1 | Codex CLI 0.156.0 | on | passed | 1,267 | 0 | 149,336 | 21,592 | 127,744 | $0.00407 to $0.00461 | 54 | 55.9 | |
| 7 | 6 Luna | Build | No skill | 2 | Codex CLI 0.156.0 | on | passed | 1,357 | 0 | 164,523 | 23,723 | 140,800 | $0.00446 to $0.00505 | 59 | 40.9 | |
| 7 | 6 Luna | Build | No skill | 3 | Codex CLI 0.156.0 | on | passed | 1,115 | 0 | 118,560 | 20,000 | 98,560 | $0.00354 to $0.00404 | 56 | 30.6 | |
| 7 | 6 Luna | Build | karpathy-skills | 1 | Codex CLI 0.156.0 | on | passed | 1,138 | 0 | 107,858 | 23,378 | 84,480 | $0.00375 to $0.00434 | 44 | 29.2 | |
| 7 | 6 Luna | Build | karpathy-skills | 2 | Codex CLI 0.156.0 | on | passed | 1,054 | 0 | 122,165 | 20,533 | 101,632 | $0.00360 to $0.00411 | 46 | 29.2 | |
| 7 | 6 Luna | Build | karpathy-skills | 3 | Codex CLI 0.156.0 | on | passed | 1,070 | 0 | 125,860 | 20,132 | 105,728 | $0.00361 to $0.00411 | 47 | 31.3 | |
| 7 | 6 Luna | Build | my control file | 1 | Codex CLI 0.156.0 | on | passed | 1,099 | 0 | 140,340 | 20,532 | 119,808 | $0.00380 to $0.00431 | 44 | 30.4 | |
| 7 | 6 Luna | Build | my control file | 2 | Codex CLI 0.156.0 | on | passed | 1,336 | 0 | 180,310 | 25,174 | 155,136 | $0.00474 to $0.00537 | 46 | 37.4 | |
| 7 | 6 Luna | Build | my control file | 3 | Codex CLI 0.156.0 | on | passed | 1,371 | 0 | 181,013 | 22,805 | 158,208 | $0.00455 to $0.00512 | 50 | 37.2 | |
| 7 | 6 Luna | Build | ponytail | 1 | Codex CLI 0.156.0 | on | passed | 1,170 | 0 | 184,807 | 27,623 | 157,184 | $0.00492 to $0.00561 | 44 | 38.4 | |
| 7 | 6 Luna | Build | ponytail | 2 | Codex CLI 0.156.0 | on | passed | 1,264 | 0 | 183,749 | 24,517 | 159,232 | $0.00468 to $0.00529 | 50 | 36.2 | |
| 7 | 6 Luna | Build | ponytail | 3 | Codex CLI 0.156.0 | on | passed | 1,145 | 0 | 179,404 | 24,268 | 155,136 | $0.00455 to $0.00516 | 51 | 33.4 | |
| 7‡ | Opus 5.5 | Explain | karpathy-skills | 1 | Claude Code 2.1.280 | on | passed | 1,688 | 462 | 4 | 11,318 | 26,794 | $0.130* | 0 | 19.2 | |
| 7‡ | Opus 5.5 | Explain | karpathy-skills | 2 | Claude Code 2.1.280 | on | passed | 1,738 | 513 | 4 | 17,996 | 37,427 | $0.186 | 0 | 19.6 | |
| 7‡ | Opus 5.5 | Explain | karpathy-skills | 3 | Claude Code 2.1.280 | on | passed | 1,630 | 360 | 4 | 17,990 | 37,423 | $0.184 | 0 | 18.6 | |
| 7‡ | Opus 5.5 | Explain | my control file | 1 | Claude Code 2.1.280 | on | passed | 1,422 | 273 | 4 | 17,738 | 37,169 | $0.178 | 0 | 14.9 | |
| 7‡ | Opus 5.5 | Explain | my control file | 2 | Claude Code 2.1.280 | on | passed | 1,265 | 231 | 4 | 17,733 | 37,165 | $0.175 | 0 | 17 | |
| 7‡ | Opus 5.5 | Explain | my control file | 3 | Claude Code 2.1.280 | on | passed | 1,471 | 414 | 4 | 17,729 | 37,162 | $0.179 | 0 | 17.8 | |
| 7‡ | Opus 5.5 | Explain | ponytail | 1 | Claude Code 2.1.280 | on | passed | 1,429 | 276 | 4 | 18,006 | 37,438 | $0.180 | 0 | 17.2 | |
| 7‡ | Opus 5.5 | Explain | ponytail | 2 | Claude Code 2.1.280 | on | passed | 1,510 | 368 | 4 | 18,006 | 37,438 | $0.182 | 0 | 15.1 | |
| 7‡ | Opus 5.5 | Explain | ponytail | 3 | Claude Code 2.1.280 | on | passed | 1,217 | 246 | 4 | 18,040 | 37,471 | $0.176 | 0 | 15.2 | |
| 8 | Fable 5.1 | Explain | caveman | 1 | Claude Code 2.1.283 | on | passed | 1,725 | 722 | 4 | 13,163 | 28,653 | $0.357* | 0 | 23.5 | |
| 8 | Fable 5.1 | Explain | caveman | 2 | Claude Code 2.1.283 | on | passed | 1,327 | 593 | 4 | 22,683 | 20,236 | $0.525* | 0 | 19.4 | |
| 8 | Fable 5.1 | Explain | No skill | 1 | Claude Code 2.1.283 | on | passed | 1,859 | 567 | 4 | 18,148 | 36,679 | $0.465 | 0 | 29.3 | |
| 8 | Fable 5.1 | Explain | No skill | 2 | Claude Code 2.1.283 | on | passed | 1,503 | 527 | 4 | 17,390 | 20,236 | $0.428* | 0 | 21.6 | |
| 8 | Opus 5.5 | Explain | No skill | 1 | Claude Code 2.1.283 | on | passed | 1,679 | 510 | 4 | 10,284 | 25,776 | $0.121* | 0 | 17.7 | |
| 8 | Sonnet 5 | Explain | No skill | 1 | Claude Code 2.1.283 | on | passed | 981 | 34 | 6 | 11,357 | 75,606 | $0.0704* | 0 | 14.8 | |
| 8 | 5.6 Luna | Explain | caveman | 1 | Codex CLI 0.157.1 | on | passed | 264 | 26 | 37,899 | 17,931 | 19,968 | $0.00430 to $0.00520 | 0 | 13.2 | |
| 8 | 5.6 Luna | Explain | caveman | 2 | Codex CLI 0.157.1 | on | passed | 268 | 87 | 37,904 | 14,864 | 23,040 | $0.00376 to $0.00450 | 0 | 9.7 | |
| 8 | 5.6 Luna | Explain | No skill | 1 | Codex CLI 0.157.1 | on | passed | 287 | 75 | 34,521 | 14,553 | 19,968 | $0.00365 to $0.00438 | 0 | 11.2 | |
| 8 | 5.6 Luna | Explain | No skill | 2 | Codex CLI 0.157.1 | on | passed | 352 | 104 | 34,526 | 14,558 | 19,968 | $0.00373 to $0.00446 | 0 | 12.6 | |
| 8 | 5.6 Sol | Explain | caveman | 1 | Codex CLI 0.157.1 | on | passed | 183 | 0 | 41,911 | 15,927 | 25,984 | $0.0778 to $0.0937 | 0 | 10.2 | |
| 8 | 5.6 Sol | Explain | caveman | 2 | Codex CLI 0.157.1 | on | passed | 202 | 0 | 41,911 | 15,927 | 25,984 | $0.0781 to $0.0941 | 0 | 8.7 | |
| 8 | 5.6 Sol | Explain | No skill | 1 | Codex CLI 0.157.1 | on | passed | 374 | 135 | 38,373 | 14,309 | 24,064 | $0.0743 to $0.0887 | 0 | 12.2 | |
| 8 | 5.6 Sol | Explain | No skill | 2 | Codex CLI 0.157.1 | on | passed | 394 | 131 | 38,545 | 14,225 | 24,320 | $0.0745 to $0.0887 | 0 | 12.6 | |
| 8 | 6 Astra | Explain | caveman | 1 | Codex CLI 0.157.1 | on | passed | 188 | 0 | 44,694 | 16,790 | 27,904 | $0.205 to $0.247 | 0 | 14.1 | |
| 8 | 6 Astra | Explain | caveman | 2 | Codex CLI 0.157.1 | on | passed | 181 | 0 | 44,705 | 16,801 | 27,904 | $0.205 to $0.247 | 0 | 16.2 | |
| 8 | 6 Astra | Explain | No skill | 1 | Codex CLI 0.157.1 | on | passed | 250 | 0 | 41,313 | 16,993 | 24,320 | $0.207 to $0.249 | 0 | 13.7 | |
| 8 | 6 Astra | Explain | No skill | 2 | Codex CLI 0.157.1 | on | passed | 302 | 0 | 55,775 | 15,327 | 40,448 | $0.209 to $0.247 | 0 | 15.2 | |
| 8 | 6 Luna | Explain | caveman | 1 | Codex CLI 0.157.1 | on | passed | 159 | 0 | 42,759 | 16,647 | 26,112 | $0.00200 to $0.00242 | 0 | 7.8 | |
| 8 | 6 Luna | Explain | caveman | 2 | Codex CLI 0.157.1 | on | passed | 129 | 0 | 42,698 | 16,586 | 26,112 | $0.00198 to $0.00240 | 0 | 7.4 | |
| 8 | 6 Luna | Explain | No skill | 1 | Codex CLI 0.157.1 | on | passed | 197 | 0 | 39,341 | 15,277 | 24,064 | $0.00187 to $0.00225 | 0 | 9.6 | |
| 8 | 6 Luna | Explain | No skill | 2 | Codex CLI 0.157.1 | on | passed | 164 | 0 | 39,337 | 15,273 | 24,064 | $0.00185 to $0.00223 | 0 | 9 | |
| 8 | 6 Sol | Explain | caveman | 1 | Codex CLI 0.157.1 | on | passed | 168 | 0 | 43,258 | 16,250 | 27,008 | $0.0396 to $0.0477 | 0 | 10.3 | |
| 8 | 6 Sol | Explain | caveman | 2 | Codex CLI 0.157.1 | on | passed | 144 | 0 | 43,256 | 16,248 | 27,008 | $0.0393 to $0.0475 | 0 | 10.9 | |
| 8 | 6 Sol | Explain | No skill | 1 | Codex CLI 0.157.1 | on | passed | 392 | 118 | 39,920 | 14,576 | 25,344 | $0.0381 to $0.0454 | 0 | 15.3 | |
| 8 | 6 Sol | Explain | No skill | 2 | Codex CLI 0.157.1 | on | passed | 249 | 67 | 39,899 | 14,555 | 25,344 | $0.0367 to $0.0439 | 0 | 13.1 | |
| 9 | Sonnet 5.5 | Feature | caveman | 1 | Claude Code 2.1.284 | on | passed | 530 | 0 | 10 | 23,310 | 131,401 | $0.125 | 8 | 8.5 | |
| 9 | Sonnet 5.5 | Feature | caveman | 2 | Claude Code 2.1.284 | on | passed | 510 | 0 | 10 | 22,383 | 131,057 | $0.121 | 8 | 6.7 | |
| 9 | Sonnet 5.5 | Feature | caveman | 3 | Claude Code 2.1.284 | on | passed | 625 | 0 | 10 | 22,519 | 131,213 | $0.123 | 6 | 7.4 | |
| 9 | Sonnet 5.5 | Feature | No skill | 1 | Claude Code 2.1.284 | on | passed | 652 | 0 | 10 | 17,368 | 114,904 | $0.0990 | 6 | 8.6 | |
| 9 | Sonnet 5.5 | Feature | No skill | 2 | Claude Code 2.1.284 | on | passed | 498 | 0 | 10 | 17,275 | 114,744 | $0.0970 | 6 | 6.9 | |
| 9 | Sonnet 5.5 | Feature | No skill | 3 | Claude Code 2.1.284 | on | passed | 506 | 0 | 10 | 17,269 | 114,732 | $0.0971 | 6 | 9.8 | |
| 9 | Sonnet 5.5 | Feature | karpathy-skills | 1 | Claude Code 2.1.284 | on | passed | 792 | 0 | 10 | 19,195 | 121,315 | $0.109 | 5 | 8 | |
| 9 | Sonnet 5.5 | Feature | karpathy-skills | 2 | Claude Code 2.1.284 | on | passed | 687 | 0 | 10 | 19,125 | 121,267 | $0.108 | 5 | 7.4 | |
| 9 | Sonnet 5.5 | Feature | karpathy-skills | 3 | Claude Code 2.1.284 | on | passed | 768 | 27 | 10 | 19,224 | 121,325 | $0.109 | 6 | 8.4 | |
| 9 | Sonnet 5.5 | Feature | my control file | 1 | Claude Code 2.1.284 | on | passed | 844 | 0 | 10 | 18,735 | 119,695 | $0.107 | 8 | 10.1 | |
| 9 | Sonnet 5.5 | Feature | my control file | 2 | Claude Code 2.1.284 | on | passed | 858 | 0 | 10 | 18,764 | 119,074 | $0.107 | 8 | 9.2 | |
| 9 | Sonnet 5.5 | Feature | my control file | 3 | Claude Code 2.1.284 | on | passed | 830 | 0 | 10 | 18,740 | 119,705 | $0.107 | 8 | 8.2 | |
| 9 | Sonnet 5.5 | Feature | ponytail | 1 | Claude Code 2.1.284 | on | passed | 574 | 0 | 10 | 19,251 | 120,741 | $0.107 | 6 | 7 | |
| 9 | Sonnet 5.5 | Feature | ponytail | 2 | Claude Code 2.1.284 | on | passed | 691 | 0 | 10 | 19,154 | 121,304 | $0.108 | 6 | 9.1 | |
| 9 | Sonnet 5.5 | Feature | ponytail | 3 | Claude Code 2.1.284 | on | passed | 784 | 0 | 10 | 19,621 | 121,755 | $0.111 | 6 | 10.8 | |
| 9 | Sonnet 5.5 | Bug fix | caveman | 1 | Claude Code 2.1.284 | on | passed | 616 | 106 | 8 | 14,120 | 71,163 | $0.0769* | 2 | 6 | |
| 9 | Sonnet 5.5 | Bug fix | caveman | 2 | Claude Code 2.1.284 | on | passed | 554 | 0 | 8 | 22,017 | 98,653 | $0.113 | 2 | 6.5 | |
| 9 | Sonnet 5.5 | Bug fix | caveman | 3 | Claude Code 2.1.284 | on | passed | 634 | 107 | 8 | 22,113 | 98,659 | $0.115 | 2 | 7.2 | |
| 9 | Sonnet 5.5 | Bug fix | No skill | 1 | Claude Code 2.1.284 | on | passed | 771 | 0 | 10 | 17,260 | 114,539 | $0.0997 | 6 | 8 | |
| 9 | Sonnet 5.5 | Bug fix | No skill | 2 | Claude Code 2.1.284 | on | passed | 628 | 0 | 10 | 17,734 | 115,035 | $0.100 | 6 | 11.4 | |
| 9 | Sonnet 5.5 | Bug fix | No skill | 3 | Claude Code 2.1.284 | on | passed | 505 | 0 | 10 | 17,350 | 114,869 | $0.0974 | 2 | 6.9 | |
| 9 | Sonnet 5.5 | Bug fix | karpathy-skills | 1 | Claude Code 2.1.284 | on | passed | 660 | 40 | 8 | 18,666 | 92,028 | $0.0997 | 2 | 7.6 | |
| 9 | Sonnet 5.5 | Bug fix | karpathy-skills | 2 | Claude Code 2.1.284 | on | passed | 634 | 37 | 8 | 18,671 | 92,046 | $0.0994 | 2 | 9.2 | |
| 9 | Sonnet 5.5 | Bug fix | karpathy-skills | 3 | Claude Code 2.1.284 | on | passed | 500 | 49 | 8 | 12,351 | 69,201 | $0.0683* | 2 | 6.2 | |
| 9 | Sonnet 5.5 | Bug fix | my control file | 1 | Claude Code 2.1.284 | on | passed | 884 | 63 | 10 | 18,371 | 118,907 | $0.106 | 3 | 8.7 | |
| 9 | Sonnet 5.5 | Bug fix | my control file | 2 | Claude Code 2.1.284 | on | passed | 955 | 98 | 10 | 19,194 | 118,833 | $0.110 | 7 | 16.5 | |
| 9 | Sonnet 5.5 | Bug fix | my control file | 3 | Claude Code 2.1.284 | on | passed | 772 | 35 | 8 | 18,184 | 90,974 | $0.0987 | 3 | 10.4 | |
| 9 | Sonnet 5.5 | Bug fix | ponytail | 1 | Claude Code 2.1.284 | on | passed | 1,316 | 196 | 12 | 19,994 | 151,183 | $0.123 | 8 | 12.1 | |
| 9 | Sonnet 5.5 | Bug fix | ponytail | 2 | Claude Code 2.1.284 | on | passed | 886 | 183 | 8 | 19,380 | 92,585 | $0.105 | 4 | 10.6 | |
| 9 | Sonnet 5.5 | Bug fix | ponytail | 3 | Claude Code 2.1.284 | on | passed | 728 | 113 | 8 | 18,822 | 92,107 | $0.101 | 4 | 10.3 | |
| 9 | Sonnet 5.5 | Build | caveman | 1 | Claude Code 2.1.284 | on | passed | 1,349 | 69 | 8 | 24,513 | 100,268 | $0.132 | 55 | 17.2 | |
| 9 | Sonnet 5.5 | Build | caveman | 2 | Claude Code 2.1.284 | on | passed | 1,358 | 43 | 8 | 24,498 | 100,277 | $0.132 | 53 | 11.5 | |
| 9 | Sonnet 5.5 | Build | caveman | 3 | Claude Code 2.1.284 | on | passed | 1,298 | 59 | 8 | 23,929 | 99,909 | $0.129 | 58 | 10.9 | |
| 9 | Sonnet 5.5 | Build | No skill | 1 | Claude Code 2.1.284 | on | passed | 1,187 | 0 | 8 | 11,154 | 65,454 | $0.0696* | 43 | 11.3 | |
| 9 | Sonnet 5.5 | Build | No skill | 2 | Claude Code 2.1.284 | on | passed | 1,319 | 80 | 8 | 18,955 | 89,668 | $0.107 | 49 | 14.5 | |
| 9 | Sonnet 5.5 | Build | No skill | 3 | Claude Code 2.1.284 | on | passed | 1,108 | 0 | 8 | 18,355 | 89,158 | $0.102 | 49 | 11.3 | |
| 9 | Sonnet 5.5 | Build | karpathy-skills | 1 | Claude Code 2.1.284 | on | passed | 1,405 | 101 | 8 | 20,951 | 93,609 | $0.117 | 44 | 11.8 | |
| 9 | Sonnet 5.5 | Build | karpathy-skills | 2 | Claude Code 2.1.284 | on | passed | 1,349 | 176 | 8 | 21,737 | 94,637 | $0.119 | 43 | 15.4 | |
| 9 | Sonnet 5.5 | Build | karpathy-skills | 3 | Claude Code 2.1.284 | on | passed | 1,275 | 76 | 8 | 11,920 | 65,282 | $0.0735* | 43 | 9.8 | |
| 9 | Sonnet 5.5 | Build | my control file | 1 | Claude Code 2.1.284 | on | passed | 1,444 | 124 | 8 | 20,079 | 92,222 | $0.113 | 52 | 12.9 | |
| 9 | Sonnet 5.5 | Build | my control file | 2 | Claude Code 2.1.284 | on | passed | 1,629 | 79 | 8 | 19,886 | 91,744 | $0.114 | 59 | 13.1 | |
| 9 | Sonnet 5.5 | Build | my control file | 3 | Claude Code 2.1.284 | on | passed | 1,571 | 96 | 8 | 20,571 | 92,581 | $0.117 | 62 | 13.6 | |
| 9 | Sonnet 5.5 | Build | ponytail | 1 | Claude Code 2.1.284 | on | passed | 1,101 | 126 | 8 | 20,686 | 93,669 | $0.113 | 31 | 10.3 | |
| 9 | Sonnet 5.5 | Build | ponytail | 2 | Claude Code 2.1.284 | on | passed | 1,159 | 136 | 8 | 20,320 | 93,326 | $0.112 | 31 | 10.6 | |
| 9 | Sonnet 5.5 | Build | ponytail | 3 | Claude Code 2.1.284 | on | passed | 1,248 | 205 | 8 | 20,671 | 93,594 | $0.114 | 34 | 13 | |
| 9‡ | Sonnet 5.5 | Explain | caveman | 1 | Claude Code 2.1.284 | on | passed | 653 | 0 | 4 | 18,698 | 38,360 | $0.0890 | 0 | 6.8 | |
| 9‡ | Sonnet 5.5 | Explain | caveman | 2 | Claude Code 2.1.284 | on | passed | 746 | 0 | 4 | 18,694 | 38,357 | $0.0899 | 0 | 9.2 | |
| 9‡ | Sonnet 5.5 | Explain | caveman | 3 | Claude Code 2.1.284 | on | passed | 747 | 0 | 4 | 9,067 | 28,407 | $0.0494* | 0 | 8.9 | |
| 9‡ | Sonnet 5.5 | Explain | No skill | 1 | Claude Code 2.1.284 | on | passed | 964 | 73 | 4 | 6,417 | 25,759 | $0.0405* | 0 | 10.5 | |
| 9‡ | Sonnet 5.5 | Explain | No skill | 2 | Claude Code 2.1.284 | on | passed | 858 | 0 | 4 | 16,049 | 35,712 | $0.0799 | 0 | 8.3 | |
| 9‡ | Sonnet 5.5 | Explain | No skill | 3 | Claude Code 2.1.284 | on | passed | 854 | 0 | 4 | 9,342 | 25,850 | $0.0511* | 0 | 7.5 | |
| 9‡ | Sonnet 5.5 | Explain | karpathy-skills | 1 | Claude Code 2.1.284 | on | passed | 785 | 0 | 4 | 17,064 | 36,727 | $0.0835 | 0 | 8 | |
| 9‡ | Sonnet 5.5 | Explain | karpathy-skills | 2 | Claude Code 2.1.284 | on | passed | 868 | 0 | 4 | 17,061 | 36,725 | $0.0843 | 0 | 9.1 | |
| 9‡ | Sonnet 5.5 | Explain | karpathy-skills | 3 | Claude Code 2.1.284 | on | passed | 840 | 0 | 4 | 17,063 | 36,726 | $0.0840 | 0 | 8.3 | |
| 9‡ | Sonnet 5.5 | Explain | my control file | 1 | Claude Code 2.1.284 | on | passed | 889 | 0 | 4 | 16,834 | 36,497 | $0.0835 | 0 | 10.6 | |
| 9‡ | Sonnet 5.5 | Explain | my control file | 2 | Claude Code 2.1.284 | on | passed | 910 | 0 | 4 | 10,098 | 26,604 | $0.0548* | 0 | 9.4 | |
| 9‡ | Sonnet 5.5 | Explain | my control file | 3 | Claude Code 2.1.284 | on | passed | 930 | 0 | 4 | 16,797 | 36,461 | $0.0838 | 0 | 8.7 | |
| 9‡ | Sonnet 5.5 | Explain | ponytail | 1 | Claude Code 2.1.284 | on | passed | 818 | 0 | 4 | 17,079 | 36,741 | $0.0839 | 0 | 9.9 | |
| 9‡ | Sonnet 5.5 | Explain | ponytail | 2 | Claude Code 2.1.284 | on | passed | 724 | 0 | 4 | 17,078 | 36,740 | $0.0829 | 0 | 7.1 | |
| 9‡ | Sonnet 5.5 | Explain | ponytail | 3 | Claude Code 2.1.284 | on | passed | 785 | 0 | 4 | 17,082 | 36,743 | $0.0835 | 0 | 10 | |
| 9 | Opus 5.5 | Explain | No skill | 1 | Claude Code 2.1.284 | on | passed | 1,628 | 478 | 4 | 8,211 | 25,687 | $0.103* | 0 | 16.4 | |
| 9 | Opus 5.5 | Explain | No skill | 2 | Claude Code 2.1.284 | on | passed | 1,335 | 370 | 4 | 10,284 | 25,777 | $0.114* | 0 | 17.7 | |
| 9 | Sonnet 5 | Explain | caveman | 1 | Claude Code 2.1.284 | on | passed | 318 | 17 | 4 | 22,612 | 59,051 | $0.105 | 0 | 5.3 | |
| 9 | Sonnet 5 | Explain | caveman | 2 | Claude Code 2.1.284 | on | passed | 302 | 12 | 4 | 15,933 | 49,172 | $0.0766* | 0 | 4.3 | |
| 9 | Sonnet 5 | Explain | No skill | 1 | Claude Code 2.1.284 | on | passed | 807 | 26 | 6 | 21,069 | 94,399 | $0.111 | 0 | 14.2 | |
| 9 | Sonnet 5 | Explain | No skill | 2 | Claude Code 2.1.284 | on | passed | 876 | 27 | 6 | 20,151 | 94,370 | $0.108 | 0 | 10.3 | |
| 10 | 6.1 Sol | Feature | caveman | 1 | Codex CLI 0.159.0 | on | passed | 555 | 0 | 107,410 | 18,834 | 88,576 | $0.0521 to $0.0615 | 6 | 30.3 | |
| 10 | 6.1 Sol | Feature | caveman | 2 | Codex CLI 0.159.0 | on | passed | 542 | 0 | 122,113 | 20,993 | 101,120 | $0.0575 to $0.0680 | 6 | 27.5 | |
| 10 | 6.1 Sol | Feature | caveman | 3 | Codex CLI 0.159.0 | on | passed | 398 | 0 | 102,104 | 20,440 | 81,664 | $0.0530 to $0.0632 | 6 | 21 | |
| 10 | 6.1 Sol | Feature | No skill | 1 | Codex CLI 0.159.0 | on | passed | 569 | 12 | 105,599 | 17,919 | 87,680 | $0.0503 to $0.0593 | 6 | 28.2 | |
| 10 | 6.1 Sol | Feature | No skill | 2 | Codex CLI 0.159.0 | on | passed | 706 | 14 | 88,946 | 17,522 | 71,424 | $0.0492 to $0.0580 | 8 | 30.2 | |
| 10 | 6.1 Sol | Feature | No skill | 3 | Codex CLI 0.159.0 | on | passed | 669 | 10 | 88,689 | 17,393 | 71,296 | $0.0486 to $0.0573 | 8 | 30.1 | |
| 10 | 6.1 Sol | Feature | karpathy-skills | 1 | Codex CLI 0.159.0 | on | passed | 620 | 11 | 95,892 | 19,476 | 76,416 | $0.0528 to $0.0625 | 5 | 28.2 | |
| 10 | 6.1 Sol | Feature | karpathy-skills | 2 | Codex CLI 0.159.0 | on | passed | 571 | 11 | 113,556 | 19,860 | 93,696 | $0.0548 to $0.0647 | 5 | 27.9 | |
| 10 | 6.1 Sol | Feature | karpathy-skills | 3 | Codex CLI 0.159.0 | on | passed | 556 | 0 | 112,583 | 19,399 | 93,184 | $0.0537 to $0.0634 | 5 | 30.2 | |
| 10 | 6.1 Sol | Feature | my control file | 1 | Codex CLI 0.159.0 | on | passed | 634 | 28 | 109,279 | 18,911 | 90,368 | $0.0532 to $0.0627 | 6 | 30.8 | |
| 10 | 6.1 Sol | Feature | my control file | 2 | Codex CLI 0.159.0 | on | passed | 561 | 0 | 93,263 | 19,023 | 74,240 | $0.0511 to $0.0606 | 6 | 27.4 | |
| 10 | 6.1 Sol | Feature | my control file | 3 | Codex CLI 0.159.0 | on | passed | 443 | 0 | 94,056 | 18,664 | 75,392 | $0.0493 to $0.0586 | 6 | 22.4 | |
| 10 | 6.1 Sol | Feature | ponytail | 1 | Codex CLI 0.159.0 | on | passed | 536 | 22 | 94,533 | 18,757 | 75,776 | $0.0505 to $0.0598 | 5 | 24.7 | |
| 10 | 6.1 Sol | Feature | ponytail | 2 | Codex CLI 0.159.0 | on | passed | 658 | 58 | 94,422 | 19,286 | 75,136 | $0.0527 to $0.0623 | 5 | 30.5 | |
| 10 | 6.1 Sol | Feature | ponytail | 3 | Codex CLI 0.159.0 | on | passed | 615 | 27 | 111,941 | 19,141 | 92,800 | $0.0537 to $0.0633 | 5 | 28.2 | |
| 10 | 6.1 Sol | Bug fix | caveman | 1 | Codex CLI 0.159.0 | on | passed | 471 | 28 | 121,085 | 20,733 | 100,352 | $0.0562 to $0.0666 | 2 | 29.3 | |
| 10 | 6.1 Sol | Bug fix | caveman | 2 | Codex CLI 0.159.0 | on | passed | 486 | 19 | 141,349 | 21,029 | 120,320 | $0.0590 to $0.0695 | 2 | 26 | |
| 10 | 6.1 Sol | Bug fix | caveman | 3 | Codex CLI 0.159.0 | on | passed | 381 | 13 | 107,309 | 22,061 | 85,248 | $0.0565 to $0.0675 | 2 | 20.2 | |
| 10 | 6.1 Sol | Bug fix | No skill | 1 | Codex CLI 0.159.0 | on | passed | 498 | 21 | 103,588 | 20,132 | 83,456 | $0.0536 to $0.0637 | 2 | 25 | |
| 10 | 6.1 Sol | Bug fix | No skill | 2 | Codex CLI 0.159.0 | on | passed | 537 | 39 | 88,325 | 17,157 | 71,168 | $0.0468 to $0.0554 | 2 | 26.3 | |
| 10 | 6.1 Sol | Bug fix | No skill | 3 | Codex CLI 0.159.0 | on | passed | 652 | 49 | 104,598 | 17,686 | 86,912 | $0.0506 to $0.0594 | 5 | 40.6 | |
| 10 | 6.1 Sol | Bug fix | karpathy-skills | 1 | Codex CLI 0.159.0 | on | passed | 580 | 13 | 112,965 | 19,525 | 93,440 | $0.0542 to $0.0640 | 2 | 27 | |
| 10 | 6.1 Sol | Bug fix | karpathy-skills | 2 | Codex CLI 0.159.0 | on | passed | 625 | 16 | 95,541 | 19,381 | 76,160 | $0.0526 to $0.0623 | 2 | 30.1 | |
| 10 | 6.1 Sol | Bug fix | karpathy-skills | 3 | Codex CLI 0.159.0 | on | passed | 561 | 28 | 95,214 | 19,182 | 76,032 | $0.0516 to $0.0612 | 2 | 25 | |
| 10 | 6.1 Sol | Bug fix | my control file | 1 | Codex CLI 0.159.0 | on | passed | 583 | 34 | 108,597 | 18,613 | 89,984 | $0.0521 to $0.0614 | 2 | 28.4 | |
| 10 | 6.1 Sol | Bug fix | my control file | 2 | Codex CLI 0.159.0 | on | passed | 630 | 52 | 108,931 | 19,075 | 89,856 | $0.0534 to $0.0630 | 5 | 28.5 | |
| 10 | 6.1 Sol | Bug fix | my control file | 3 | Codex CLI 0.159.0 | on | passed | 597 | 59 | 109,986 | 19,234 | 90,752 | $0.0535 to $0.0631 | 5 | 27.5 | |
| 10 | 6.1 Sol | Bug fix | ponytail | 1 | Codex CLI 0.159.0 | on | passed | 542 | 77 | 94,563 | 18,915 | 75,648 | $0.0508 to $0.0603 | 2 | 27.3 | |
| 10 | 6.1 Sol | Bug fix | ponytail | 2 | Codex CLI 0.159.0 | on | passed | 555 | 59 | 111,526 | 19,110 | 92,416 | $0.0530 to $0.0626 | 2 | 27.6 | |
| 10 | 6.1 Sol | Bug fix | ponytail | 3 | Codex CLI 0.159.0 | on | passed | 604 | 77 | 95,041 | 19,009 | 76,032 | $0.0517 to $0.0612 | 2 | 26.5 | |
| 10 | 6.1 Sol | Explain | caveman | 1 | Codex CLI 0.159.0 | on | passed | 173 | 0 | 44,891 | 29,147 | 15,744 | $0.0616 to $0.0762 | 0 | 10.8 | |
| 10 | 6.1 Sol | Explain | caveman | 2 | Codex CLI 0.159.0 | on | passed | 186 | 0 | 44,895 | 16,863 | 28,032 | $0.0384 to $0.0468 | 0 | 10.1 | |
| 10 | 6.1 Sol | Explain | caveman | 3 | Codex CLI 0.159.0 | on | passed | 184 | 0 | 44,913 | 16,881 | 28,032 | $0.0384 to $0.0468 | 0 | 11.3 | |
| 10 | 6.1 Sol | Explain | No skill | 1 | Codex CLI 0.159.0 | on | passed | 232 | 0 | 41,529 | 15,161 | 26,368 | $0.0353 to $0.0429 | 0 | 13.5 | |
| 10 | 6.1 Sol | Explain | No skill | 2 | Codex CLI 0.159.0 | on | passed | 302 | 0 | 56,042 | 15,466 | 40,576 | $0.0380 to $0.0457 | 0 | 17 | |
| 10 | 6.1 Sol | Explain | No skill | 3 | Codex CLI 0.159.0 | on | passed | 286 | 0 | 56,019 | 15,443 | 40,576 | $0.0378 to $0.0455 | 0 | 16.2 | |
| 10 | 6.1 Sol | Explain | karpathy-skills | 1 | Codex CLI 0.159.0 | on | passed | 237 | 0 | 42,621 | 15,741 | 26,880 | $0.0365 to $0.0444 | 0 | 12.3 | |
| 10 | 6.1 Sol | Explain | karpathy-skills | 2 | Codex CLI 0.159.0 | on | passed | 262 | 0 | 42,626 | 15,746 | 26,880 | $0.0368 to $0.0447 | 0 | 12.8 | |
| 10 | 6.1 Sol | Explain | karpathy-skills | 3 | Codex CLI 0.159.0 | on | passed | 253 | 0 | 42,614 | 15,734 | 26,880 | $0.0367 to $0.0446 | 0 | 13.4 | |
| 10 | 6.1 Sol | Explain | my control file | 1 | Codex CLI 0.159.0 | on | passed | 216 | 0 | 42,448 | 15,696 | 26,752 | $0.0362 to $0.0441 | 0 | 12.8 | |
| 10 | 6.1 Sol | Explain | my control file | 2 | Codex CLI 0.159.0 | on | passed | 225 | 0 | 42,467 | 15,587 | 26,880 | $0.0361 to $0.0439 | 0 | 13.8 | |
| 10 | 6.1 Sol | Explain | my control file | 3 | Codex CLI 0.159.0 | on | passed | 226 | 0 | 42,445 | 15,693 | 26,752 | $0.0363 to $0.0442 | 0 | 11.6 | |
| 10 | 6.1 Sol | Explain | ponytail | 1 | Codex CLI 0.159.0 | on | passed | 223 | 0 | 42,725 | 15,717 | 27,008 | $0.0364 to $0.0442 | 0 | 12.5 | |
| 10 | 6.1 Sol | Explain | ponytail | 2 | Codex CLI 0.159.0 | on | passed | 211 | 0 | 42,719 | 15,711 | 27,008 | $0.0362 to $0.0441 | 0 | 11.3 | |
| 10 | 6.1 Sol | Explain | ponytail | 3 | Codex CLI 0.159.0 | on | passed | 211 | 0 | 42,745 | 15,737 | 27,008 | $0.0363 to $0.0442 | 0 | 11.1 | |
| 10 | 6.1 Sol | Build | caveman | 1 | Codex CLI 0.159.0 | on | passed | 896 | 0 | 110,834 | 23,922 | 86,912 | $0.0655 to $0.0775 | 51 | 35.9 | |
| 10 | 6.1 Sol | Build | caveman | 2 | Codex CLI 0.159.0 | on | passed | 1,082 | 44 | 124,937 | 22,793 | 102,144 | $0.0666 to $0.0780 | 60 | 44.3 | |
| 10 | 6.1 Sol | Build | caveman | 3 | Codex CLI 0.159.0 | on | passed | 936 | 0 | 131,953 | 24,049 | 107,904 | $0.0682 to $0.0803 | 57 | 39 | |
| 10 | 6.1 Sol | Build | No skill | 1 | Codex CLI 0.159.0 | on | passed | 1,026 | 19 | 107,353 | 19,161 | 88,192 | $0.0574 to $0.0670 | 50 | 40.7 | |
| 10 | 6.1 Sol | Build | No skill | 2 | Codex CLI 0.159.0 | on | passed | 1,077 | 18 | 108,001 | 18,529 | 89,472 | $0.0568 to $0.0660 | 54 | 42.1 | |
| 10 | 6.1 Sol | Build | No skill | 3 | Codex CLI 0.159.0 | on | passed | 978 | 12 | 92,226 | 18,882 | 73,344 | $0.0549 to $0.0643 | 50 | 40.2 | |
| 10 | 6.1 Sol | Build | karpathy-skills | 1 | Codex CLI 0.159.0 | on | passed | 1,035 | 131 | 97,356 | 22,732 | 74,624 | $0.0633 to $0.0746 | 45 | 43.6 | |
| 10 | 6.1 Sol | Build | karpathy-skills | 2 | Codex CLI 0.159.0 | on | passed | 1,060 | 86 | 97,545 | 22,921 | 74,624 | $0.0639 to $0.0754 | 47 | 45 | |
| 10 | 6.1 Sol | Build | karpathy-skills | 3 | Codex CLI 0.159.0 | on | passed | 1,042 | 62 | 98,136 | 20,824 | 77,312 | $0.0598 to $0.0702 | 46 | 42.2 | |
| 10 | 6.1 Sol | Build | my control file | 1 | Codex CLI 0.159.0 | on | passed | 1,168 | 31 | 115,000 | 20,792 | 94,208 | $0.0627 to $0.0731 | 66 | 44.4 | |
| 10 | 6.1 Sol | Build | my control file | 2 | Codex CLI 0.159.0 | on | passed | 1,144 | 12 | 114,375 | 26,439 | 87,936 | $0.0731 to $0.0863 | 66 | 44.2 | |
| 10 | 6.1 Sol | Build | my control file | 3 | Codex CLI 0.159.0 | on | passed | 1,318 | 36 | 113,958 | 20,774 | 93,184 | $0.0640 to $0.0744 | 67 | 51.6 | |
| 10 | 6.1 Sol | Build | ponytail | 1 | Codex CLI 0.159.0 | on | passed | 1,044 | 109 | 97,175 | 20,759 | 76,416 | $0.0596 to $0.0700 | 43 | 40.1 | |
| 10 | 6.1 Sol | Build | ponytail | 2 | Codex CLI 0.159.0 | on | passed | 1,124 | 212 | 115,122 | 23,090 | 92,032 | $0.0666 to $0.0782 | 41 | 44.4 | |
| 10 | 6.1 Sol | Build | ponytail | 3 | Codex CLI 0.159.0 | on | passed | 1,180 | 265 | 99,057 | 21,233 | 77,824 | $0.0620 to $0.0727 | 43 | 46.4 | |
| 10 | Opus 5.5 | Explain | No skill | 1 | Claude Code 2.1.284 | on | passed | 1,728 | 446 | 4 | 17,017 | 36,452 | $0.178 | 0 | 21.4 | |
| 10 | Opus 5.5 | Explain | No skill | 2 | Claude Code 2.1.284 | on | passed | 1,771 | 558 | 4 | 10,281 | 25,775 | $0.123* | 0 | 19.7 | |
| 10 | 6 Sol | Explain | caveman | 1 | Codex CLI 0.159.0 | on | passed | 126 | 0 | 43,236 | 24,292 | 18,944 | $0.0536 to $0.0658 | 0 | 16 | |
| 10 | 6 Sol | Explain | caveman | 2 | Codex CLI 0.159.0 | on | passed | 161 | 0 | 43,235 | 16,227 | 27,008 | $0.0395 to $0.0476 | 0 | 10.4 | |
| 10 | 6 Sol | Explain | No skill | 1 | Codex CLI 0.159.0 | on | passed | 277 | 79 | 39,894 | 14,550 | 25,344 | $0.0369 to $0.0442 | 0 | 13.6 | |
| 10 | 6 Sol | Explain | No skill | 2 | Codex CLI 0.159.0 | on | passed | 281 | 81 | 39,897 | 20,953 | 18,944 | $0.0485 to $0.0590 | 0 | 11.1 | |
| 13 | Opus 5.5 | Explain | caveman | 1 | Claude Code 2.1.284 | off | passed | 990 | 183 | 4 | 9,289 | 29,740 | $0.100 | 0 | 12.1 | |
| 13 | Opus 5.5 | Explain | caveman | 2 | Claude Code 2.1.284 | off | passed | 880 | 202 | 4 | 9,288 | 29,739 | $0.0979 | 0 | 11.9 | |
| 13 | Opus 5.5 | Explain | caveman | 3 | Claude Code 2.1.284 | off | passed | 907 | 179 | 4 | 9,287 | 29,738 | $0.0984 | 0 | 10.1 | |
| 13 | Opus 5.5 | Explain | No skill | 1 | Claude Code 2.1.284 | off | passed | 1,066 | 179 | 4 | 6,641 | 27,094 | $0.0799 | 0 | 12.8 | |
| 13 | Opus 5.5 | Explain | No skill | 2 | Claude Code 2.1.284 | off | passed | 1,167 | 179 | 4 | 6,646 | 27,097 | $0.0819 | 0 | 12.3 | |
| 13 | Opus 5.5 | Explain | No skill | 3 | Claude Code 2.1.284 | off | passed | 1,429 | 313 | 4 | 6,641 | 27,094 | $0.0871 | 0 | 15.1 | |
| 13 | Opus 5.5 | Explain | karpathy-skills | 1 | Claude Code 2.1.284 | off | passed | 1,176 | 230 | 4 | 7,662 | 28,112 | $0.0905 | 0 | 14.6 | |
| 13 | Opus 5.5 | Explain | karpathy-skills | 2 | Claude Code 2.1.284 | off | passed | 1,420 | 374 | 4 | 7,661 | 28,111 | $0.0953 | 0 | 14.7 | |
| 13 | Opus 5.5 | Explain | karpathy-skills | 3 | Claude Code 2.1.284 | off | passed | 1,223 | 109 | 4 | 7,659 | 28,109 | $0.0914 | 0 | 12.7 | |
| 13 | Opus 5.5 | Explain | my control file | 1 | Claude Code 2.1.284 | off | passed | 1,066 | 157 | 4 | 7,393 | 27,845 | $0.0860 | 0 | 13 | |
| 13 | Opus 5.5 | Explain | my control file | 2 | Claude Code 2.1.284 | off | passed | 1,210 | 230 | 4 | 7,396 | 27,847 | $0.0890 | 0 | 12.7 | |
| 13 | Opus 5.5 | Explain | my control file | 3 | Claude Code 2.1.284 | off | passed | 1,227 | 212 | 4 | 7,391 | 27,843 | $0.0893 | 0 | 13.3 | |
| 13 | Opus 5.5 | Explain | ponytail | 1 | Claude Code 2.1.284 | off | passed | 1,294 | 248 | 4 | 7,680 | 28,128 | $0.0930 | 0 | 13.9 | |
| 13 | Opus 5.5 | Explain | ponytail | 2 | Claude Code 2.1.284 | off | passed | 1,272 | 305 | 4 | 7,671 | 28,121 | $0.0924 | 0 | 15.7 | |
| 13 | Opus 5.5 | Explain | ponytail | 3 | Claude Code 2.1.284 | off | passed | 1,010 | 100 | 4 | 13,951 | 20,236 | $0.136 | 0 | 13.5 | |
| 13 | Sonnet 5 | Explain | caveman | 1 | Claude Code 2.1.284 | off | passed | 289 | 17 | 4 | 12,807 | 49,249 | $0.0640 | 0 | 5.9 | |
| 13 | Sonnet 5 | Explain | caveman | 2 | Claude Code 2.1.284 | off | passed | 291 | 15 | 4 | 12,808 | 49,251 | $0.0640 | 0 | 6.3 | |
| 13 | Sonnet 5 | Explain | caveman | 3 | Claude Code 2.1.284 | off | passed | 239 | 0 | 4 | 12,791 | 49,250 | $0.0634 | 0 | 4.9 | |
| 13 | Sonnet 5 | Explain | No skill | 1 | Claude Code 2.1.284 | off | passed | 984 | 27 | 6 | 10,197 | 74,452 | $0.0655 | 0 | 12.8 | |
| 13 | Sonnet 5 | Explain | No skill | 2 | Claude Code 2.1.284 | off | passed | 744 | 11 | 6 | 10,235 | 74,580 | $0.0633 | 0 | 9.9 | |
| 13 | Sonnet 5 | Explain | No skill | 3 | Claude Code 2.1.284 | off | passed | 976 | 20 | 6 | 10,359 | 74,786 | $0.0662 | 0 | 10.8 | |
| 13 | Sonnet 5 | Explain | karpathy-skills | 1 | Claude Code 2.1.284 | off | passed | 745 | 7 | 4 | 11,166 | 47,617 | $0.0616 | 0 | 8 | |
| 13 | Sonnet 5 | Explain | karpathy-skills | 2 | Claude Code 2.1.284 | off | passed | 785 | 20 | 4 | 11,182 | 47,620 | $0.0621 | 0 | 8.9 | |
| 13 | Sonnet 5 | Explain | karpathy-skills | 3 | Claude Code 2.1.284 | off | passed | 907 | 36 | 6 | 11,394 | 76,835 | $0.0700 | 0 | 11.6 | |
| 13 | Sonnet 5 | Explain | my control file | 1 | Claude Code 2.1.284 | off | passed | 963 | 35 | 6 | 11,121 | 76,298 | $0.0694 | 0 | 12.1 | |
| 13 | Sonnet 5 | Explain | my control file | 2 | Claude Code 2.1.284 | off | passed | 743 | 11 | 6 | 11,129 | 76,309 | $0.0672 | 0 | 8.5 | |
| 13 | Sonnet 5 | Explain | my control file | 3 | Claude Code 2.1.284 | off | passed | 724 | 24 | 6 | 11,126 | 76,305 | $0.0670 | 0 | 9.8 | |
| 13 | Sonnet 5 | Explain | ponytail | 1 | Claude Code 2.1.284 | off | passed | 749 | 16 | 6 | 11,400 | 76,851 | $0.0685 | 0 | 11.8 | |
| 13 | Sonnet 5 | Explain | ponytail | 2 | Claude Code 2.1.284 | off | passed | 767 | 42 | 6 | 11,410 | 76,862 | $0.0687 | 0 | 9.3 | |
| 13 | Sonnet 5 | Explain | ponytail | 3 | Claude Code 2.1.284 | off | passed | 917 | 32 | 6 | 11,420 | 76,871 | $0.0702 | 0 | 11.8 | |
| 13 | Sonnet 5.5 | Explain | caveman | 1 | Claude Code 2.1.284 | off | passed | 694 | 0 | 4 | 8,919 | 28,582 | $0.0483 | 0 | 6.6 | |
| 13 | Sonnet 5.5 | Explain | caveman | 2 | Claude Code 2.1.284 | off | passed | 603 | 0 | 4 | 8,916 | 28,579 | $0.0474 | 0 | 7.3 | |
| 13 | Sonnet 5.5 | Explain | caveman | 3 | Claude Code 2.1.284 | off | passed | 621 | 0 | 4 | 8,922 | 28,584 | $0.0476 | 0 | 6.3 | |
| 13 | Sonnet 5.5 | Explain | No skill | 1 | Claude Code 2.1.284 | off | passed | 909 | 0 | 4 | 6,178 | 25,845 | $0.0390 | 0 | 8.7 | |
| 13 | Sonnet 5.5 | Explain | No skill | 2 | Claude Code 2.1.284 | off | passed | 864 | 0 | 4 | 6,268 | 25,932 | $0.0389 | 0 | 7.5 | |
| 13 | Sonnet 5.5 | Explain | No skill | 3 | Claude Code 2.1.284 | off | passed | 841 | 0 | 4 | 6,266 | 25,931 | $0.0387 | 0 | 7.8 | |
| 13 | Sonnet 5.5 | Explain | karpathy-skills | 1 | Claude Code 2.1.284 | off | passed | 900 | 0 | 4 | 7,289 | 26,951 | $0.0436 | 0 | 11 | |
| 13 | Sonnet 5.5 | Explain | karpathy-skills | 2 | Claude Code 2.1.284 | off | passed | 845 | 0 | 4 | 7,289 | 26,951 | $0.0430 | 0 | 8.9 | |
| 13 | Sonnet 5.5 | Explain | karpathy-skills | 3 | Claude Code 2.1.284 | off | passed | 857 | 0 | 4 | 7,290 | 26,952 | $0.0431 | 0 | 7.4 | |
| 13 | Sonnet 5.5 | Explain | my control file | 1 | Claude Code 2.1.284 | off | passed | 788 | 0 | 4 | 7,020 | 26,684 | $0.0413 | 0 | 8.9 | |
| 13 | Sonnet 5.5 | Explain | my control file | 2 | Claude Code 2.1.284 | off | passed | 848 | 0 | 4 | 7,021 | 26,685 | $0.0419 | 0 | 9.7 | |
| 13 | Sonnet 5.5 | Explain | my control file | 3 | Claude Code 2.1.284 | off | passed | 732 | 0 | 4 | 7,025 | 26,688 | $0.0408 | 0 | 7.3 | |
| 13 | Sonnet 5.5 | Explain | ponytail | 1 | Claude Code 2.1.284 | off | passed | 842 | 0 | 4 | 7,306 | 26,967 | $0.0430 | 0 | 8.4 | |
| 13 | Sonnet 5.5 | Explain | ponytail | 2 | Claude Code 2.1.284 | off | passed | 774 | 0 | 4 | 7,301 | 26,963 | $0.0423 | 0 | 10 | |
| 13 | Sonnet 5.5 | Explain | ponytail | 3 | Claude Code 2.1.284 | off | passed | 718 | 0 | 4 | 7,297 | 26,960 | $0.0418 | 0 | 11.1 |
The list has 455 runs: the 372 counted above, 45 that round 13 replaced (‡) and 38 from rounds 8, 9 and 10. Rounds 1 to 4 ran Claude Sonnet 5 and GPT-5.6 Luna. Round 5 rechecked their explain task. Rounds 6 and 7 ran Claude Opus 5.5 and GPT-6 Luna. Round 8 ran more models on the explain task, plus one no-skill explain run each on Claude Opus 5.5 and Claude Sonnet 5. Round 9 ran Claude Sonnet 5.5, with Sonnet 5 and Opus 5.5 beside it on the explain task. Round 10 ran GPT-6.1 Sol, with GPT-6 Sol and Opus 5.5 beside it on the explain task. Rounds 11 and 12 were the first real app work, on the Models page. Round 13 reran the Claude models' explain task with the account extras off. A double dagger (‡) marks a run that round 13 replaced: it stays in this list and out of the charts. Output includes hidden thinking. Claude input counts uncached tokens only, while Codex CLI input includes cached tokens. Cost is at API prices. A star marks a light run, one that started with fewer tokens of instructions and tools than the model's other runs.
Tokens by model
Pick a task and a skill to see what each model read and wrote with it. These are the runs behind the charts above. The Claude explain rows come from round 13, and every other row ran earlier with the account extras on, both Lunas at low effort. The Models page uses two no-skill explain runs per model, so its medians differ from these.
Total tokens#
Everything the model read and wrote in the whole run
Cost at API prices#
What the run's tokens cost at the model's published API prices, cached tokens at the cache price. GPT bars are the lower bound: Codex logs no cache writes, which OpenAI prices up to a quarter above input
Tokens read#
All the text sent to the model, counted each time it was sent, with Codex CLI's hidden first request
Read from cache#
Share of what it read that came from the cache, which is cheaper. Counts Codex CLI's hidden first request
Cache writes#
Text saved to the cache for reuse. Codex CLI always reports 0, so for Codex CLI I count what it read that did not come from the cache
Output tokens#
Everything it wrote: the answer, its tool calls and its hidden thinking
Hidden thinking#
The part of the output spent thinking, which the answer doesn't show
The table scrolls sideways on a small screen.
| Model | Passed | Tokens read | From cache | Cache writes | Output | Thinking | Visible | Cost at API prices |
|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5Claude Code 2.1.284, high effort, Claude Code's default | 3 of 3 | 84,821 | 88% | 10,235 | 976 | 20 | 956 | $0.0655 |
| Claude Opus 5.5Claude Code 2.1.284, medium effort, Claude Code's default | 3 of 3 | 33,739 | 80% | 6,641 | 1,167 | 179 | 988 | $0.0819 |
| Claude Sonnet 5.5Claude Code 2.1.284, medium effort, Claude Code's default | 3 of 3 | 32,201 | 81% | 6,266 | 864 | 0 | 864 | $0.0389 |
| GPT-5.6 LunaCodex CLI 0.153.3, low effort, set by me | 3 of 3 | 32,584 | 61% | 12,616 | 240 | 14 | 226 | $0.00321 to $0.00384 |
| GPT-6 LunaCodex CLI 0.156.0, low effort, set by me | 2 of 3 | 39,358 | 61% | 15,294 | 180 | 0 | 180 | $0.00186 to $0.00224 |
| GPT-6.1 SolCodex CLI 0.159.0, medium effort, set by me | 3 of 3 | 56,019 | 72% | 15,443 | 286 | 0 | 286 | $0.0378 to $0.0455 |
Other rounds
Newer models ran the explain task with caveman, and Claude Sonnet 5.5 ran every task on its launch day.
More models on the explain task
In round 8, Claude Fable 5.1 and five GPT models ran the explain task with caveman in newer releases of Claude Code and Codex CLI. Both Lunas ran again at medium effort. After one run to confirm it could read the file, each model ran twice with no skill and twice with caveman. These runs stay out of the charts above, where both Lunas ran at low effort.
| Model | No skill | caveman | Change |
|---|---|---|---|
| Claude Fable 5.1 | 1,134* | 869* | −23% |
| GPT‑5.6 Luna | 230 | 210 | −9% |
| GPT‑6 Luna | 181 | 144 | −20% |
| GPT‑5.6 Sol | 251 | 193 | −23% |
| GPT‑6 Sol | 228 | 156 | −32% |
| GPT‑6 Astra | 276 | 185 | −33% |
All 24 runs passed. Claude Code 2.1.283, high effort, Claude Code's default. Codex CLI 0.157.1, medium effort, set by me.
* At least half the runs behind a starred number are light. Light runs start with fewer tokens of instructions and tools than the model's other runs, and some ended with a remark about the account's connectors, so their counts differ.
caveman cut visible output 9% to 33% on these models, never near its claimed 65%. Claude Sonnet 5 is still the only model where the claim held. With 2 runs a side, read these as direction.
The Models page runs these models again with no skill, in round 13 on newer releases.
Sonnet 5.5 on launch day
Claude Sonnet 5.5 came out on 28 September 2026, and that day it ran all four tasks with each skill, three runs apiece. Its coding tasks on this page come from that day. Its explain task comes from round 13. Its speed and token use with no skill, beside Sonnet 5 and Opus 5.5 in the same Claude Code release, are on the Models page. I logged every request Claude Code sent to check that its token counts add up.
Changes
- I wrote the plan before the first run.
- After two trial runs, before any counted run, I changed the explain check to the words ValueError and month.
- Round 1 ran 24 runs.
- I corrected the first write-up, which pooled the tasks and misquoted caveman's claim.
- I logged round 2 before running it: 36 runs.
- I logged round 3 before running it. It added karpathy-skills and the build task and brought ponytail to three runs per task.
- I logged round 4 before running it. It added my control file, which I lengthened twice before its first run.
- Rounds 1 to 4 finished with 144 runs, no errors and every run passed its test. 120 are shown here; the other 24 tested cutweight.
- I took cutweight, a skill I make, off this page, so the page only measures other skills against their own claims.
- I logged round 5 before running it: 18 explain-task runs to recheck the largest effects, 12 shown here. Every run now records a hash of its instruction file, and every hash matched.
- Two uncounted trial runs checked the model names of Claude Opus 5.5 and GPT-6 Luna. Then I logged round 6: 12 explain-task runs on those models, with the CLI versions released the same day.
- I logged round 7 before running it: 108 runs that give Claude Opus 5.5 and GPT-6 Luna every task and setup. GPT-6 Luna failed 9 of its 15 explain runs by answering without reading the file.
- I checked GPT-6 Luna's failed explain runs. Its shell tools worked, and none of four harness changes stopped it answering without reading the file, so the failures stay on the page.
- Each claim is now checked with the skill's own measure: visible output tokens for caveman, and lines added, total tokens, cost and time for ponytail. I had compared ponytail's token claim with output tokens.
- Round 8, on Claude Code 2.1.283 and Codex CLI 0.157.1 with Codex CLI at medium effort. One explain run per model first checked that each model reads the files, then 2 runs without a skill and 2 with caveman for Claude Fable 5.1, GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol, GPT-6 Sol and GPT-6 Astra. GPT-6 Luna read the file on every run. The earlier rounds stay as they ran, at low effort for Codex CLI. Run dates on this site are in UTC.
- Added the Models page at /numbers. It shows every model's input tokens, cache reads and writes, output, hidden thinking, time and output per second side by side, for round 8 and for each task and skill in rounds 1 to 7.
- Codex CLI token counts now include the hidden first request Codex CLI leaves out of its own count, 9,751 to 12,371 tokens a run, measured for each model and CLI release. Codex CLI cache writes, which Codex CLI always reports as 0, are now the prompt tokens it did not read from cache.
- Round 9 ran Claude Sonnet 5.5 on its launch day, with Sonnet 5 rerun on the same Claude Code release the same day, on the explain task.
- Sonnet 5.5 ran every task and skill, so round 9 now shows each skill's own claim on it and a full results table.
- Opus 5.5 ran the explain task twice on Claude Code 2.1.284, beside Sonnet 5.5 and Sonnet 5 on the same release. The Models page adds a total tokens chart and link anchors on every section and chart.
- Added an effort table: the effort Claude Code sent for each Claude model, logged from its requests, and the effort I set for each Codex CLI model.
- Timed every model on the explain task, two runs each, with the same method for Claude Code and Codex CLI: total time split into getting ready, waiting on the model and running tools, plus time to first token and when the written answer starts. The Models page now opens with a sortable leaderboard.
- New layout: section links in a bar at the top, each section's title above its intro, and tables at the full page width like the charts.
- The site now has two pages. Skills tests each skill against the same task with no skill, and Models shows every model with no skill. The per-skill token charts moved to Skills and the effort table to Models, and old links forward to the new place.
- The Sonnet 5.5 chart on the Models page now names the effort each model ran at: medium for Sonnet 5.5 and Opus 5.5, high for Sonnet 5, each Claude Code's default.
- The Models page now opens with the leaderboard under a plain title and one sentence on the task. The table keeps time to finish, tokens used and share from cache, and fits a phone; first token and answer start are in the speed charts below it.
- GPT-6.1 Sol came out during OpenAI's DevDay, and I ran it that day at medium effort on Codex CLI 0.159.0, the first release that runs it on a ChatGPT login: 2 explain runs with no skill, 2 with caveman and 2 timed runs. GPT-6 Sol reran on the same release and Opus 5.5 on Claude Code 2.1.284 the same day. All three are on the leaderboard and the overall chart with these runs.
- Every run on the Models page now shows what its tokens cost at the model's published API prices, cached tokens at the cache rate, with the price table and its sources. For Claude it matches Claude Code's own estimate on all 194 Claude runs.
- GPT-6.1 Sol joins the skill runs: every task with every skill, three runs each, at medium effort on Codex CLI 0.159.0. It is in every chart and table on the Skills page, and the claim counts, ranges and grid counts now include it.
- The Models page now says what the test is, under the leaderboard: the explain task, how it passes, and that it writes no code. The coding tasks are on the Skills page.
- For the coding tasks, a run's tests are the only quality measure. Extra tests with new inputs, written after the runs, reran every coding run whose changes I kept, and all passed. How a run works says so.
- Added real app work to the Models page: a bug hunt with 20 planted bugs and a tickets test with 17 hidden tests, in a small TypeScript web app, 3 runs each for Opus 5.5, Sonnet 5.5, GPT-5.6 Sol, GPT-6 Sol and GPT-6.1 Sol, graded by hidden tests, with cost at API prices and time.
- Correction: caveman's README dropped its 65% figure on 8 September 2026, before my runs. Its GitHub description still says it cuts 65% of tokens, so that is the claim I check, and its card now says the README no longer does. karpathy-skills' card links its CLAUDE.md.
- Correction: the effort table left out two settings the runs used, Claude Fable 5.1 on Claude Code 2.1.284 in the timed runs and GPT-5.6 Sol on Codex CLI 0.159.0 in the real app work. Both are listed now, and QA fails if a run's release is ever missing again.
- Redesigned both pages. The Skills page opens with the count of claimed cuts that held, one chart of each claim against the measured cut, and each skill's claim, finding and one-word verdict. The Models page leads with real app work. Text runs the full page width, long detail sits in folds, charts save as images with CSV, and bar charts show bars only.
- Correction: some Claude Code runs started with thousands fewer tokens of instructions and tools. They ran before Claude Code had synced the account's claude.ai plugins and skills and loaded its connectors, so they cost less and count fewer total tokens. A number where they make up at least half the runs now carries a star, a new limit explains how I found them, and I removed a remark about the connectors from the published answers. No claim check changes. The count of Claude runs in the 29 September cost entry is corrected from 254 to 194; it had counted Sonnet 5.5's runs twice.
- One GPT-5.6 Sol tickets run edited an existing test file, which the prompt said not to do. The grader restores the original tests first, so its 17 of 17 stood. Round 13 replaced that run, and no rerun edited a test.
- Round 13 reran the explain task for every model on the Models page, every Claude model's explain runs, all 30 real app work runs and the timed runs, all with the account extras off. Those numbers no longer carry stars, and Claude's explain-task token counts and costs dropped. caveman now cuts Sonnet 5.5's explanations 28%, where round 9 showed 13%, and Sonnet 5's 72%, still above its 65% claim. All five models still passed all 17 hidden ticket tests, and Opus 5.5 cost more, $0.96 a ticket run.
- The Skills page now prices GPT runs at OpenAI's published rates, so the GPT models appear in every cost comparison, and the headline counts 29 checks.
- Redesigned the findings, tables and colors. Each finding has its own framed chart with horizontal bars and a saved image, the skill cards and real-work tables line up and sort like the Models page table, each vendor keeps one color family, and dark mode is warm.
- Correction: the Skills page gave Sonnet 5.5's release day as 30 September, when it came out on 28 September. Two table notes still said cost covered Claude only, and the effort table left out the Codex CLI 0.159.0 row for three GPT models. All three are fixed, and QA now reads every run group.
- Correction: the recheck table labeled Sonnet 5's round 13 runs as rounds 1 to 4, and some chart panels named an older Claude Code release than their runs used. The table now shows all three sessions of caveman on Sonnet 5, at 66%, 62% and 72%, and every release and round label comes from the runs. The run list adds the runs round 13 replaced, marked, with each run's CLI and account extras.
- The chart for any measure and task now uses the same framed horizontal bars as the findings, with a saved image. The Models page notes how much round 13 moved its numbers and has its own list of changes, and on a phone its table stays a table.
- Correction: the page gave three sets of caveman runs on Claude Sonnet 5's explain task and left out round 9's, which cut 64%. All four are shown now, 66%, 62%, 64% and 72%, and two fell under the 65% claim. The claim still counts as held, because the charts' own runs reached it.
- The chart for any measure and task is now its own section, Every measure and task. Charts with three skills stack at full width with a labeled scale, bars from failed runs show hollow, and the Models page notes every model the round 13 timing slowed.
- Correction: 12 Claude coding runs also started light. Until now I had checked only the explain task. They now carry a star in the run list, and no median has half its runs light. The Models table title gives the time spread with and without the slow OpenAI day, and the cost panels say their range adds up each task's cheapest to costliest run.
- On a phone, bar charts put each model's name above its bar. Saved images keep their panels side by side, the Models token charts are horizontal bars, and the Skills page adds a table of every model's cut under the top chart.
- The set-by-set table now shows every model with more than one clean set of caveman runs. In round 13, with the account extras off, caveman cut more on every Claude model than in any earlier set.
- The GPT models that ran slower in round 13's timing carry a dagger in the explain table and the time chart, and the real app work note says their runs slowed the same day. Costs use one format on every table and chart.
- Bar charts with the name above each bar leave a clear gap between rows, and their lines no longer cross a label. Grid cells that could be chance are outlined like the bars. The cut table names every model and marks the one cut that met its claim.
- The set-by-set table adds GPT-6 Sol, from rounds 8 and 10, and stars a median that comes from a light run. The Every measure and task title counts only clear cuts, the filled bars.
- Costs use one format everywhere, three significant figures under ten cents and three decimals above. The cost-vs-time title and dots mark the slowed GPT models, and the run list adds round 10.
- The cut table fits tablets and phones, where each claim becomes a block of model and cut pairs. Saved images explain their daggers, small cost bars get their own scale, and tables and charts in folds share one title style.
- Round 8's table now stars its light medians. The real-work title and note use the table's cost figures, and the cost-vs-time title compares cost only. The page says a claim held, in one word, everywhere.
- Output and thinking counts carry the light-run star too. The slowed GPT models are marked § instead of a dagger. The cut table says whether a cell was not compared or failed, and dates and model names no longer break across lines on phones.
- The page reports each claim from the latest round. caveman on Claude Sonnet 5 cut 72% in round 13, and its earlier sets, at 62% to 66%, stay in the set-by-set table.
- The claimed-against-measured chart uses bars like the other charts: each claim's average cut against a light track that ends at the claimed number, with each model's cut in the table under it.
- The tick strip and the byline are gone: the chart under the intro shows the same checks, and its source line carries the date.
- The real app work chart and the Models board views drop the rings and the scores on each dot, and the rerun note is two sentences.
- The real app work chart says what its tests can't do: every model scored at or near full marks, so they show cost and time on a small job, not which model is best.
- A dot on the real app work chart names its score when it fell short of full marks, as Claude Sonnet 5.5's bug hunt dot does at 19 of 20.
- New page, GPT-6.1 Sol timed three days running: one short Python file explained on launch day, the day after, and on 1 October after OpenAI said it added capacity. The Models board keeps the 30 September runs.
- The GPT-6.1 Sol page says what the test is up front and groups the chart by day. The ChatGPT apps and plugins setting moved to the method, in plain words.
- The GPT-6.1 Sol page adds four runs from the afternoon of 1 October, after @sama said it should be much better: 13.5 to 16.8 s, faster than that morning and still slower than launch day.
- New page, Karpathy's ASD-STE100 tip, measured: six models with his tip as a one-line instruction on the explain task, against no instruction in the same setting.
- The ASD-STE100 chart uses one bar color for every model; a change the runs don't clearly show now says "could be chance" in its label instead of a pale outline.
- The ASD-STE100 chart title counts every model whose answers got shorter, then how many of those clearly.
- The GPT-6.1 Sol page adds four runs from the afternoon of 3 October, after a 2 October post on X still called it slow: 15.6 to 26.6 s, slower by median than the afternoon of 1 October. Corrected 5 October: the two afternoons' runs overlap.
- The ASD-STE100 page adds a chart of ASD-STE100 beside caveman, ponytail, karpathy-skills and my control file on the same explain task, each against its own no-instruction runs.
- The ASD-STE100 chart's longest labels no longer run off its right edge on tablet and small laptop screens.
- The GPT-6.1 Sol page adds four runs from the afternoon of 4 October: 13.2 to 22.9 s, all slower than launch day.
- The GPT-6.1 Sol page adds four runs from 5 October, before OpenAI's speed post: 14.1 to 18.6 s, all slower than launch day.
- The GPT-6.1 Sol page adds four runs from 5 October, minutes after OpenAI said its speed change would be felt within two hours: 14.3 to 19.5 s.
- Readers were right that the GPT-6.1 Sol page's tokens a second was not TPS. It divides output by the time spent waiting on the model's replies, so the column is now tokens a second of waiting.
- Headlines say a setup cut more, won or held its claim only when its runs sit apart from the other side's. Where runs overlap they now say by median, so three headlines changed, and the all-tasks totals add up each side's full run range.
- The GPT-6.1 Sol page adds four runs from a 5 October retest, still inside the two hours OpenAI gave: 11.3 to 20.8 s, three of them inside launch day's range.
- A TPS column briefly showed a rate from an OpenAI server-side timing field. That field times the engine, not delivery: on launch day it read 3.7 to 3.9 times the rate text reached my machine. It came down. A TPS proxy column now counts the pieces of answer text Codex logged a second, pauses included.
- Two test runs of the updated timing script on 5 October are listed under How I ran it on the GPT-6.1 Sol page, not charted.
- The GPT-6.1 Sol and ASD-STE100 pages said each run got a fresh container. Each run gets a fresh copy of the project; runs share the bench's container. The Sol page now says a run passes when its answer contains both "month" and "ValueError".
- The GPT-6.1 Sol page's intro said posts on X on 3 October called it slow again. No such post could be found, so it now cites the 2 and 4 October posts that did.
- The GPT-6.1 Sol page adds four runs from 5 October, after the two hours OpenAI gave: 11.4 to 17.7 s, streaming at 39.2 to 52.9 text pieces a second.
- Readers asked how launch day was as quick with a slower stream. The GPT-6.1 Sol page now says why: the answer's stream is under half of every run, launch day's answers were shorter, and the runs since that made two model requests, like launch day, finished inside launch day's range.
- Readers asked to sort the GPT-6.1 Sol run table. Click any column heading: Time puts the fastest first, TPS proxy the highest first, a second click reverses, and Day restores day order.
- The GPT-6.1 Sol chart took more than a page. It is now one row per sitting in two panels, TPS proxy and seconds, each a median with launch day and the sittings since the 5 October retest picked out; the 5 October sittings are named by OpenAI's post, not the time of day; every run stays in the table. The intro is one line, and the background folds under How I ran it.
- Readers asked whether the GPT-6.1 Sol page has true TPS. The table now has TPS by OpenAI's own token count for the eight runs since 5 October's retest: the answer's output tokens over the seconds from its first text event to its last, a rate observed at my machine. Earlier rows have no count and stay blank; two script test runs also logged it, at 36.2 and 43.4, and are in the CSV. The text-event rate is now labeled text pieces a second.
- Readers couldn't find the GPT-6.1 Sol speed page from the main pages. It now has its own tab in the bar on every page (Sol on a phone), and the Models page's time section links it.
- The Skills page's shorter-answers section now links the ASD-STE100 page, which had no link from the main pages.
- The GPT-6.1 Sol page adds four runs from 6 October: 10.3 to 15.7 s, the fastest run yet at 10.3 s, and 43.4 to 51.2 tokens a second by OpenAI's count. Every sitting from 5 October's retest on now counts as since the retest.
- A reader asked for Claude Opus 5.5 and Claude Sonnet 5.5 beside the GPT-6.1 Sol test. The Sol page now shows them timed the same day and task, 4 runs each, start to finish in each model's own tool: Sonnet 5.5 7.8 to 9.8 s, Sol 10.3 to 15.7 s, Opus 5.5 15.2 to 17.7 s. No TPS for the Claude models: this bench's Claude Code capture does not give a streaming rate yet.
- Readers loved the speed race videos on X, so the GPT-6.1 Sol page now has the same race, drawn live from every run: one lane per sitting, all leaving together at 5x with real seconds on the clock. It plays once when it scrolls into view and has a Replay button; with reduced motion it waits for Play.
- Every section of the GPT-6.1 Sol page has a heading with a # link that copies its address, and an On this page row jumps between them: every sitting, the race, same day with other models, and how I ran it.
- Readers said the race's dots were softer than the videos'. They are now drawn at full screen density with a sharp edge, a glow sized to the dot and the videos' white highlight.
- The site now counts visits with Cloudflare Web Analytics, which sets no cookies. The footer, which said no analytics, now says so.
- Each page now ships its text in the HTML, so search engines and AI tools that do not run scripts can read it. The site also has a sitemap, an llms.txt and a footer line linking every page.
- Readers said the link cards on X did not show the site well. Each card now has the site's bar with its page marked, one finding and one chart, sized to read in a feed; the GPT-6.1 Sol card shows the streaming chart alone.
- The GPT-6.1 Sol page's method now says the group since 5 October's retest was set after seeing the 5 October runs, and every later sitting joins it whatever it shows. The link cards have larger chart text.
- New page: Claude Haiku 5.5 against GPT-6 Luna, run the day Haiku 5.5 launched, at medium and xhigh effort: speed with TPS at the terminal, the token-saving skills, ASD-STE100, real app work and cost, with the plan and every change to it published.
- Haiku vs Luna: TPS now has its own test, a long answer to the same task. More than half of GPT-6 Luna's short answers arrived in a few bunches, too few to time, so the page's 58.5 and 61.0 came only from the runs that streamed word by word; on the long answer it is 122 at medium and 126 at xhigh, Haiku 5.5 239 and 297.
- Haiku vs Luna: results use passed runs. The failed runs, all GPT-6 Luna at medium answering without trying a tool, are counted under How I ran it, with a probe showing Codex had given it its tools.
- Haiku vs Luna: the skills claims table is now two bar charts, text output and code output, and ASD-STE100 is a bar chart.
- Haiku vs Luna: readers asked for real app work higher up, so it now comes right after TPS, before the skills tests.
- Haiku vs Luna: readers found "by median" confusing, so titles now say typical run, usually or close, and charts say each bar is the midpoint of the runs.
- New page: Gemini 4 Argon on launch day, in Cursor CLI, beside Claude Opus 5.5 and GPT-6.1 Sol, with a speed sitting where the three took turns and every launch-suite check.
- New page: the Deep suite, eight harder tasks from three public benchmarks (features, bug fixes, speed-ups, security fixes), three runs each for Claude Opus 5.5, GPT-6.1 Sol, Claude Haiku 5.5 and GPT-6 Luna.
- Deep suite: the headline now names the models within one run of each other, and the task table says what each model got, when a run passes and shows one square per run. The task sets are credited once, in the method.
- Deep suite: a cost and time table per model at list prices, after a reader asked for one.
- Deep suite: shorter text; speed-ups and security fixes are now one row per model, and the runs chart is one sorted bar chart, after readers found the old tables hard to follow.
- GPT costs on every page are now a range, from OpenAI's input rate to its cache-write rate, because Codex logs no cache writes. Fable, Grok, Astra and Gemini Argon agreed on this. Claude costs are unchanged.
- Deep suite: one look for every chart. Each is framed with its finding as the title, the models are always the columns in one order, and every grid uses the same run squares.
- Deep suite: plain words throughout. A try is one attempt at a task, and the page now says passed instead of solved, after readers found the text confusing.
- Deep suite version 2 is published: my own tasks in one real app, three areas, three tries each. Version 1 moved to /deep-suite/v1, and /deep-suite shows the newest version.
- Deep suite version 3 is published: six of my own tasks in six languages, two tries each, each task scored on its own. Version 2 moved to /deep-suite/v2, and /deep-suite shows the newest version.
- The site has a home page and four sections in the bar: Models, Skills, Deep suite and Launch days, after readers found the navigation confusing. The Skills page moved to /skills, and old links forward. Deep suite v3 adds a chart of score against cost with an estimated frontier, and bars no longer carry claim or reference lines.