---
identifier: draft-imran-reading-a-benchmark-00
section: "5.20"
title: "How to Read a Benchmark Number"
summary: "A published score is the product of a task set, an answer key, a retry policy and a harness. Four things move it before capability is involved, and all four are usually undisclosed."
state: holding
confidence: 0.85
published: 2026-09-01
revised: 2026-09-01
expires: 2027-03-05
retires:
- "A widely quoted agent benchmark that publishes, as a matter of routine, its answer-key audit rate, its contamination analysis, the attempt count behind every headline figure and the harness used. If disclosure becomes normal, this paper is describing a solved problem."
- "Evidence that leaderboard rank predicts production outcome well enough to use directly: several teams whose model choice by leaderboard matched their choice by task-specific evaluation, across different task types."
- "A demonstration that pooled multi-attempt scores and single-attempt scores rank models identically in practice, which would make the distinction in Section 4 pedantic rather than load bearing."
- "An audit of a major benchmark finding its answer key substantially correct, suggesting the SWE-bench Verified result is an outlier rather than what happens when anybody looks."
license: https://creativecommons.org/licenses/by/4.0/
---
Abstract. Somebody sends you a chart and you make a decision from it.
This paper is about the four things sitting between that number and anything you could call
capability. The answer key can be wrong, and on the most quoted coding benchmark in the
industry an audit found flawed tests in over 59 percent of the sampled problems, which is
why OpenAI stopped reporting against it. The model may have seen the answers. The headline
may be several attempts pooled rather than one. And the harness is a co-author of the result,
which is paper 5.18. None of the four is a scandal. All four are undisclosed by default.
Confidence 0.85. The examples are strong and public. The gap from 0.95 is that the
prescription in Section 6 is more expensive than I make it sound.
## 1. The claim
Nobody in this story is cheating. That is what makes it worth writing down.
A benchmark number is produced by a pipeline: a set of tasks, an answer key that decides
what counts as correct, a policy for how many attempts a model gets, and a harness that
decides what the model can see and do. A capability difference between two models is one
input to that pipeline. The claim is that the other four inputs move the output by more than
the model gap you are trying to read, and that they are almost never published alongside the
number.
So when you compare 62 against 55, you are not necessarily comparing two models. You may be
comparing two attempt policies, or one contaminated model against one clean one, or two
harnesses, and the arithmetic will look exactly the same in every case.
## 2. The answer key can be wrong
This is the one that should change how you read every chart you see, because it happened on
the benchmark everybody quotes.
SWE-bench Verified is the standard for agentic coding. OpenAI audited 138 of its problems,
roughly 27.6 percent of the set, concentrating on ones models often failed. **At least 59.4
percent of the audited problems had flawed tests.** Forty-nine tests were too narrow and
rejected functionally correct submissions. Twenty-six were too wide and demanded behaviour
the issue never asked for. OpenAI's conclusion was to stop evaluating against it and publish
why.
Read the direction of that failure carefully. Tests that are too narrow reject correct work,
which means the unsolved pile was never entirely a pile of model failures. Some fraction of
every "the model could not do this" was "the grader would not accept it". Separately, a study
found more than 15 percent of Verified instances carry incomplete test patches that let
wrong or partial solutions through, so the errors run in both directions at once. Work on
test adequacy suggested leaderboard scores may be inflated by six to seven points on that
basis alone.
Six to seven points is larger than the gap that decides most procurement arguments.
## 3. The model may have seen the answer key
SWE-bench Verified draws on public GitHub issues, and its 500 tasks and their resolutions
have been sitting in public repositories for years. Any model trained on GitHub data after
mid-2024 has plausibly read them, solutions included, and contamination has been reported
across frontier models generally rather than at one lab.
This is not a fixable oversight so much as a structural property of building benchmarks from
public data and then training on public data. A benchmark's usefulness decays from the day it
is published, and the decay is invisible in the number.
The same shape shows up in speech recognition. When reference transcripts contain errors,
models that have optimised against the benchmark reproduce the erroneous transcript rather
than what the audio actually says. Part of a leaderboard lead is memorising the key rather
than doing the task, and from the outside those two look identical.
## 4. The headline may be several attempts pooled
This one is the easiest to check and the most commonly missed.
Aikido ran ten models three times each against 32 freshly disclosed CVEs, asking each to
rediscover the vulnerability from source. DeepSeek V4 Pro's headline **28 of 32 is the union
of three runs**, not one. The write-up's own finding is that running a cheaper model a few
times and pooling reliably beats a single pass of a stronger, pricier one, and that DeepSeek
V4 Flash reached 24 of 32 across three runs at more than ten times less cost than a frontier
competitor.
That is a genuinely useful result about how to spend money. It is a different result from
"this model finds 28 of 32", and the second sentence is the one that travels.
The question to ask of any score is how many attempts bought it, because pass at three and
pass at one are different quantities with the same units. If your production path gives the
model one shot, a pooled number is not a forecast of anything you will experience.
Figure 1. The bar lengths are illustrative and deliberately not measured, because
the honest answer is that nobody publishes enough to draw this to scale. That is the
complaint.
## 5. Why this persists
Because every party is behaving reasonably.
Party
Incentive
Result
Benchmark authors
Ship something useful; auditing 500 tasks properly is a year of unfunded work
Keys go out with errors in them
Model vendors
Report the configuration that shows the model at its best, which is a normal thing to do
Attempt counts and harnesses go unmentioned
Readers
Need one number to end an argument in a meeting
The caveat gets dropped in the retelling
Everyone
Public data makes benchmarks cheap to build
The same data trains the models
The one genuinely encouraging thing in this paper is that OpenAI published the audit that
made its own strong results on that benchmark unusable. That is the behaviour you want, it
is rarer than it should be, and it is the source I would trust most in the whole argument
precisely because of who it costs.
## 6. Reading one properly
Four questions, in order of how much they change the answer.
#
Ask
Why it moves the number
1
How many attempts?
Pooling several runs is a different quantity in the same units. If your production path is single-shot, a pooled figure forecasts nothing.
2
Whose harness, and is it specified?
Same weights have moved 28 to 49 and 30 to 100 on harness changes alone.
3
Has the key been audited, and when?
The best-known one had flaws in a majority of audited problems.
4
How old is the task set?
Public benchmarks decay into training data. Age is a proxy for contamination.
If you cannot answer any of the four, you have a number and no idea what it is a number of,
and the correct move is to treat the leaderboard as a shortlist rather than a ranking. Take
the top three, then run your own thirty tasks in your own harness, which is the same
prescription as 5.17 and 5.18 and is starting to look like the only prescription this site
has.
I said in the abstract that this is more expensive than it sounds. Building thirty
representative tasks with a defensible answer key is genuinely a week of somebody's time, and
the reason everyone reaches for the leaderboard is that the leaderboard is free. The argument
is that a decision worth a year of inference spend deserves a week of somebody's time, and
that almost nobody is making that trade consciously.
## 7. The strongest objection
Benchmarks improved this field enormously, and a paper that makes people distrust them may do
more harm than the flaws do. A noisy shared yardstick beats everybody privately claiming
their model is best, which is the world benchmarks replaced and which was much worse.
I accept that and it is why the four questions in Section 6 are questions rather than a
recommendation to ignore scores. What fails here is measurement quoted
without its conditions, which is a reporting problem and a reading problem, and both are
fixable without burning the practice down.
## 8. What this paper does not claim
It does not claim SWE-bench is bad work. Auditing your own widely adopted benchmark and
publishing the flaws, as its ecosystem did, is how this is supposed to go.
It does not claim contaminated models are not useful. A model that memorised solutions to
real GitHub issues has memorised something genuinely valuable. It just is not the thing the
benchmark says it measured.
And it does not put a number on the total distortion. I drew Figure 1 without a scale on
purpose, because assigning sizes to those four bars would be inventing the exact kind of
figure this paper is complaining about.