The Harness Is Half the Solver

1. The claim

You read that model X scores 62 on some agentic benchmark and model Y scores 55, and you file that away as a fact about X and Y. What you actually have is a fact about X inside somebody’s scaffolding and Y inside somebody’s scaffolding, and where those were different scaffolds, the seven points you just learned may belong entirely to the code around the model.

This matters because the two halves are procured differently. You choose the model with a contract and a price per million tokens. You inherit the harness from whatever framework you picked in week one, and then never look at it again, because it does not appear on any comparison chart and nobody sells it to you.

The claim is narrow. The harness accounts for enough of the variance in agent results that attributing a benchmark score to the model alone is a measurement error, not a simplification.

2. What a harness actually is

Worth being concrete, because the word gets used loosely and the paper falls apart if you picture the wrong thing.

A model takes text in and emits text out. That is the whole of it. Everything else that makes an agent work is ordinary software somebody wrote, and it makes decisions the model never sees:

None of that is intelligence. All of it changes the score.

3. Three measurements

Sydney Lewis froze the weights. The study runs 169 tasks at a 20,480 token context cap with a fixed 480 second endpoint, and changes exactly one thing: how the harness manages context. The control feeds the full conversation in time order. The treatment mechanically shortens older tool results as the window fills and reacts when work repeats or stalls. Mean per-task fail-to-pass went from 28 percent to 49, and complete solutions from 43 to 72. The same frozen treatment lifted three more models of different design without any per-model tuning. The paper’s own conclusion is the sentence I would put on the wall: evaluations “should treat the model and harness together as the tested solver.”

Note what the treatment was. Trimming stale tool output and noticing a loop. That is a Tuesday afternoon of ordinary engineering, and it was worth twenty-one points.

NVIDIA published a perfect score and its own caveat. Claude Opus 5 inside NVIDIA’s AVO system completed the 25 environment public set of ARC-AGI-3, all 183 levels, at 100.00 RHAE in 6,624 environment actions. ARC Prize separately reports Opus 5 at about 30 percent on the public benchmark at high reasoning effort. NVIDIA says plainly that the two numbers came from different evaluation frameworks and are not a direct comparison, which is the honest thing to say and is also the entire point: when a harness can move a result that far, no two numbers from different harnesses are comparable, including the ones on the chart you are using to pick a vendor. The semi-private and private competition sets were not part of the run.

A scaffold that helped and hurt in the same study. The ledger-based manager and worker work runs on the 100 latest hard LiveCodeBench problems at a 128k cap. Qwen3.8-27B went from 63.0 percent single-call to 86.4 under the manager, a gain of 23.4. GPT-5.6-Terra went 77.0 to 85.0, which brought it within noise of Claude Fable 5’s single-call 87.4 at about a fifth of the cost, $11.71 against $61.11. And Qwen3.6-35B lost ground, between one and nine points, with reasoning off. The manager roughly triples the token bill.

Figure 2. Six numbers from three different studies on three different task sets. They are not comparable to each other and that is deliberate, because the point is the size of the movement within each pair, where the weights did not change.

4. It is not a free lever

The tempting reading is that harness work is cheap upside. Two of the three studies say otherwise.

The manager scaffold triples the token bill, so a 23 point gain arrives attached to a roughly threefold cost increase, and whether that trade is good depends on numbers only you have. Paper 5.17 is the arithmetic for that.

The same scaffold cost Qwen3.6-35B up to nine points. A harness is tuned, explicitly or accidentally, against some set of models, and a model that plans well internally can be actively harmed by a wrapper that plans for it. So a harness behaves like a component that has to be matched to a particular model, which is the opposite of how it gets adopted. Teams pick one and expect it to lift whatever they point it at.

And Lewis found the gap largely closes at a wide context window. The 21 point swing was measured at 20,480 tokens, where trimming matters enormously. Give the same setup room and the arms converge. Harness effects are largest exactly where you are constrained, which is where most production systems live and where almost no benchmark runs.

5. Why nobody fixes this

The confound survives because everybody publishing has a reason to leave it in.

Nobody in that table is lying. Each is answering the question they were asked, and the composition of four honest answers is a misleading one.

6. What to do about it on a Tuesday

The remedy is the same shape as the one in 5.17 and it costs about the same.

Run both candidates inside your own harness. Twenty or thirty tasks from your own logs, your context policy, your tools, your timeouts. This is the only comparison that answers the question you are actually asking, which is not “which model is better” but “which model is better inside the thing I have already built”.

Write down your harness once. Context policy, tool list, truncation rule, stall handling, stop conditions. One page. Then when a result surprises you, you have something to diff against. Most teams cannot answer “what do you do when a tool result is 400 lines” without reading the code, which is a strange thing not to know about a system you are buying models for.

Change one half at a time. Swapping the model and the framework in the same sprint produces a number you cannot attribute, which is exactly the error the whole paper is about, committed at home instead of in a press release.

Look at the harness before the model when results are bad. It is cheaper to change, you own it, and on the evidence above it has comparable leverage. The Qwen chat template work is the cleanest example of this shape: a shipped template produced an eighty percent plus premature turn abort rate, and flattening it fixed the behaviour. The weights were never the problem, and anybody debugging that by swapping models would have burned a week.

7. The strongest objection, unanswered

If harness effects are this large, then a benchmark that holds the harness fixed is not measuring anything you care about either, because it is measuring model-inside-that-harness and you will not use that harness. Standardising the scaffold makes the comparison internally valid and externally useless.

I do not have a good answer. The best I can offer is that a fixed, published harness at least lets you read the result correctly, and that the honest output of an agent benchmark may be a range across several scaffolds rather than a number. That is more work for the people running benchmarks and less satisfying for the people reading them, which is probably why it does not happen.

This objection is the reason the confidence value sits at 0.85 rather than higher.

8. What this paper does not claim

It does not claim models are interchangeable. Opus 5 at 91 percent in one pass in the ledger study is doing something the smaller models are not, and the gap between model generations is real.

It does not claim harness work always pays. It cost one model nine points and tripled a bill.

And it does not claim the vendors are behaving badly. NVIDIA published the caveat next to the headline. Lewis published the wide-window result that weakens the finding. The failure is at the point where four honest publications get read side by side by somebody making a decision, and that person is usually you.