The Same AI Model Scored 3x Higher. The Benchmark Harness Changed

OpenAI reports that the same GPT-5.6 Sol model went from 13.3% to 38.3% on the ARC-AGI-3 public task set. The model did not change. Two settings in the benchmark harness did.

The higher score came with another result: OpenAI says the model used six times fewer output tokens.

I did not run this benchmark, so these are OpenAI’s results, not mine. But the gap is large enough to make one point hard to ignore: a leaderboard measures the system around a model too.

What a benchmark harness actually does

A benchmark harness is the software that sits between a model and a test. It decides what the model can see, which tools it can use, how actions are returned, what stays in context, and what disappears when the context fills up.

Those choices are easy to treat as plumbing. They are part of the evaluation.

In the official ARC-AGI-3 harness, GPT-5.6 Sol could see its previous actions and short notes. Its private reasoning was discarded after each action. As the history grew, a rolling window also removed the oldest actions.

OpenAI rebuilt the test with its Responses API and changed both behaviors.

Setting 1: retain reasoning between actions

ARC-AGI-3 asks an agent to explore unfamiliar 2D games and infer the rules without instructions. That requires more than remembering which button it pressed. The agent needs to carry a working theory from one action to the next.

When reasoning is discarded, the model can see the action history but loses the plan that produced it. It has to reconstruct that plan on the next turn.

OpenAI’s harness retained reasoning by passing the previous response ID between turns. According to OpenAI, the model then spent less time working out each game again and followed more coherent strategies over longer runs.

This is not extra intelligence added to the model. It is continuity added to the interaction.

Setting 2: compact the useful history

The official harness used rolling truncation once its history exceeded 175,000 characters. The oldest messages disappeared first.

That policy is simple, but age is a poor proxy for usefulness. An early discovery about a game’s rules may matter hundreds of actions later.

Compaction takes a different approach. It turns the useful parts of a long history into a shorter representation, then continues with that summary in context. OpenAI says this helped GPT-5.6 Sol preserve what it had learned while using fewer output tokens.

The distinction matters for any long-running agent. A context window is not memory by default. The harness decides what survives.

The higher score does not settle the fairness question

The official ARC harness is intentionally generic. That makes different models easier to compare under one shared setup. OpenAI’s harness is closer to how GPT-5.6 Sol runs inside its own products.

These setups answer different questions.

The generic harness asks how models perform under the same restrictions. A production-style harness asks how well a model can perform when its native features are enabled.

Neither result should quietly stand in for the other. A vendor-optimized harness may show a model’s practical ceiling, but it can also introduce model-specific advantages. A generic harness may improve comparability while hiding capabilities users would normally enable.

That is why the harness needs to be visible. The score alone is incomplete.

What to check before trusting an AI leaderboard

Before comparing model scores, look for these details:

1. Exact model and version. A family name is not enough.
2. Tools and API settings. Check what the model could use during the run.
3. Retained state. Find out whether reasoning, tool results, and earlier actions survived each turn.
4. Context policy. Look for truncation, summarization, compaction, and the actual limit.
5. Prompt and budget. Compare the instructions, token budget, retries, and stopping rules.

If those details are missing, treat the ranking as a claim that still needs context.

OpenAI’s reported jump from 13.3% to 38.3% does not prove that every benchmark is wrong. It shows how much one result can move when the harness changes.

A model leaderboard is also a harness leaderboard. Read both before choosing the model.

Reporting, not firsthand testing. Source: OpenAI’s ARC-AGI-3 harness report, published July 29, 2026.