How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
OpenAI Blog · 2026-07-29
OpenAI reveals that enabling retained reasoning and context compaction in its Responses API tripled GPT-5.6 Sol's scores on the ARC-AGI-3 benchmark while cutting output tokens by 6x, demonstrating how harness design choices can dominate reported model performance.
Appears in
Extraction
Topics: benchmark-methodologyarc-agi-3reasoning-retentioncontext-managementmodel-evaluation
Claims
- GPT-5.6 Sol initially scored only 7.8% on ARC-AGI-3 because the benchmark harness discarded private reasoning between actions and applied rolling truncation to conversation history.
- Retaining reasoning across turns allowed the model to build on prior thoughts rather than reanalyzing each game from scratch, reducing thinking time per action.
- Replacing rolling truncation with compaction preserved learned game knowledge across longer runs and reduced output token usage.
- Together, retained reasoning and compaction tripled ARC-AGI-3 scores while cutting output tokens by 6x.
- Benchmarks measure not just model capability but also the specific API settings, harness design, and prompting choices used by the evaluation runner.
Key quotes
Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.
Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.
This isn't the first time we've been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.