# Astra's ARC Score Moves 45.1 Points When the Harness Changes **Summary:** At high reasoning effort, GPT-6 Astra scored 54.8% with ARC Prize's Standard harness and 99.9% with the Provider Adapter. I want both numbers before choosing an agent because its memory system is part of the result. - Canonical: https://markhuang.ai/news/astra-arc-score-45-1-point-harness-gap - Language: en - Author: [Mark Huang](https://markhuang.ai/about) - Published: 2026-09-03 - Section: News - Tags: GPT-6 Astra, ARC-AGI-3, AI Agents, Agent Harnesses, Model Evaluation - Source: [ARC Prize](https://arcprize.org/blog/astra) - License: https://creativecommons.org/licenses/by-nc/4.0/ --- ![A luminous reasoning core enters the same abstract puzzle world through a fragmented memory path and a continuous memory path](https://cdn.markhuang.ai/news/astra-arc-score-45-1-point-harness-gap/hero.webp) *One model enters the same world twice. The path that preserves its discoveries reaches much farther.* [ARC Prize tested OpenAI's GPT-6 Astra](https://arcprize.org/blog/astra) on ARC-AGI-3 and got two answers to what looks like one question. At high reasoning effort, Astra scored 54.8% with ARC Prize's Standard harness and 99.9% with a Provider Adapter that preserves opaque reasoning state and compacts long conversations. That is a 45.1-point gap. I believe the near-perfect result is real, but the model weights cannot claim it alone. The result tells me Astra can learn unfamiliar environments unusually well when its working state survives. It also tells me that the memory contract around the model can decide whether I see that ability at all. Both facts matter more than the 99.9% headline if I am choosing an agent. ## The adapter changes what the score means ARC-AGI-3 puts agents into abstract, turn-based environments without explicit rules or goals. The agent must explore, infer how the world works, identify a target, and plan its actions. ARC Prize says people can solve all of the environments, which gives the benchmark a human reference instead of an arbitrary pass mark. The two harnesses keep the games, actions, limits, and scoring fixed, according to ARC Prize's [open-source benchmarking repository](https://github.com/arcprize/arc-agi-3-benchmarking). What changes is the state carried between model calls. The Standard harness uses provider-neutral text history and asks the model to save useful discoveries in visible notes. The Provider Adapter uses OpenAI's native conversation handling, including retained opaque reasoning and compaction. The scores split widely at every published effort level. The cleanest comparison is high effort: 54.8% versus 99.9%. At maximum effort, the comparison is 62.7% versus 98.6%. Mixing those rows would imply that 62.7% and 99.9% came from identical settings. They did not, and the result is already strong enough without that shortcut. > **Info:** > > The Standard score asks how Astra performs through a common interface. The Provider Adapter score asks what the deployed OpenAI stack can produce when its native memory features are available. One is better for comparison; the other is closer to a product result. ## Remembering is part of the capability This is not the first ARC-AGI-3 result to move with context handling. In July 2026, [OpenAI reported](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/) that retained reasoning and compaction raised GPT-5.6 Sol from 13.3% to 38.3% on the public task set while using six times fewer output tokens. The settings changed together, so that experiment did not isolate how much each one contributed. It did show how badly an agent can perform when every action makes it reconstruct its own earlier thinking. Astra's replays make the benefit easier to picture. ARC Prize observed the model compressing game state into short symbolic notes, recording objects, coordinates, rules, and unfinished plans. In the Provider Adapter runs across Public and Semi-Private tasks, ARC Prize says the system was about 3.66 times faster by aggregate recorded elapsed time and used 49% fewer tokens across 167 game and reasoning-effort pairs that both harnesses solved. I would carry that result into a product review. Persistent state helped Astra repeat less work in these runs. ARC Prize also reports that Astra at maximum effort used fewer actions than the median tested human on 96.0% of levels and averaged 51.7% fewer actions per level under the Provider Adapter. Action efficiency has a boundary. ARC Prize's [scoring methodology](https://docs.arcprize.org/methodology) counts interactions that affect the environment rather than internal reasoning steps, tool calls, or retries. The test tells me Astra learned and acted with little external trial and error. Energy use, required supervision, and recovery from a wrong production assumption need their own measurements. ## A fair benchmark can still be an unfair product test There is an obvious objection to the Standard result: why test a model with memory behavior unlike the environment it was trained and deployed to use? OpenAI made that case after the earlier GPT-5.6 result. I find it persuasive when a buyer wants to know, "What can I get from OpenAI's supported stack?" The 99.9% result answers that question. It is less persuasive when the question is, "Which model travels best across providers and runtimes?" A native adapter can reveal available capability, but it ties the result to provider-specific state handling that another system may not reproduce or inspect. The opacity matters too. If a preserved false hypothesis survives several actions, a developer may see the consequences without being able to examine the state that kept it alive. Astra is not an isolated example. [TechCrunch reported](https://techcrunch.com/2026/08/21/nvidia-just-showed-that-the-harness-not-the-ai-model-is-now-the-real-hero/) that an Nvidia research harness took Claude Opus 5 from 30% to 100% on ARC-AGI-3 by adding memory support and a supervisor component. The setups are not directly comparable, but both make the same buying mistake easy to see: a model name is an incomplete agent specification. Public reaction jumped quickly from the score to whether ARC-AGI-3 is saturated or whether Astra counts as AGI. A short [Reddit discussion](https://www.reddit.com/r/OpenAI/comments/1w6g0is/gpt_astra_benchmarks/) captured both questions. ARC Prize is more careful. Its article says saturating this tightly bounded, deterministic benchmark is not proof of AGI, and that the real world remains more open-ended. ## I want two lanes on the scorecard I would keep both harnesses. A common lane reveals portability and makes cross-provider comparisons less slippery. A native lane reveals the best supported system a customer can actually deploy. Publishing only the common result can hide useful capability; publishing only the native result can hide how much the provider layer contributes. For a production decision, I would version the model together with its harness, state policy, compaction behavior, tools, and effort level. Then I would run the actual workflow with deliberate state changes. I would revise the goal halfway through, inject a bad observation, cross a compaction boundary, and resume after a long pause. The failure I would test is a system that keeps a bad premise alive with confidence. This extends the argument I made about the [Agentic Index](https://markhuang.ai/news/agentic-index-needs-your-failure-test): a leaderboard earns a model test time rather than production credentials. Astra's 99.9% earns serious attention, while the 54.8% run tells me what to inspect next. The memory layer belongs in the product specification and the evaluation record. Its failure modes belong in the risk review.