Why GPT-6 Astra’s Benchmark Rankings Don’t Agree

Benchmark headlines can make GPT-6 Astra look simultaneously dominant, tied, and behind. The apparent contradiction is mostly methodological: a composite leaderboard, an interactive-agent test, a formal-math benchmark, and API pricing each evaluate a different system under different constraints.

The practical conclusion is not that one result must be wrong. It is that there is no universal model ranking without specifying the workload and evaluation setup.

A leaderboard score is a measurement design, not a single intelligence meter

Composite indices combine multiple tasks into one number. Their outcome depends on which tasks are included, how each is weighted, which model configuration is tested, and how reasoning effort, tool use, and cost are handled. A model that is strong on broad capability coverage can lead one aggregate while trailing in a coding-heavy or agent-focused index.

That also makes point scores time- and configuration-sensitive. The supplied Artificial Analysis comparison, for example, lists GPT-6 Astra (max) at 55 and Claude Fable 5.1 at 57—not the 61 and 66 figures reported in some earlier comparison articles. 3 The discrepancy is a reason to record the exact evaluation configuration and date, rather than treating any one snapshot as a permanent ranking.

The claimed Epoch AI 169-point Capabilities Index position and its underlying methodology are not substantiated by the available Epoch primary-source material. It should not be used as a decisive comparison without the underlying leaderboard, benchmark weights, model settings, and uncertainty information.

ARC-AGI-3: the harness changed the result

ARC-AGI-3 is an interactive benchmark rather than a static question set. Agents must explore unfamiliar environments, acquire goals, build adaptable world models, and adjust their strategy through experience. 50

ARC Prize reported two standout GPT-6 Astra results on its Semi-Private evaluation:

Evaluation condition	Best reported score	Reported cost per run
Standard harness, max reasoning	62.7%	$26,098
OpenAI Provider Adapter, high reasoning	99.9%	$18,817 (reported as about $19K)

5152

The gap is not a minor technical footnote. Under the Standard harness, an agent uses a minimal, provider-neutral interface and can carry forward notes it explicitly chooses to retain. The Provider Adapter can preserve opaque provider reasoning state between requests and use context compaction for longer conversations. 5153

The Provider Adapter score therefore measures the performance of Astra together with OpenAI’s native context-management integration. That is a meaningful product capability, but it is not automatically apples-to-apples with models evaluated only through the neutral interface. ARC Prize explicitly treats the harnesses as different evaluation conditions that answer different questions. 53

ARC Prize also reported that Astra used fewer actions than the median tested human on 96% of levels. 52 That is an action-efficiency result, not a claim that the model is broadly better than people at all work, nor a substitute for comparing task completion, reliability, and total cost in a real workflow.

What FrontierMath Erdős can—and cannot—establish

FrontierMath Erdős contains 68 difficult Erdős problems that were open as of August 2026. To solve a task, a system must prove or disprove the conjecture in Lean, so submitted proofs can be mechanically checked. 3337

This design makes the benchmark unusually rigorous for formal mathematical correctness. But it does not turn a math result into an overall coding, agent, or general-reasoning ranking. It measures success on a narrow and demanding formal-proof task under a stated budget.

The provided primary-source material does not substantiate the requested total number of Astra solutions, standardized versus non-standard run budgets, or which results were excluded from the FrontierMath Erdős evaluation. Those details should be reported only from Epoch’s specific results table or evaluation report, with the budget and eligibility criteria attached.

A separate Epoch FrontierMath: Open Problems page does credit a pre-release GPT-6 Astra evaluation with finding a genus-2 curve with 648 rational points. That is a distinct benchmark result and should not be conflated with a score on FrontierMath Erdős. 43

Token pricing is not task pricing

OpenAI lists GPT-6 Astra at $10 per million input tokens and $50 per million output tokens. 19 Comparison reporting in the supplied sources says Claude Fable 5.1 used the same headline input and output rates, while charging $0.25 per million cached-input tokens versus Astra’s $1.00 per million. 4

That leads to two different cost questions:

• Long-running coding agents with repeated repository context: lower cache-read pricing can materially reduce cost, provided the workload benefits from caching.
• Completed-task cost: a model can still be cheaper if it reaches a correct result with fewer output tokens, tool calls, retries, actions, or human interventions.

OpenAI says Astra achieved lower estimated API cost per task in several of its evaluations by using substantially fewer output tokens. 18 This is vendor-reported evidence, not a general guarantee. A benchmark score and a token price cannot, by themselves, prove which model is cheaper for a particular codebase or agent workflow.

How to compare models without being misled

Before choosing between Astra, Claude Fable 5.1, or GPT-5.6 Sol, define the task and normalize the conditions:

1. Match the workload. Evaluate repository maintenance, browser automation, research, formal proofs, or document work separately.
2. Fix the inference policy. Set comparable reasoning effort, token limits, tool permissions, and retry rules.
3. Separate model quality from integration quality. Report neutral-harness results alongside provider-native results rather than collapsing them into one ranking.
4. Measure success per dollar. Track correct completions, latency, retries, cache use, and required human review—not only input and output token rates.
5. Preserve uncertainty. Small index gaps and changing configurations deserve cautious interpretation, especially when the public evidence is incomplete.

Astra’s ARC-AGI-3 Provider Adapter result is striking, and its standard-harness result remains strong. But neither result invalidates an independent index that favors another model under a different task mix and configuration. The useful question is not “Which model won?” It is “Which evaluated setup most closely matches the work this system must actually do?”