AI Scorecards Get Harder To Fake: Inside Artificial Analysis Intelligence Index v5, With 2 New Tests, 3 Upgrades and 60% Private Data

By Sarah Chen · MyTechDailyNews · Tech Daily

Leaderboards for AI models rarely make sense to anyone outside the labs that build them, and Artificial Analysis just reset its most-quoted one. Today it published the Artificial Analysis Intelligence Index v5, described as the biggest overhaul of the Artificial Analysis Intelligence Index since v4 first appeared in January. The mechanics of the refresh are easy to summarise: 2 new benchmarks joined the suite, 3 existing evaluations were upgraded, and 60% of the composite now rests on private test sets or held-out answers — a deliberate step away from questions models may already have met in training.

A second release travels with it. The Artificial Analysis Progress Index v1 plots model capability over time on one continuous scale, instead of only ranking whoever happens to be shipping right now.

What actually moved on the test list

FrontierCode 2.0 and Terminal-Bench-Science are the two newcomers. Cognition built FrontierCode 2.0 as a private benchmark of real-world coding work, and its grading question is an ordinary engineering one: would the maintainer of a repository merge the change the agent produced? Terminal-Bench-Science, meanwhile, puts agents inside a terminal environment and hands them 70 agentic scientific research tasks.

On the upgrade side, HLE-Diamond takes the slot formerly held by Humanity's Last Exam. It is the refined 1,000-question edition assembled by the Center for AI Safety (CAIS) and Scale AI, split evenly between reasoning and knowledge items. AA-Omniscience v1.1 is a tidy-up of the knowledge-and-hallucination measure: saturated and public set questions were stripped out, fresh questions were added, and ambiguous or erroneous ones were corrected. CritPt v2 now runs on a private hard subset of 40 questions, revised by the CritPt authors and executed with a code-execution tool.

Not everything shifted. Here is the in-and-out ledger:

• Still counted: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
• Now gone: GDPval-AA v2.1, SciCode and AA-LCR v1.1

How the number is built: the v5 weighting

Four blocks carry equal shares of the Index, which is worth remembering before treating one headline score as a verdict on every job.

Category	Share of score	Evaluations inside it
Knowledge Work	25%	AA-Briefcase (15%), AutomationBench-AA (5%), GDP.pdf (5%)
Coding	25%	FrontierCode 2.0 (15%), Terminal-Bench 4.0 (10%)
Science	25%	Terminal-Bench-Science (15%), CritPt v2 hard subset (10%)
General	25%	AA-Omniscience v1.1 15% (10% accuracy, 5% non-hallucination), HLE-Diamond (10%)

Where the leaderboard landed

Claude Opus 5.5 (max) sits first at 58, with GPT-6 Astra (max) next at 57 and GPT-6.1 Sol (max) third at 55. Anthropic and OpenAI occupy the entire top five, while Gemini 4 Argon (high) trails the group at 50.

The distance between the frontier and the open weights camp grew from 3 points to 12. Kimi K3.1 Pro heads that table at 46, although its weights are not yet released. Among models whose weights are actually available, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) peak at 32 — 26 points behind the leader, and still below Claude Haiku 5.5 (max), which reaches 36.

Price behaves on its own axis. Among the leading models, cost per task varies by more than 13x: of the 10 highest-scoring entries shown, GPT-6.1 Terra (max) is cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) is most expensive at $13.04.

FrontierCode 2.0: coding judged like a real pull request

Cognition refreshed this private coding benchmark earlier today. It comprises 132 tasks in real open-source repositories, scored on whether a maintainer would merge the agent's change. Fail any blocking check and the attempt scores 0; the tasks stay unpublished to avoid contamination, and each one is run three times in Stirrup, Artificial Analysis' agentic harness.

Claude Opus 5.5 (max) leads at 46%. GPT-6 Astra (max), GPT-6.1 Terra (max) and Claude Sonnet 5.5 (max) follow at 42%. DeepSeek V4.1 Flash 1017 (max) is the strongest open weights entry here at 28%, 18 points off the lead.

Terminal-Bench-Science: research chores in a terminal

The test asks whether agents can carry out scientific research workflows from a command line. Experts curate and review every task, and a task passes only if all of its tests pass.

GPT-6 Astra (max) leads at 63%, Claude Opus 5.5 (max) reaches 59% and GPT-6.1 Sol (max) 58%. Below that cluster, scores fall sharply: Kimi K3.1 Pro (weights not yet released) records 34%, and Claude Haiku 5.5 (max) manages 20%. DeepSeek V4.1 Flash 1017 (max), the best open weights model with released weights, scores 13%, while most open weights models benchmarked land under 10%.

HLE-Diamond: thinking versus knowing

HLE-Diamond replaces Humanity's Last Exam in the Index — the refined 1,000-question version from the Center for AI Safety (CAIS) and Scale AI, half reasoning and half knowledge. Claude Opus 5.5 (max) leads at 68%, GPT-6 Astra (max) posts 64%, and Claude Fable 5.1 (max) 60%.

At the top of the table, knowing facts is proving harder than working through them: Claude Opus 5.5 (max) and GPT-6 Astra (max) both score 78% on reasoning, compared with 58% and 50% on knowledge.

AA-Omniscience v1.1: a cleaned-up honesty check

AA-Omniscience v1.1 is the scrubbed version of the knowledge and hallucination benchmark, with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised. Results stay broadly aligned with the earlier series, although models average 2 points lower on the AA-Omniscience v1.1 Index (median 0.6). A public slice, AA-Omniscience-Public-v1.1, also landed on Hugging Face, recalibrated so that accuracy, hallucination rate and overall performance line up between the public and private sets.

CritPt v2: physics problems in an offline sandbox

For CritPt v2 the Index uses a private 40-question hard subset: of the 70 research-level physics questions, the 30 that seven frontier models solved in all five attempts at launch were removed. Models work in an offline Python sandbox in Stirrup, Artificial Analysis' agentic harness.

GPT-6 Astra leads at 72% on high effort, GPT-6.1 Sol follows at 69% on max effort, and Claude Opus 5.5 (max), the top model in the Index overall, scores 61%. Kimi K3.1 Pro (weights not yet released) is the highest-scoring open weights model at 45%.

No single model sweeps the board

Across all nine evaluations, v5 still leaves significant room for progress, and no one model leads them all. Claude Opus 5.5 leads three; five other models share the remaining six:

• Claude Opus 5.5: FrontierCode 2.0 (46%), HLE-Diamond (68%) and AA-Omniscience v1.1 (48)
• GPT-6 Astra: Terminal-Bench-Science (63%) and CritPt v2 hard subset (72%)
• Claude Sonnet 5.5: AA-Briefcase (1812 Elo)
• Kimi K3.1 Pro: Terminal-Bench 4.0 (67%)
• Gemini 4 Argon: AutomationBench-AA (78%)
• Muse Wonder Full: GDP.pdf (33% all-pass)

The open weights gap stretched

Open weights models fall further behind in v5. The best of them, Kimi K3.1 Pro (weights not yet released), scores 46 — 12 points back from the leader, against a 3-point gap in v4.3. Of models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them.

The divide is widest on HLE-Diamond and Terminal-Bench-Science, where Kimi K3.1 Pro scores 37% and 34%, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max).

Price, effort and token burn

Claude Opus 5.5 (max) costs $10.04 per Intelligence Index task, GPT-6 Astra (max) $6.24, and GPT-6.1 Sol (max) $1.27. That last figure puts GPT-6.1 Sol (max) within 4 points of Claude Opus 5.5 (max) at about an eighth of the cost per task.

Effort settings matter more in v5 than in v4.3. Claude Opus 5.5 scores 35 at low effort and 58 at max, at 10x the cost per Intelligence Index task ($0.98 vs $10.04). Claude Sonnet 5.5 spans 29 points from low to max, up from 20 in v4.3, and GPT-6.1 Sol's spread doubles from 10 to 20 points.

The tasks also demand substantially more work. Every measured model generates more output tokens per Intelligence Index task than it did in v4.3, with the median rising by two thirds, from 35k to 59k.

• Agentic evaluations drive the bill: Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase and FrontierCode 2.0 make up 55% of the Index weight but a median ~90% of output tokens
• Terminal-Bench-Science is the most token-intensive on its own, accounting for about a third of output Tokens per Task

Progress Index v1, for reading the long curve

The Artificial Analysis Progress Index v1 arrives alongside v5. It places language models released since November 2022 on one capability scale across a wide span of benchmarks, so it answers a different question from the leaderboard: how far underlying capability has moved over time, including between models released years apart. For choosing a model for current use cases, or understanding the present frontier, the Intelligence Index remains the primary measure of current model intelligence. Artificial Analysis built on learnings from the Epoch Capabilities Index in designing it.

Three ways readers misread these numbers

• Top of the Index means best at everything. Not in v5: across all nine evaluations no single model leads them all, and Claude Opus 5.5 (max), the overall leader, trails GPT-6 Astra and GPT-6.1 Sol on CritPt v2.
• A higher score guarantees a higher bill. GPT-6.1 Sol (max) lands within 4 points of Claude Opus 5.5 (max) at about an eighth of the cost per task, and the 10 highest-scoring models already span more than 13x in price, from $0.96 to $13.04.
• Open weights now sit just behind the frontier. The gap in v5 is 12 points rather than the 3 seen in v4.3, and among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score 32 — below Claude Haiku 5.5 (max) at 36.

FAQ: the v5 update, answered

What is the Intelligence Index in plain terms?

It is a composite measure of model capabilities across tasks that reflect real-world usage today, assembled from four blocks that each carry 25% of the weight: Knowledge Work, Coding, Science and General.

Why does it matter that most of it is private?

Public questions can be memorised, so scores drift toward contamination rather than capability. With v5, 60% of the Index is based on evaluations with private test sets or held-out answers, and benchmarks such as FrontierCode 2.0 keep their tasks private for exactly that reason.

Which model is first right now?

Claude Opus 5.5 (max) at 58, ahead of GPT-6 Astra (max) at 57 and GPT-6.1 Sol (max) at 55, with Anthropic and OpenAI holding the top five and Gemini 4 Argon (high) next at 50.

Should readers use the Progress Index instead?

Use the Progress Index to compare how underlying capability has changed over time, including between models released years apart. Use the Intelligence Index for today's choices, since it remains the primary measure of current model intelligence.

Do effort settings really change results?

Sharply. Claude Opus 5.5 moves from 35 at low effort to 58 at max at 10x the cost per task ($0.98 vs $10.04), Claude Sonnet 5.5 spans 29 points, and GPT-6.1 Sol's spread doubled from 10 to 20 points versus v4.3.

What to do with this as a tech reader

• Compare models inside the block you actually use — Coding, Science, Knowledge Work and General each count 25%, and different vendors win each one.
• Check the effort setting quoted on any screenshot before repeating a claim; 29 points of movement on one model is possible between low and max.
• Watch cost per Intelligence Index task rather than score alone if you scale usage: the top 10 scorers differ by more than 13x.
• Expect agentic workloads to dominate token spend — a median ~90% of output tokens sits in evaluations worth 55% of the Index weight, with Terminal-Bench-Science alone about a third.
• For open weights requirements, keep the weights-not-released distinction in view: Kimi K3.1 Pro tops that group at 46, while released-weight leaders DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) sit at 32.

Who is behind the update

Artificial Analysis credited Cognition, Steven Dillmann and the Terminal-Bench team, the Center for AI Safety, Scale AI, and Minhui Zhu, Minyang Tian, Hao Peng and the CritPt team for the benchmarks behind this update, and thanked Epoch AI and Anthropic in connection with the Progress Index.

For everyday readers, the story in v5 is less about crowning a winner than about tightening the referee: tougher, unpublished tasks, a wider spread between what a model can do and what that costs, and a reminder that a scorecard from the last cycle — the one that put the open weights field 3 points back in v4.3 — no longer describes the market.