The Agentic Index Ranks 24 Models. I Still Need a Failure Test.

Artificial Analysis now gives agentic performance its own leaderboard. When I inspected it, the page showed published Agentic Index results for 24 of 176 listed models. Two Claude Opus 5 configurations led with scores of 55, followed by GPT-5.6 Sol at max effort with 54.

I like that change. Tool use, planning, and multi-step work tell me more than another collection of short questions. Still, I would not let the winning score pick a production model for me. The index gives me an informed shortlist built from two useful tests. It cannot see my workflow, permissions, failure costs, or recovery path.

Quick answer

Question	My read
What does the index measure?	It combines agentic knowledge work and tool-using customer interaction into one score.
What is useful?	The dashboard puts capability beside cost per task, time per task, and output-token use.
What is the catch?	Two benchmark families cannot reproduce every agent harness, permission model, data source, or costly failure.
How would I use it?	I would build a shortlist, then run repeated tests on my own tasks and score the failures I cannot afford.

Two tests carry the whole score

Artificial Analysis documents the index as an equal-weighted average. Half comes from GDPval-AA v2, which evaluates economically valuable work across 44 occupations and nine major industries. The other half comes from τ³-Banking, a customer-support test that requires an agent to retrieve policy, talk with a user, and make the correct changes through tools.

Those are not toy multiple-choice exams. GDPval-AA v2 allows trajectories of up to 250 turns and uses human expert performance as the 1000-point Elo baseline. The underlying τ-Banking environment contains 698 documents across 21 product categories and roughly 195,000 tokens. Its tasks average 9.5 required tool calls. That is a meaningful attempt to test whether a model can keep working when the answer is scattered across documents and actions.

That coverage is broad, but it is also quite specific. A model that is excellent at preparing knowledge-work deliverables and resolving simulated banking requests may still be the wrong choice for code review, browser automation, procurement, or a workflow with strict human approvals. Equal weighting is one definition of agentic ability, not a law of nature.

The useful part sits beside the score

I paid more attention to the panels next to the ranking. Artificial Analysis also exposes cost per task, total index cost, weighted time per task, and output tokens per task. Those numbers make the page the beginning of a purchasing decision instead of a trophy cabinet.

A one-point score advantage can be irrelevant if it costs far more, runs too slowly for the queue, or produces so many tokens that review becomes the bottleneck. The opposite is also true. A cheaper model is not cheaper when its failures trigger refunds, manual cleanup, or a compliance incident. The dashboard gives me some of the inputs for that tradeoff, but I still have to price the consequences in my own system.

I also want to keep the model and the agent separate. MIT's 2025 AI Agent Index makes this distinction plainly: agent evaluations depend on downstream tools and autonomy levels, so model-level evaluation is insufficient. The same model can behave differently when the scaffold changes, a search tool disappears, permissions widen, or the retry loop rewards persistence over caution.

Benchmark maintenance is part of the result

Agent benchmarks age quickly because models improve and the tests themselves get audited. Sierra Research says it fixed more than 50 airline and retail tasks while preparing τ³-Bench, correcting ambiguous instructions, impossible constraints, and wrong expected actions. That work makes the benchmark better. It also means a score needs a benchmark version and date beside it.

Stanford's 2026 AI Index reports that difficult evaluations can saturate within months and cites invalid-question rates ranging from 2% to 42% in reviewed benchmarks. I do not read that as a reason to dismiss leaderboards. I read it as a reason to ask what changed before comparing an old score with a new one.

An earlier Artificial Analysis revision prompted a split Reddit discussion. Some commenters welcomed harder tests after older ones became saturated. Others worried that large score changes made long-term comparison less credible. Benchmarks need harder tasks as old ones saturate, but the version history has to remain legible.

I still need a failure test

My next step would be small and specific. I would choose a few candidates that fit the index, latency, and cost envelope, then give each one the same representative jobs inside the actual harness. Each job needs repeats, a count of human interventions, and a clear split between recoverable mistakes and failures that create an external commitment.

That last distinction matters more to me than the podium. A malformed draft can be regenerated. An unauthorized payment, a policy-violating account change, or a confident message sent to a customer is a different class of failure. Average task success hides that difference unless the evaluator names it.

The Agentic Index is useful because it asks models to do work instead of merely answering questions. Its two components are demanding, documented, and relevant to many buyers. It gives me a smarter place to spend my evaluation time. The final test still belongs inside the workflow where the model will run.