Artificial Analysis Intelligence Index v4.2 Released
New update introduces advanced evaluations and private test sets.
1Introduction to Index v4.2
The Artificial Analysis Intelligence Index v4.2 introduces more complex and realistic tasks, along with private test sets to prevent gaming. This update aims to bring the Index closer to real-world use cases. Key additions include AA-Briefcase and GDP.pdf, which evaluate agentic knowledge work and long-context document reasoning, respectively. The Index has been stable through recent major model launches, but the frontier's rapid movement necessitated this interim update.
2AA-Briefcase Evaluation
AA-Briefcase is an in-house evaluation that tests models on realistic agentic knowledge work tasks. These tasks involve multi-week projects with thousands of input source files. Models are graded on verifiable task success, analytical quality, and presentation quality. This evaluation provides a holistic view of agentic capability in knowledge work. AA-Briefcase uses a private held-out test set to ensure robustness and prevent gaming.
3GDP.pdf Document Reasoning
GDP.pdf, created by Surge AI, evaluates single-turn professional document reasoning across 100 PDFs and ten domains. Models must synthesize evidence from 4,592 pages, including text, tables, charts, and footnotes. Responses are graded against 1,275 expert-authored atomic criteria. The All-pass Rate credits a task only when every criterion is satisfied. This evaluation tests models' ability to handle long-context documents.
4Weighting and Grading Improvements
Index v4.2 increases the weighting of private, held-out test sets to 40%, double the figure from v4.1. This reduces the ability for labs to game evaluations. Grading infrastructure has also been upgraded, with improvements to scoring accuracy and robustness. For example, AA-LCR v1.1 includes a grading system prompt and corrected errors in answer keys. These changes make the Index more reliable and robust.
5Leaderboard Highlights
Anthropic’s Claude Fable 5.1 leads the Index, followed by OpenAI’s GPT-6 Astra, which shows a 4-point gain over GPT-5.6 Sol. Meta ranks third, followed by SpaceXAI, Moonshot/Kimi, Z.AI, and Google. GPT-6 Astra is more token-efficient than most models near the intelligence frontier. Claude Fable 5.1 and Opus 5 lead AA-Briefcase, while GPT-6 Astra leads GDP.pdf.
6Cost per Task Frontier
The Cost per Task Pareto frontier is shared by four labs: Anthropic, OpenAI, Meta, and Z.AI. GPT-6 Astra dominates the output token frontier, using fewer tokens than GPT-5.6 Sol for similar performance. However, higher prices offset its efficiency gains. These metrics help labs optimize their models for cost-effectiveness while maintaining high performance.
7Future Plans for Index v5
The team is already working on Index v5, which will include further incremental releases. Elements of v5 have been in development for eight months since the launch of Index v4 in January. The focus remains on increasing the Index's relevance and usefulness to users. Future updates will continue to introduce more challenging and realistic tasks, along with further improvements to grading infrastructure.
8Significance of Index v4.2
Index v4.2 represents a significant step forward in AI benchmarking. By introducing more complex tasks and private test sets, it provides a more accurate measure of models' capabilities in real-world scenarios. The updates also address issues of gaming and robustness, ensuring the Index remains a reliable tool for evaluating AI progress. These improvements pave the way for the upcoming Index v5.