OpenAI Adjusts Astra's Evaluation Metrics Post-Launch
Changes in benchmarks highlight competitive AI industry dynamics.
1Initial Launch and Delays
OpenAI planned to release its GPT-6 Astra blog post at 2 p.m. ET on September 3. However, technical issues delayed its publication until 3:50 p.m. CEO Sam Altman acknowledged deployment problems, and many users encountered errors accessing the post. OpenAI later revealed the blog was briefly published and retracted due to undisclosed reasons unrelated to performance metrics.
2Metric Changes Favor Astra
Upon republishing, OpenAI updated Astra's evaluation metrics, showing improved performance. For example, Astra's hallucination rate dropped from 4.2% to 2%. Meanwhile, metrics for GPT-5.6 Sol and Anthropic's Fable 5.1 worsened. These changes occurred amid intense competition in the AI industry, raising questions about benchmark reliability.
3Fluctuating Hallucination Rates
Astra's hallucination rate initially stood at 4.2% in early snapshots. By 5:20 p.m., it halved to 2%, but later reverted to 4.2%. GPT-5.6 Sol's rate also fluctuated, dropping from 12.2% to 9.4% before returning to its original figure. OpenAI continues to investigate these changes, emphasizing the complexity of evaluation metrics.
4Mathematics Performance
Astra excels in mathematics, maintaining a 97.6% score on the FrontierMath Tier 4 eval. However, OpenAI briefly altered scores for GPT-5.6 Sol and Anthropic's Fable 5.1, making Astra appear superior. Fable 5.1's score dropped from 87.8% to 78% before settling at 83%, while GPT-5.6 Sol's score fluctuated between 80.5% and 83%.
5Cybersecurity Evaluation Boost
GPT-5.6 Sol saw a significant boost in its ExploitBench cybersecurity evaluation, jumping from 5.5% to 11.5%. OpenAI is investigating reverting this change, as the higher score reflects reasoning levels not commercially available. This highlights the challenges of accurately representing model capabilities in evaluations.
6ARC-AGI-3 Benchmark
Astra's ARC-AGI-3 evaluation score increased from 98.6% in a pre-publication draft to 99.99% in the final blog. OpenAI noted that the Arc Prize Foundation independently assessed Astra at 99.9% with a powerful harness. With the standard harness, Astra scored 63%, still outperforming other publicly available AI models.
7Benchmarking Challenges
OpenAI acknowledges that evaluation scores represent maximum performance under optimal conditions, which may differ from production models. The company includes disclaimers and caveats for each metric, emphasizing the influence of factors like harness and reasoning level on results. This transparency aims to provide meaningful comparisons for users.
8Industry Implications
The frequent changes in OpenAI's metrics underscore the competitive nature of the AI industry. Companies strive to outperform rivals by optimizing benchmark results, raising concerns about manipulation. As AI models evolve, standardized evaluation methods will be crucial to ensure transparency and trust in performance claims.
Original article: Fortune | FORTUNE — OpenAI Adjusts Astra's Evaluation Metrics Post-Launch