# OpenAI Adjusts Astra's Evaluation Metrics Post-Launch Changes in benchmarks highlight competitive AI industry dynamics. - **Source:** [Fortune | FORTUNE](https://brifing.si/source/769282ef-327a-4454-909a-3c7936216004) - **Published:** 2026-09-05 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/169bb9cd-be98-4732-b2c4-0212088aa8b6 ## Initial Launch and Delays OpenAI planned to release its GPT-6 Astra blog post at 2 p.m. ET on September 3. However, technical issues delayed its publication until 3:50 p.m. CEO Sam Altman acknowledged deployment problems, and many users encountered errors accessing the post. OpenAI later revealed the blog was briefly published and retracted due to undisclosed reasons unrelated to performance metrics. ## Metric Changes Favor Astra Upon republishing, OpenAI updated Astra's evaluation metrics, showing improved performance. For example, Astra's hallucination rate dropped from 4.2% to 2%. Meanwhile, metrics for GPT-5.6 Sol and Anthropic's Fable 5.1 worsened. These changes occurred amid intense competition in the AI industry, raising questions about benchmark reliability. ## Fluctuating Hallucination Rates Astra's hallucination rate initially stood at 4.2% in early snapshots. By 5:20 p.m., it halved to 2%, but later reverted to 4.2%. GPT-5.6 Sol's rate also fluctuated, dropping from 12.2% to 9.4% before returning to its original figure. OpenAI continues to investigate these changes, emphasizing the complexity of evaluation metrics. ## Mathematics Performance Astra excels in mathematics, maintaining a 97.6% score on the FrontierMath Tier 4 eval. However, OpenAI briefly altered scores for GPT-5.6 Sol and Anthropic's Fable 5.1, making Astra appear superior. Fable 5.1's score dropped from 87.8% to 78% before settling at 83%, while GPT-5.6 Sol's score fluctuated between 80.5% and 83%. ## Cybersecurity Evaluation Boost GPT-5.6 Sol saw a significant boost in its ExploitBench cybersecurity evaluation, jumping from 5.5% to 11.5%. OpenAI is investigating reverting this change, as the higher score reflects reasoning levels not commercially available. This highlights the challenges of accurately representing model capabilities in evaluations. ## ARC-AGI-3 Benchmark Astra's ARC-AGI-3 evaluation score increased from 98.6% in a pre-publication draft to 99.99% in the final blog. OpenAI noted that the Arc Prize Foundation independently assessed Astra at 99.9% with a powerful harness. With the standard harness, Astra scored 63%, still outperforming other publicly available AI models. ## Benchmarking Challenges OpenAI acknowledges that evaluation scores represent maximum performance under optimal conditions, which may differ from production models. The company includes disclaimers and caveats for each metric, emphasizing the influence of factors like harness and reasoning level on results. This transparency aims to provide meaningful comparisons for users. ## Industry Implications The frequent changes in OpenAI's metrics underscore the competitive nature of the AI industry. Companies strive to outperform rivals by optimizing benchmark results, raising concerns about manipulation. As AI models evolve, standardized evaluation methods will be crucial to ensure transparency and trust in performance claims. ## Sources covering this story - https://fortune.com/2026/09/04/openai-quietly-boosts-some-of-astras-evaluation-metrics-amid-rare-delay-in-publication-of-the-modeblog-post-announcement/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._