First Proof Exam Highlights AI Math Performance Disparity
Proprietary frontier models dominate research-level mathematics while open-weight alternatives struggle to keep pace.
1The First Proof Benchmark
The First Proof project provides an independent evaluation of AI models in research-level mathematics. Launched in February 2026, the initiative aims to cut through the noise of imperfect benchmarks. By utilizing expert-selected problems and mock peer reviews, the team offers a rigorous assessment of how LLMs handle complex, high-stakes mathematical reasoning tasks.
2Frontier Versus Open Models
Results from the third batch of testing reveal a significant performance gap between proprietary frontier models and open-weight alternatives. While subscription-based models demonstrate exceptional capabilities in solving research problems, open-source models currently lag far behind. This disparity suggests that advanced mathematical reasoning is becoming a pay-to-play domain for top-tier research applications.
3Performance Metrics and Growth
The top-performing model, ProofCouncil, successfully solved nine out of ten research problems in the latest round. This represents a rapid ascent compared to February, where the top model solved only a few problems. By June, the winner managed six or seven, highlighting the dramatic acceleration of AI capabilities throughout the year 2026.
4The Open-Weight Struggle
Despite the progress seen in proprietary systems, open-weight models like Kimi K3 failed to solve any of the ten research problems in the latest evaluation. Mathematicians involved in the study noted that these models fell flat, emphasizing an urgent need for the community to invest more resources into improving open-source mathematical reasoning capabilities.
5Methodology and Peer Review
The First Proof team solicited ten problems from experts across various subdisciplines. Models were required to answer without human intervention, ensuring an objective test of their reasoning. Three graders evaluated each submission using a mock peer-review process, determining if the answers were acceptable for publication in a peer-reviewed academic journal.
6Privacy and Data Concerns
Many mathematicians are seeking alternatives to frontier models due to privacy concerns and potential data leakage. Researchers are increasingly paranoid about using closed systems for sensitive work, leading to requests for reliable open-source options. The First Proof team is actively testing these alternatives to help the scientific community find secure, transparent research tools.
7Transparency and Model Access
The First Proof rules require entrants to be openly available, which excluded OpenAI's latest blockbuster model from the evaluation. This lack of transparency prevents outside researchers from conducting direct performance comparisons. Without access to the internal workings of frontier models, the scientific community remains unable to fully verify the claims surrounding these breakthroughs.
8Future of Mathematical Research
AI models have become so proficient at proving well-defined statements that future testing of frontier models on this specific task may lose relevance. As LLMs approach the ability to supersede human mathematicians, the focus must shift toward ensuring these powerful tools remain accessible, transparent, and secure for the broader global research community.