Benchmarking AI Decision-Making Capabilities via Texas Hold'em Poker
Analyzing how frontier models, decision-specific AIs, and scripts perform under uncertainty.
1Benchmark Design and Methodology
The benchmark evaluates AI decision-making by simulating 7,700 hands of heads-up no-limit Texas hold'em. By resetting stacks after every hand and using 50/100 blinds, the study isolates decision quality from variance. It compares specialized decision models, chat-based LLMs, frontier models, and simple scripts to measure performance under uncertainty.
2Decision Models vs. LLMs
Decision models like Jev and Solar provide probability distributions for actions rather than free text. While these endpoints are fast and cheap, costing 5 to 10 cents per thousand decisions, they exhibit erratic behavior. For instance, Solar folded premium hands like aces in 30% of observed spots, underperforming against basic scripts.
3The Frontier Model Advantage
Frontier models like Claude Opus and GPT-6.1 Sol demonstrate superior calibration, with win-probability estimates correlating 0.87 to 0.88 with actual equity. Despite their high cost, they never produced invalid answers and maintained consistency. They outperformed specialized decision models, though they only drew against cheaper reasoning models in head-to-head matches.
4Calibration and Self-Assessment
Most models struggle to accurately estimate their own win probability. Specialized decision models like Jev and Solar underestimate their equity by 15 to 17 points. In contrast, chat LLMs show smaller deviations of 4 to 9 points, while simple scripts remain the most accurately calibrated, showing errors of less than 1 point.
5Consistency and Determinism
Decision models are fully deterministic, providing identical outputs for repeated inputs. Conversely, chat LLMs show significant variance; GLM repeats actions only 55% of the time. Frontier models prove the most stable, with Opus maintaining consistent responses even when suit names are swapped, suggesting a higher level of logical robustness in reasoning.
6The Cost of Reasoning
Latency is a critical factor, with DeepSeek requiring up to five minutes per decision, significantly slowing down the league. While models like Luna and GLM offer high value at 60 to 70 cents per thousand hands, frontier models can cost thirty times more for similar results, highlighting a clear trade-off between efficiency and performance.
7Local Model Performance
Bielik 11B, a Polish-focused model, struggled significantly when run locally on an RTX 3090 Ti. It finished last, performing worse than a random-action bot. It failed to fold, put money into 96% of pots, and overestimated its chances by 20 points, proving that instruction tuning alone does not guarantee logical poker play.
8Convergence of Playstyles
Analysis of 40 frozen spots reveals that strong language models converge toward similar strategies. Despite originating from different labs, models like DeepSeek and Opus choose the same actions in 65% to 80% of cases. This suggests that high-level reasoning models are developing a shared, textbook approach to handling complex, uncertain scenarios.
9Limitations of the Study
The benchmark uses a narrow slice of poker with fixed betting sizes, which limits the complexity of the game. Results are based on specific prompts, and different instructions could alter the rankings. Furthermore, these models would likely fail against specialized solvers, as the focus remains on comparative AI performance rather than superhuman play.
10Practical Implications for Developers
For developers needing to automate decisions under uncertainty, the choice of model depends on the required precision and budget. While frontier models offer the most reliable reasoning, simple scripts remain surprisingly competitive for specific tasks. The open-source dataset on Hugging Face provides a foundation for further research into model decision-making capabilities.