AI Financial Advice Errors Reach 88% in Complex Queries
Major study reveals chatbots' alarming failure rates in tax and pension calculations.
1Widespread inaccuracies found
Saturn's comprehensive testing of 18 AI models including ChatGPT, Claude, and Gemini revealed an average 57% error rate across financial queries. The study involved over 10,000 requests with complex multi-calculation questions showing 88% inaccuracy. Free versions performed notably worse than paid models, with some reaching 99% error rates in complicated scenarios.
2Tax and pension vulnerabilities
Chatbots frequently provided incorrect information about tax regulations and pension rules, sometimes inventing non-existent policies. One glaring example involved Claude Haiku 4.5 miscalculating pension taxation that could lead to £17,500 in unexpected charges. The systems particularly struggled with anticipating future legislative changes, a critical requirement for sound financial planning.
3Best performer still flawed
Claude Opus 5 in reasoning mode achieved the best results but still delivered wrong answers 39% of the time. The research tested both free and paid versions across 100+ money-related questions, with each query repeated up to five times to ensure statistical significance. Newer model versions generally outperformed older ones, showing incremental improvements.
4Real-world financial risks
Saturn CEO Amal Jolly emphasized how millions trusting AI for financial decisions risk substantial losses. One fabricated response about student loans claimed payments could stop when moving abroad. The unregulated nature of AI financial advice leaves users without consumer protections available through human financial advisors, compounding the potential damage from errors.
5Industry response
AJ Bell's Sarah Coles acknowledged AI's utility for basic budgeting while stressing the need for professional advice on complex matters. The findings align with FCA research showing 20% of UK adults would let AI make financial decisions, particularly in debt management and investments. This creates tension between accessibility and reliability in financial services.
6Regulatory vacuum
Unlike human financial advisors, AI advice systems currently operate without oversight frameworks. Jolly highlighted this regulatory gap enables potentially harmful misinformation at scale. The timing is critical as OpenAI recently launched financial sector-specific ChatGPT services, expanding AI's role in sensitive monetary decisions.
7Testing methodology
Researchers designed the study to mirror real-world usage patterns, testing both simple and multi-layered financial queries. The 10,000+ requests spanned tax optimization, retirement planning, and investment scenarios. Repeated trials revealed inconsistency in responses, with some models providing different answers to identical questions across attempts.
8Broader implications
The study underscores fundamental limitations in current generative AI's ability to handle nuanced financial calculations. While useful for information retrieval, the technology's propensity for confident yet incorrect answers poses unique risks in financial contexts where errors directly translate to monetary losses, demanding careful consideration of deployment boundaries.
Original article: INCRYPTED — AI Financial Advice Errors Reach 88% in Complex Queries