AI Chatbots Deliver Inaccurate Financial Advice 57% of the Time
Major AI models risk consumer harm with faulty pension and tax guidance
1Widespread AI financial misinformation
Saturn's research tested 18 AI chatbots including ChatGPT, Claude, and Gemini on 121 financial questions. The models delivered incorrect advice 57% of the time, with error rates jumping to 88% on complex queries. Free versions performed worse, with 63% inaccuracy versus 49% for paid models. Claude Haiku 4.5 was the worst performer, providing wrong answers 82% of the time.
2Costly pension advice errors
One particularly damaging error involved pension tax rules, where Claude Haiku's incorrect advice would have cost a saver £17,500 in HMRC charges. The study found numerous types of errors including faulty calculations, ignoring tax changes, and inventing false financial rules. Google's Gemini 3.1 Pro ranked second-worst with a 73% failure rate in the tests.
3Regulatory protection gap
While human financial advisors must meet FCA standards, AI chatbots operate in an unregulated space. This creates a consumer protection gap - customers can complain about regulated financial services, but have no recourse for AI-generated bad advice. Saturn CEO Amal Jolly warns this situation risks widespread consumer harm as millions trust AI for financial guidance.
4Testing methodology
Saturn conducted 10,000 total tests across the AI models, focusing on practical financial questions consumers might ask. The research revealed systematic weaknesses in AI's ability to handle complex financial scenarios accurately. Performance degraded significantly on harder questions, suggesting current models lack the nuance required for reliable financial advice.
5Industry response
The FCA acknowledges AI's potential for financial education but emphasizes it's not regulated advice. A spokesperson cautioned consumers to verify AI-generated information and focus on long-term goals rather than quick gains. The regulator's stance highlights the tension between innovation and consumer protection in fintech.
6Commercial vs free model disparity
The study found a notable performance gap between free and paid AI models. Free versions delivered incorrect answers 63% of the time compared to 49% for paid versions. This suggests companies may be investing more in accuracy for commercial products while treating free versions as lower priorities for financial applications.
7Consumer behavior risks
With millions seeking financial advice from AI, the prevalence of incorrect guidance poses systemic risks. Users may make irreversible financial decisions based on faulty information, especially regarding complex topics like pension rules where errors can have severe long-term consequences.
8Call for industry standards
The findings highlight the urgent need for standards in AI financial advice. Potential solutions could include certification programs, accuracy benchmarks, or regulatory oversight. Until then, consumers should treat AI financial guidance with extreme caution and always consult qualified professionals for important decisions.