AI Shopping Agents Struggle With Real-World Purchase Accuracy
Visa study reveals AI agents fail nearly half of consumer shopping tasks.
1AI Shopping Performance Gap
A recent study by Visa tested 22 different AI agents on their ability to purchase everyday items like TVs and laptops. The results revealed a significant performance gap, with an overall success rate of only 57%. This indicates that current AI technology still struggles with the complexities of autonomous consumer shopping tasks.
2Top Performing AI Models
The study highlighted that cutting-edge models achieved the highest success rates. Google's Gemini Aragon 4 and Meta's Making Wonder both reached an 82% success rate. Meanwhile, an OpenAI agent ranked sixth with 64%, and a Claude agent secured third place with 76%, demonstrating that model sophistication directly correlates with shopping task completion.
3Cost Versus Success Correlation
Visa analyzed the relationship between the cost of running an inquiry and the agent's success. The five most expensive models achieved an average success rate of 67%, while the five cheapest models only managed 50%. This suggests that higher computational resources currently play a vital role in improving AI decision-making accuracy.
4The Challenge of Ambiguity
Agents often failed because they could not handle incomplete consumer information. In the experiment, agents had to identify products based on partial requirements, such as gaming console compatibility or specific display features. Success required the agents to actively elicit missing details from the shopper, a skill that many models currently lack.
5Future of Agentic Commerce
Rubail Birwadker, Visa’s Global Head of Growth, stated that while the technology is not yet ready for full trust, it is advancing rapidly. The industry is now focusing on product discovery as a foundational element. Companies are working to integrate these agents into mainstream commerce through improved payment and security infrastructures.
6Payment and Security Hurdles
Beyond product selection, companies like Visa and Mastercard are addressing the complexities of secure payments for AI agents. Facilitating so-called agentic commerce requires new tools that ensure transactions are safe and reliable. Retailers and technology firms are currently collaborating to define the standards necessary for this emerging digital shopping ecosystem.
7Unresolved Legal Liability Issues
The rise of autonomous shopping agents introduces significant legal questions regarding liability. It remains unclear who is responsible if an agent makes rogue purchases or commits fraud. Industry experts anticipate these issues will be resolved over time, similar to how early e-commerce regulations evolved to protect both merchants and consumers.
8Interactive Consumer Testing Platform
To increase transparency, Visa launched a new website allowing consumers to interact directly with the 22 models tested in the study. This platform provides insights into how these agents perform across 15 separate shopping expeditions. It serves as a practical tool for understanding the current limitations and potential of AI-driven shopping.
Original article: Fortune — AI Shopping Agents Struggle With Real-World Purchase Accuracy