Amazon Cuts AI Costs by Replacing LLMs with Decision Models
Switching from Claude Sonnet to Strands Decider saves Amazon millions in annual token fees.
1Transitioning from Generative AI
Amazon engineers Carr and Mehul replaced Claude Sonnet 4.6 with the Strands Decider model for internal ticket labeling. While Sonnet previously balanced intelligence and cost, the new decision model achieves identical performance at a 98% lower price point. This strategic shift aims to save millions of dollars in annual token expenditures.
2Decision Model Mechanism
Unlike generative models that produce text, decision models output probabilities for predefined options. By mapping ticket data to a fixed catalog of 20 categories, the system eliminates unnecessary text generation. This approach fits perfectly for classification tasks where the output is a specific label rather than a conversational response.
3Technical Implementation Details
The team deployed the E4B Gemma 4 build on SageMaker ml.g5.xlarge instances. Each request processes in approximately 640 milliseconds, utilizing 19 GiB of memory. The infrastructure leverages AWS SQS for event queuing, Lambda for processing, and AppConfig for managing deployment stages, ensuring a scalable and isolated environment for the model.
4Optimizing Token Consumption
The cost reduction stems from shorter prompts and the elimination of JSON output requirements. Engineers condensed 20 label descriptions into concise criteria, fitting them within the model's 4,096-token window. By truncating older ticket comments, the system maintains efficiency without sacrificing the context necessary for accurate classification of customer escalations or security findings.
5Performance and Latency Gains
Internal data shows a 74% reduction in average Lambda execution duration following the switch. At full utilization, the new setup costs roughly $500 per million tickets, representing a 44-fold improvement over previous expenses. These savings exclude additional efficiency gains realized through reduced Lambda runtime, further lowering the total cost of ownership.
6Critical Lessons in Deployment
A surprising discovery during testing was that JSON key order significantly impacted output probabilities, shifting results by up to 0.19. To ensure consistency, the team now sorts keys before every request. This technical nuance highlights the importance of standardized input formatting when serving decision models behind an API for production.
7Risk Mitigation Strategy
The team utilized a three-way switch via AWS AppConfig to compare Sonnet and Strands Decider on live traffic. This allowed them to judge disagreements in real-time rather than relying solely on offline evaluations. This safety mechanism meant that reverting to the previous model would have required only a 45-second configuration change.
8Future of Classification Agents
This transition demonstrates that decision models are highly effective for closed-choice tasks with high volumes. However, they are not universal replacements; LLMs remain necessary if labels require extensive explanations or if the catalog changes weekly. For classification agents, decision models offer a compelling path toward more efficient, cost-effective AI operations.