Amazon Cuts AI Costs by Replacing LLMs with Decision Models
Engineers reduced ticket labeling expenses by 98% using the Strands Decider model.
1Transitioning from Generative Models
Amazon engineers Carr and Mehul replaced the Claude Sonnet 4.6 model with Strands Decider for internal ticket labeling. While Sonnet 4.6 previously balanced intelligence and cost, the new decision model achieves identical performance at a 98% lower token cost. This strategic shift aims to save millions of dollars annually for the company.
2Decision Model Mechanism
Unlike generative LLMs that produce text, decision models return probabilities for predefined options alongside a confidence score. The team mapped their 20-category ticket catalog as fixed options. By utilizing the Strands Decider 2B architecture, specifically the E4B Gemma 4 build, they successfully automated complex classification tasks without needing full text generation.
3Infrastructure and Deployment
The system runs on SageMaker ml.g5.xlarge instances, each equipped with an A10G GPU and 24 GB of memory. The model utilizes approximately 19 GiB of space, processing requests in roughly 640 milliseconds. The stack leverages standard AWS services including SQS queues, Lambda functions for execution, and AppConfig for managing deployment stages.
4Optimizing Token Consumption
Efficiency gains stem from shorter prompts and outputting a single choice rather than JSON. Because production labels exceeded the 4,096-token window, the team condensed criteria into one- or two-sentence descriptions. For exceptionally long tickets, the system intelligently drops older comments to ensure the input fits within the model's specific token constraints.
5Performance and Cost Metrics
The switch resulted in a 74% reduction in average Lambda execution duration. At full utilization, the cost per million tickets is approximately $500, representing a 44-fold improvement over previous expenses. This calculation accounts for instance headroom and burst capacity, significantly lowering the operational overhead compared to the previous generative AI implementation.
6Technical Implementation Surprises
Engineers discovered that JSON key order significantly influenced output probabilities, with variations reaching up to 0.19. To mitigate this, the team now sorts keys before every request. This finding highlights the importance of input consistency when serving decision models behind APIs to ensure reliable and reproducible results across different production runs.
7Risk Mitigation Strategy
The team implemented a three-way switch via AWS AppConfig to manage the transition safely. By running both models on live traffic and comparing disagreements, they validated performance beyond offline evaluations. This configuration allowed for an immediate 45-second rollback, ensuring that the system remained stable throughout the entire migration process.
8Strategic Applicability Limits
Decision models are ideal for high-volume, closed-choice classification tasks where latency is critical. However, they are not suitable if labels require extensive explanation text, if the catalog changes weekly, or if context exceeds a few thousand tokens. For these specific use cases, traditional generative LLMs remain the necessary tool for developers.
Original article: Hacker News — Amazon Cuts AI Costs by Replacing LLMs with Decision Models