Shopify Optimizes AI Performance Using Task-Specific Open Models
Shopify leverages specialized open models and NVIDIA hardware to reduce AI costs and latency.
1The Sidekick Architecture Strategy
Shopify utilizes a frontier model to manage high-level merchant interactions and task planning. For specific, context-heavy operations like GraphQL queries, the system offloads work to smaller, post-trained task experts. This hybrid approach ensures that complex reasoning remains with the frontier model while repetitive, token-intensive tasks are handled by efficient, self-hosted open models.
2Building the GraphQL Task Expert
To optimize the GraphQL subagent, Shopify post-trained the NVIDIA Nemotron 3.5 Lightning model. This expert model reads API schemas and executes queries with higher accuracy than the original frontier model. By controlling the model weights, Shopify achieved a 7x faster response time and a 90% reduction in serving costs for this specific function.
3The Model Optimization Flywheel
Shopify employs a continual learning loop to refine task experts. Product experts define scoring rubrics, while an LLM judge, tuned with GEPA and ACE optimizers, evaluates candidate models against human labels. This judge acts as the gatekeeper, ensuring that every iteration of the agent provides higher quality outputs than the previous version.
4Rigorous Production Load Testing
Before deployment, models undergo testing with NVIDIA AIPerf to simulate production traffic. Shopify contributed live-response replay support to this tool, allowing multi-turn tests to hit server prefix caches accurately. This ensures that performance metrics, such as median response times and p99 latency, reflect real-world conditions rather than just synthetic, isolated benchmarks.
5Automated Self-Healing Mechanisms
Failed production conversations are repurposed into training data through an automated self-healing process. A frontier model acts as a critic, providing hints to reruns until the judge approves the output. Over 86 nightly runs, this system repaired 69% of failed GraphQL conversations, significantly expanding the ground truth dataset used for model calibration.
6Hardware-Accelerated Training Efficiency
Shopify uses NVIDIA Megatron Bridge to scale training across nodes on Nebius without rewriting code. Transitioning to Blackwell B300 GPUs provided substantial gains, with training jobs running 1.86x faster than on H200 hardware. This hardware efficiency reduces the cost of the training loop, allowing for more frequent updates to the task expert models.
7Performance Gains and Cost Impact
The post-trained Nemotron model improved its GraphQL judge score from 37.7 to 74.5 over six flywheel loops. In production traffic replays, the model returned median responses in 0.87 seconds, compared to 6.50 seconds for the frontier model API. Self-hosting these models consistently cuts serving costs by 90% to 98% across the platform.
8Scaling the AI Infrastructure
Shopify’s strategy demonstrates the value of combining frontier model reasoning with specialized, self-hosted open models. By integrating human corrections, production traffic data, and advanced NVIDIA hardware, the company creates a compounding improvement cycle. This architecture effectively balances high-level conversational capabilities with the cost-efficiency required for high-volume, repetitive commerce tasks.