# Sidekick's Continuous Learning Loop: AI Model That Cuts Costs by 96% How converting production failures into model weights surpassed frontier model quality - **Source:** [Shopify Engineering](https://shopify.engineering) - **Published:** 2026-08-05 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/d5ef8ddb-36a7-44e4-a084-0259a92a1e6a ## The Economic Challenges of Frontier Models While frontier models enable rapid AI product launches, they become economically unfeasible at scale due to latency and cost issues. Sidekick identified that processing every request with frontier models is unsustainable. Each production failure represents valuable unlearned knowledge. Their solution: a continuous learning loop that captures this knowledge. ## The Learning Barrier of Frozen Models Deployed frontier models remain frozen - incapable of internalizing production lessons. Sidekick observed improvements accumulating in discrete patches like prompt edits and code changes, while model weights stayed unchanged. Their breakthrough: a flywheel mechanism that compresses production experience into the model's continuous weight space. ## Proven Success at Shopify Sidekick's learning loop was tested on Shopify's GraphQL agent with remarkable results: it delivered higher quality than frontier models while reducing latency and cutting costs by 96%. This demonstrated the practical viability of daily learning from production failures. ## The Critical Role of Quality Rubrics Quality definition is the most frequently overlooked step in continuous learning. Sidekick developed a scoring system called rubrics, comprising completion, execution, response quality, and safety criteria. Each has concrete scoring benchmarks. This rubric became the product team's official contract defining 'good' and 'bad' outputs. ## Inter-Annotator Agreement Testing To validate rubrics, two experts independently scored 25 random samples. Agreement measured by Cohen's kappa below ~0.2 indicates ambiguous rubrics. The goal isn't perfect agreement but a judge that matches human consensus levels. Some conversations remain inherently ambiguous. ## The Value of Detailed Explanations Sidekick mandates recording not just scores but the 'why' behind them. A score with one sentence provides insufficient learning context for calibration algorithms. The reasoning behind each score is treated as gold-standard data, providing crucial fine-tuning context. ## Calibration with DSPy and GEPA Sidekick uses the DSPy framework and reflection-based optimizers like GEPA for rubric calibration. GEPA evolves prompts by analyzing natural language failure traces, maintaining a Pareto frontier of candidates rather than selecting single winners. This enables multidimensional improvements. ## Testing Judge Metrics Judge metrics are rigorously tested against real traffic performance. They're backtested against historical A/B tests to verify they correctly predict known wins/losses. Controlled degradation tests deliberately worsen specific behaviors to confirm corresponding score drops, ensuring metric sensitivity. ## The Focused Judges Strategy Rather than cramming all product behaviors into one judge, Sidekick creates small, focused judges. This enhances interpretability and reliability. Additional judges can be added as needed, each measuring specific behaviors to contribute to holistic system improvement. ## The Auto-Research Process Sidekick treats baseline system improvement as an auto-research problem. An agent proposes changes to prompts, tool definitions, or harness code. The judge evaluates each change, keeping score-improving modifications. The entire process is configured in a single markdown file covering data sources, editable directories, metrics, and optimization settings. ## Sources covering this story - https://shopify.engineering/sidekicks-continual-learning-loop — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._