Sidekick's Continuous Learning Loop: AI Model That Cuts Costs by 96%
How converting production failures into model weights surpassed frontier model quality
1The Economic Challenges of Frontier Models
While frontier models enable rapid AI product launches, they become economically unfeasible at scale due to latency and cost issues. Sidekick identified that processing every request with frontier models is unsustainable. Each production failure represents valuable unlearned knowledge. Their solution: a continuous learning loop that captures this knowledge.
2The Learning Barrier of Frozen Models
Deployed frontier models remain frozen - incapable of internalizing production lessons. Sidekick observed improvements accumulating in discrete patches like prompt edits and code changes, while model weights stayed unchanged. Their breakthrough: a flywheel mechanism that compresses production experience into the model's continuous weight space.
3Proven Success at Shopify
Sidekick's learning loop was tested on Shopify's GraphQL agent with remarkable results: it delivered higher quality than frontier models while reducing latency and cutting costs by 96%. This demonstrated the practical viability of daily learning from production failures.
4The Critical Role of Quality Rubrics
Quality definition is the most frequently overlooked step in continuous learning. Sidekick developed a scoring system called rubrics, comprising completion, execution, response quality, and safety criteria. Each has concrete scoring benchmarks. This rubric became the product team's official contract defining 'good' and 'bad' outputs.
5Inter-Annotator Agreement Testing
To validate rubrics, two experts independently scored 25 random samples. Agreement measured by Cohen's kappa below ~0.2 indicates ambiguous rubrics. The goal isn't perfect agreement but a judge that matches human consensus levels. Some conversations remain inherently ambiguous.
6The Value of Detailed Explanations
Sidekick mandates recording not just scores but the 'why' behind them. A score with one sentence provides insufficient learning context for calibration algorithms. The reasoning behind each score is treated as gold-standard data, providing crucial fine-tuning context.
7Calibration with DSPy and GEPA
Sidekick uses the DSPy framework and reflection-based optimizers like GEPA for rubric calibration. GEPA evolves prompts by analyzing natural language failure traces, maintaining a Pareto frontier of candidates rather than selecting single winners. This enables multidimensional improvements.
8Testing Judge Metrics
Judge metrics are rigorously tested against real traffic performance. They're backtested against historical A/B tests to verify they correctly predict known wins/losses. Controlled degradation tests deliberately worsen specific behaviors to confirm corresponding score drops, ensuring metric sensitivity.
9The Focused Judges Strategy
Rather than cramming all product behaviors into one judge, Sidekick creates small, focused judges. This enhances interpretability and reliability. Additional judges can be added as needed, each measuring specific behaviors to contribute to holistic system improvement.
10The Auto-Research Process
Sidekick treats baseline system improvement as an auto-research problem. An agent proposes changes to prompts, tool definitions, or harness code. The judge evaluates each change, keeping score-improving modifications. The entire process is configured in a single markdown file covering data sources, editable directories, metrics, and optimization settings.