Cutting LLM Costs by 90% with Adaptive Model Routing
Dynamic model assignment optimizes multi-agent AI systems
1The Problem with Static LLM Assignment
Traditional multi-agent systems often use the same powerful LLM for all tasks, whether simple web searches or complex contract analysis. This one-size-fits-all approach leads to unnecessary costs, as simpler tasks don't require expensive models. For example, a system might use GPT-5-Pro for all agents including basic data collection, wasting resources on tasks that could be handled by lighter models.
2When Static Assignment Fails
Static model assignment works only when query types are predictable. In versatile assistants serving different departments, the required data sources, tools, and context vary dramatically at runtime. A finance query might need dense contract analysis while HR requests simple policy lookups. Pre-assigning models can't adapt to these dynamic needs, forcing over-provisioning of expensive models.
3Core Insight: Distributed Planning
The breakthrough idea is that comprehensive upfront planning isn't necessary. Instead, each agent dynamically invokes planning when needed. When the Researcher agent activates, it calls a Planner Agent specifically for data collection tasks. This narrow-scope planning requires minimal intelligence, allowing use of cheaper models while maintaining accuracy.
4Classification Simpler Than Execution
Determining a task's complexity requires far less intelligence than performing it. The system uses lightweight models like Gemini-flash-lite for routing decisions. These fast, cheap classifiers evaluate tasks across three dimensions: complexity, reasoning needs, and context size - enabling precise model assignment without performance overhead.
5The Routing Score System
Tasks receive scores (0-6) across three metrics: complexity (0-2), reasoning (0-2), and context size (0-2). These sum to determine model tier: 0-2 uses fast models (gpt-5-mini), 3-4 balanced (gpt-5), and 5-6 powerful (gpt-5-pro). This granular scoring prevents overuse of expensive models while maintaining quality where needed.
6Context Size Matters
In multi-agent pipelines, context accumulates significantly. While a Researcher might start with 500 tokens, the Analyst inherits this output, the Critic adds more, and the Reporter might process 7,000-8,000 tokens. The router accounts for this growth, automatically routing later-stage tasks to more powerful models regardless of individual task simplicity.
7System Architecture
The solution comprises four components: 1) Sequential agents (Researcher, Analyst, Critic, Reporter), 2) Planner Agents invoked per-stage, 3) Adaptive Router for dynamic model assignment, and 4) Configurable model tiers. The backend uses FastAPI with SSE streaming, while the frontend is React/Vite for real-time cost visualization.
8Planner Agent Efficiency
Each agent's Planner handles only its specific domain (e.g., Data Collection for Researcher). This narrow focus keeps planning prompts simple, consistently routing them to fast, cheap models. The Planner generates concrete sub-tasks for its agent, eliminating the need for complex global planning that would require expensive models.
9Real-world Cost Savings
Early implementations show 70-90% cost reductions. A system previously using GPT-5-Pro for all tasks now routes 60% of tasks to fast models, 30% to balanced, and only 10% to powerful tiers. The classifier itself adds minimal overhead (1-2% of total cost) while enabling these substantial savings through precise routing.
10Implementation Considerations
The router is provider-agnostic, with model labels and pricing set via environment variables. Teams can mix models from different providers. The system maintains granular spend visibility, showing cost breakdowns per agent and task type. This enables continuous optimization as usage patterns evolve over time.
Original article: Towards Data Science — Cutting LLM Costs by 90% with Adaptive Model Routing