Why Adding More AI Agents Slowed Down Our System
Scalability challenges in LLM-based systems and optimization strategies.
1Everything Seemed Fine at First
When we first deployed LLM agents at Planck, everything appeared normal. Customers were satisfied, response times were acceptable. But as we added more agents, mysterious latencies emerged. Initially, we blamed LLM providers, but the real issue was much closer to home.
2Misdiagnosis: Blaming LLM Providers
As delays increased, our first suspicion fell on LLM providers. We assumed 'OpenAI slowed down' or 'Anthropic is overloaded.' However, logs revealed that while LLM calls completed quickly, our system delayed responses. The problem was in our architecture.
3Discovering the Architectural Flaw
We realized all agents ran through a single service. A single request triggered all agents simultaneously, each making dozens of sub-agent calls. Despite using async/await, the system slowed unexpectedly. We needed profiling to uncover the true bottleneck.
4Python's Hidden Costs
Profiler results were surprising: JSON serialization/deserialization consumed significant time. Also, aiohttp's default connection limits (100) constrained parallel I/O. Using optimized libraries like orjson, we improved serialization speed by 70% and adjusted connection limits.
5Event Loop Congestion
Debug mode revealed event loop lag warnings. LLMs responded, but Python's GIL prevented the event loop from processing these responses. Unexpected CPU-bound workloads nullified our async advantage. This was the root cause of our timeout errors.
6Setting Up a Minimal Test Environment
To isolate the issue, we created a minimal test setup: FastAPI server, artificial delays, and controlled load tests. With large payloads, we observed latency increasing linearly with agent count. Absolute values mattered less than the trend.
7Connection Pool Optimization
We optimized aiohttp's connection pool limits. Increasing the default 100 limit improved performance, but only to a point. Beyond a threshold, more connections hurt performance. Extensive testing helped find the sweet spot.
8The Real Bottleneck: CPU-Bound Tasks
Surprisingly, the real bottleneck wasn't I/O but CPU. JSON processing, logging, and other 'small' tasks blocked the event loop due to GIL. In our supposedly I/O-bound system, CPU-bound tasks degraded performance. This paradigm shift changed our solution approach.
9Solution Strategies
We implemented three key optimizations: 1) Accelerated serialization with orjson, 2) Created separate threads for CPU-intensive tasks, 3) Fine-tuned connection pool settings. These changes reduced latency from 500ms (5 agents) to 800ms (50 agents).
10Key Takeaways for Scalable AI Systems
This experience showed that scalability in LLM systems depends not just on model performance but also system architecture. Python's async/await isn't a magic wand. Especially in multi-agent systems, balancing CPU and I/O becomes the critical performance factor.
Original article: Towards Data Science — AI Ajanları Eklendikçe Sistem Neden Yavaşladı?