Analyzing Developer Adoption Patterns Through 40 Trillion AI Tokens
Brendan O'Leary reveals how trust and architectural feedback loops drive real-world AI agent adoption.
1Real-World Data vs Benchmarks
Brendan O'Leary challenges the industry's reliance on model benchmarks, arguing they fail to predict actual developer adoption. By analyzing 40 trillion tokens from Kilo Code, he highlights that trust, rather than raw capability, dictates success. Engineering leaders must look beyond leaderboard scores to understand how agents perform in complex, real-world production environments.
2The Four-Rung Adoption Ladder
AI adoption follows a distinct four-rung ladder, starting with simple autocomplete and progressing toward autonomous agent orchestration. Each level requires increasing levels of trust because the cost of failure rises significantly. If foundational rungs like autocomplete are shaky, developers lose confidence, preventing them from ever reaching the higher, more complex rungs of agentic workflows.
3The Role of Trust
Trust in AI tools is not merely a subjective feeling but a measurable metric that evolves as developers climb the adoption ladder. At lower levels, trust is established through synchronous, low-stakes interactions. As agents gain autonomy, the feedback loops become more delayed, requiring sophisticated instrumentation to maintain developer confidence and ensure system reliability.
4Technical Failure Points
Adoption predictably breaks at three critical technical junctures: context construction, model routing, and feedback loops. O'Leary emphasizes that each of these areas requires concrete architectural fixes to scale agentic systems. Addressing these specific points allows engineering teams to move beyond experimental phases and build reliable, long-term dependencies on AI-assisted coding tools.
5Kilo Code Infrastructure
Kilo Code provides an open-source agentic engineering platform with surfaces in the CLI, VS Code, and JetBrains. By offering a gateway to over 500 models, the platform captures diverse usage data. This infrastructure allows developers to bring their own inference, providing a unique, non-biased view into how engineers actually select and utilize models.
6Data Methodology and Scope
The dataset comprises 40 trillion tokens processed through Kilo Code’s infrastructure. To ensure accuracy, the analysis excludes anonymous traffic, internal usage, and automated bot activity. By focusing specifically on manual model selections made by developers within the interface, the data provides a clear picture of user preference versus automated backend routing decisions.
7Early Adopter Bias
The user base analyzed consists primarily of early adopters who actively seek out configurable, open-source tooling. While this cohort is not a perfectly random sample of the entire industry, it serves as a leading indicator. These teams are moving fast, using production environments to discover which AI workflows actually deliver tangible productivity gains.
8The Autocomplete Foundation
Autocomplete represents the first rung of the adoption ladder, characterized by synchronous, low-stakes interaction. Because developers can accept or reject suggestions with a single keystroke, the cost of failure is essentially zero. This immediate feedback loop builds the initial trust necessary for developers to eventually experiment with more complex, agent-driven coding tasks.
9Chat-Based Interaction
The second rung involves chat-based AI, where developers request code blocks or explanations. This stage mirrors the manual process of copying from Stack Overflow, requiring active review by the engineer. The feedback loop remains tight, measurable in seconds or minutes, allowing developers to maintain control while gradually increasing their reliance on AI-generated content.
10Agentic Workflow Revolution
The third rung introduces agents capable of taking autonomous actions, such as reading files, writing code, and executing commands. This represents a significant shift in the developer-AI relationship. Because the agent acts within the environment, the stakes are higher, making the reliability of the underlying model and the context construction process absolutely critical.
11Scaling Knowledge Work
While the data focuses on software engineering, the findings are applicable to other knowledge work domains. As teams look to leverage AI to increase productivity, they must prioritize the same architectural discipline. Understanding how to measure success and design feedback loops is essential for any organization deploying AI agents as active participants.
12Future Adoption Outlook
The rapid growth from 25 trillion to 40 trillion tokens reflects the ongoing asymptote of AI adoption in engineering. Over the next 12 to 18 months, the industry will likely see a shift toward more sophisticated agentic orchestration. Success will depend on building systems that prioritize trust, observability, and robust feedback mechanisms.
Original article: InfoQ — Analyzing Developer Adoption Patterns Through 40 Trillion AI Tokens