The AI Cost of JavaScript Frameworks
Evaluating framework performance and development costs in the era of AI-assisted coding.
1The Vibe-Based Development Trap
Web development decisions are frequently driven by subjective feelings rather than empirical evidence. Frameworks are often marketed as blazing fast or developer-friendly without supporting benchmarks. This trend persists even as developers transition to agentic workflows, where AI tools now generate code, yet the reliance on unsubstantiated claims remains a significant industry issue.
2Designing the Benchmark Experiment
The experiment involved building a Pokémon-themed application across various frameworks using Opus 5.5. The goal was to compare development costs, code quality, and maintainability. To ensure consistency, agents operated in sandboxed environments without specific architectural mandates, allowing for a fair comparison between different meta-frameworks and standard JavaScript approaches using Vite.
3Stochastic Nature of AI Agents
Measuring AI-generated code is inherently stochastic, as LLMs function as random-word generators. Variations in token usage and agent tangents can impact results regardless of the framework's quality. To mitigate this, the study utilized multiple iterations for each framework, acknowledging that Lighthouse metrics and agent performance can fluctuate significantly across different runs.
4Financial Costs of Frameworks
Building apps with AI agents incurs costs based on token usage and API calls. SvelteKit, Gracile, and Angular proved more expensive to build than alternatives. Higher costs for some frameworks were attributed to agents needing to read migration guides or documentation not present in their training data, highlighting the importance of framework maturity.
5The Role of MCP Skills
The experiment revealed inconsistent usage of Model Context Protocol (MCP) skills by AI agents. Some frameworks, like TanStack Start, only utilized their specific skill files in a fraction of the test runs. This inconsistency suggests that current AI tooling integration is not yet reliable enough to guarantee optimal performance during the development process.
6Performance and Lighthouse Scores
Lighthouse scores for performance and accessibility were largely saturated across most frameworks, showing little variance. However, JavaScript bundle size remained a reliable metric for distinguishing framework efficiency. Total Blocking Time and Largest Contentful Paint metrics provided deeper insights, generally favoring Multi-Page Applications (MPAs) over Single-Page Applications (SPAs) due to faster initial rendering.
7The Angular Implementation Challenge
Angular presented unique difficulties during the benchmarking process, failing to produce a working application in two out of five attempts. The agent struggled with SSR configurations, often producing code that was incompatible with the specified reverse proxy environment. These failures were treated as valid outcomes, reflecting the real-world complexity of framework-specific agentic tasks.
8Maintainability via Code Size
Total lines of code served as a primary proxy for maintainability in this study. Frameworks like Hono+htmx demonstrated the ability to achieve similar functionality to Angular while using approximately 30% less code. In an agentic era, smaller codebases directly translate to lower token costs and reduced maintenance overhead for future development cycles.
9Vanilla JavaScript Performance
Surprisingly, the vanilla JavaScript implementation using Vite underperformed compared to several specialized frameworks. This finding challenges the assumption that raw code is always the most efficient path. The results suggest that modern frameworks provide structural advantages that even advanced AI agents can leverage to produce more performant and maintainable web applications.
10Addressing Benchmark Limitations
The study deliberately avoided performance optimizations like Service Workers or image lazy-loading to simulate typical developer behavior. While the benchmark could be manipulated by pre-generating static assets, the test harness explicitly forbade this to maintain the focus on dynamic data handling. Future iterations could expand to include more complex app types or browsers.
11Evidence Over Subjective Intuition
The primary objective of this research is to prove that evidence-based decision-making is possible in the age of AI coding. By moving away from vibe-based justifications, developers can make informed choices about their tech stack. The open-source nature of this benchmark invites the community to contribute, fork, and improve upon these initial findings.
12Future of Framework Benchmarking
This experiment is not intended to be the final word on framework performance. As AI tools and framework architectures evolve, continuous benchmarking will be essential. The hope is that framework authors will use these insights to refine their tools, ultimately leading to a more performant and evidence-driven web development ecosystem for all professionals.
Original article: Hacker News (news.ycombinator.com) — The AI Cost of JavaScript Frameworks