Real-SWE: Benchmarking AI Models on Private Enterprise Codebases
New benchmark tests AI coding agents on real-world engineering tasks from private codebases.
1Introduction to Real-SWE
Real-SWE is a new benchmark designed to evaluate frontier AI models using private, real-world enterprise codebases. Unlike synthetic tasks, it uses actual engineering problems from production environments licensed from real companies. The benchmark focuses on tasks that reflect the complex, contextual work software engineers face daily, including understanding existing architecture and operational constraints.
2Real-World Task Complexity
Tasks in Real-SWE differ from synthetic benchmarks in two key ways: the underlying coding artifacts and instruction specificity. These complexities challenge current AI models by requiring them to navigate real production code with all its dependencies and business logic. Examples include fixing invoice billing systems with varied tax requirements across different businesses on a platform.
3Enterprise-Grade Evaluation
The benchmark uses native harnesses to evaluate model performance in environments that mirror how enterprise engineers work. This means testing model-and-harness combinations rather than isolated models, providing a more accurate assessment of real-world utility. The evaluation covers cross-functional work that may span multiple parts of an application.
4Codebase Selection Criteria
Real-SWE's codebases were selected through rigorous screening of real companies with substantial usage and strong engineering teams. The focus was on production code written to meet actual business needs rather than artificial benchmark tasks. This ensures the tasks preserve the architectural understanding and behavior preservation required in real engineering work.
5Task Design Philosophy
Tasks are designed to be slightly underspecified, requiring AI agents to discover implementation details in the codebase and surrounding tools - similar to how engineers work. Any behavior required by verifiers must be either stated or reasonably discoverable, balancing specificity with real-world ambiguity. This approach mirrors benchmarks like DeepSWE and Terminal Bench.
6Cross-Functional Challenges
The benchmark highlights how real engineering tasks often span multiple systems and require understanding existing business logic. A single change might affect various parts of an application, demanding that AI agents comprehend company-specific coding patterns while maintaining system functionality. This contrasts with isolated coding problems in synthetic benchmarks.
7Performance Statistics
Initial results show significant failure rates: 71.4% of rollouts under 10 minutes failed, compared to 73.4% of longer rollouts. The benchmark includes 11 files compared to 6 in FrontierCode and DeepSWE. Failure analysis uses a consistent taxonomy across models, with percentages calculated from each model's failed runs rather than all runs.
8Common Failure Modes
Analysis reveals typical AI shortcomings: building on unverified system assumptions, correct ideas implemented incorrectly in context, or delivering changes to unused code paths. These failure modes mirror challenges human engineers face when working with unfamiliar codebases, suggesting areas for AI model improvement in real-world scenarios.
9Model Performance Comparison
Preliminary results show varying performance across models: Gemini 3.8 Flash achieved 31.2% success at $2.50 per task, GPT-5.6 Sol 16.2% at $2.65, and Muse Spark 1.3 23.8% at $2.74. These metrics provide the first comparative data on how different AI models handle authentic enterprise coding challenges beyond synthetic benchmarks.
10Implementation Environment
Tasks expose only necessary services for each workflow, including tools like AWS emulator, Docker, Kubernetes, GitHub, and various databases. The environment reflects real engineering stacks with technologies like NestJS, TypeScript, PostgreSQL, MySQL, MongoDB, Redis, Go, Python, and Node.js, plus collaboration tools like Slack and Linear.
11Significance for AI Development
Real-SWE represents a significant step toward evaluating AI coding assistants on authentic engineering work. By testing on private enterprise codebases, it reveals how current models perform on the complex, contextual tasks that constitute real software engineering, moving beyond artificial benchmarks to measure practical utility in professional environments.
Original article: Hacker News — Real-SWE: Benchmarking AI Models on Private Enterprise Codebases