Real-SWE: Benchmarking AI Models on Private Enterprise Codebases
Evaluating AI's ability to handle real-world software engineering tasks.
1Introduction to Real-SWE
Real-SWE is a new benchmark designed to evaluate frontier AI models using private, real-world enterprise codebases. Each task originates from a licensed production codebase of a real company, reflecting the complexities and context engineers face daily. This benchmark aims to answer whether AI can perform tasks akin to a software engineer in practical scenarios.
2Challenges in AI Benchmarking
Traditional benchmarks often use expert-generated or synthetic tasks, which lack the real-world complexities of enterprise environments. Real-SWE tasks differ by incorporating actual coding artifacts and specific instructions, presenting a more challenging scenario for AI models to navigate and solve effectively.
3Methodology of Evaluation
The evaluation employs native harnesses that mimic how enterprise engineers operate, assessing not just the AI models but their integration with these harnesses. This approach ensures that the assessment reflects real-world application and utility of AI in software engineering tasks.
4Example Task: Fixing Invoice Billing
One task involves correcting invoice billing to ensure accurate tax application and exemption handling. This requires understanding diverse tax handling methods across businesses, integrating with tax authority providers, and ensuring compliance with legal standards, showcasing the practical challenges AI must overcome.
5Tools and Services Integration
Tasks in Real-SWE involve a variety of tools and services like AWS, Docker, Kubernetes, and GitHub, reflecting the cross-functional nature of enterprise software engineering. AI models must navigate these environments, understanding and utilizing the necessary tools to complete tasks effectively.
6Codebase Selection Criteria
Codebases for Real-SWE were selected based on rigorous criteria including substantial usage, strong engineering teams, and demanding production workloads. This ensures that the benchmark tasks are representative of real-world challenges faced by enterprise engineers.
7Complexity of Tasks
Tasks in Real-SWE are complex and cross-functional, often requiring changes across multiple parts of an application. AI models must understand existing business logic and coding patterns, ensuring that changes do not disrupt the surrounding system.
8Performance Metrics
Initial results show that 71.4% of rollouts under 10 minutes failed, compared to 73.4% of longer rollouts. These metrics highlight the difficulty AI models face in triaging systems and understanding requirements within complex codebases.
9Analysis of Task Failures
Failures in tasks are categorized based on observed submission behaviors, such as incorrect system wiring or delivering changes to unused parts of the application. This taxonomy helps in understanding the specific challenges AI models encounter in real-world tasks.
10Model Performance Comparison
Different AI models show varying success rates in Real-SWE tasks. For example, Gemini 3.8 Flash has a resolution rate of 31.2%, while GPT-5.6 Sol resolves 16.2% of tasks. These figures provide insights into the current capabilities and limitations of AI in enterprise environments.
11Implications for AI Development
The Real-SWE benchmark underscores the need for AI models to better handle real-world software engineering tasks. It highlights the gap between theoretical AI capabilities and practical application, guiding future developments towards more robust and adaptable AI solutions.