Aligning Coding-Agent Benchmarks with Real-World Developer Task Flows
Current benchmarks fail to capture the multi-step, interactive nature of real-world coding sessions.
1Benchmark Reality Gap
Current benchmarks like SWE-Bench Pro rely on single-message prompts, representing 93.8% of their tasks as feature requests or bug fixes. However, real-world developer sessions in JetBrains IDEs show that these tasks account for less than half of actual agent interactions, creating a significant disconnect between synthetic testing and practical software engineering workflows.
2Analyzing Production Sessions
Researchers analyzed 4,782 sessions containing 14,000 developer messages to understand how engineers actually utilize coding agents. While half of all sessions consist of a single message, 33% involve three or more interactions. These longer sessions reveal that developers frequently interleave planning, code review, refactoring, and execution requests alongside standard feature development.
3The SWE-TaskFlow Approach
SWE-TaskFlow is a proposed framework designed to calibrate existing issue-based benchmarks to specific user populations. It maintains the original repository, environment, and tests while transforming static issues into sequential, multi-step requests. This method preserves the integrity of the original task requirements while better simulating the interactive nature of modern IDE-based development.
4Deconstructing Complex Issues
The methodology splits a single, comprehensive issue into three distinct, ordered requests. Each segment must remain actionable within the repository state left by the previous step, ensuring logical continuity. By breaking down complex requirements, the framework forces agents to handle progressive disclosure, mirroring how developers provide instructions during iterative coding sessions in practice.
5Injecting Repository Questions
To address the absence of exploratory behavior in current benchmarks, the framework generates repository-specific questions. These questions are answerable solely through code analysis and are validated by executable proof scripts. This addition tests an agent's ability to explain project architecture, a common requirement in real-world sessions that is typically ignored by standard benchmarks.
6Optimizing Task Alignment
The framework uses the TaskFlow Alignment Score (TFAS) to optimize the sequence of requests and questions. By comparing session lengths, intent distributions, and transition probabilities, the system achieves a 77.8 score against target production data. This ensures the benchmark distribution closely mimics the actual behavior observed in professional developer IDE environments.
7Impact on Agent Costs
Splitting issues into sequential messages increases agent costs by 2.0 to 2.6 times per task. This expenditure reflects the additional computational effort required to process multi-step interactions. Control conditions confirmed that this cost increase stems from the sequential delivery of requirements rather than the specific wording used in the rewritten task prompts.
8Resolve Rate Stability
Despite the increased complexity of sequential delivery, resolve rates for models like GPT-5.6 Terra remained stable. Other models experienced a minor drop of 4 to 5 percentage points, which is statistically comparable to variance between individual runs. This indicates that progressive disclosure changes the operational effort without drastically altering the final outcome.
9Evaluating Answer Accuracy
For the 255 tasks containing repository questions, the framework evaluates agent accuracy using a separate LLM judge. Accuracy scores range from 69.8% for GPT-5.4 Mini to 83.0% for GPT-5.6 Terra. These metrics provide a granular view of an agent's ability to interpret and explain codebase details beyond simple code generation tasks.
10Limitations of Universal Benchmarks
The research concludes that no universal, realistic coding-agent benchmark exists because user populations vary significantly. Data collections like SpecStory, DataClaw, and SWE-chat exhibit distinct task flows and intent distributions. Consequently, benchmarks must be explicitly calibrated to the specific user population they intend to serve, rather than relying on generic, one-size-fits-all datasets.
11Handling Rare Behaviors
Calibrating benchmarks to frequency distributions can inadvertently push rare but critical behaviors toward zero. Destructive shell commands or security-sensitive reviews, while infrequent, remain consequential for overall system safety. The authors suggest that such behaviors should be evaluated in separate, dedicated stress-test slices rather than being diluted within the broader, frequency-based task flow distribution.
12Future of Agent Evaluation
Moving forward, the focus must shift from static, single-turn evaluations to dynamic, multi-turn interactive benchmarks. By aligning agent testing with the actual task flows of professional developers, the industry can better measure true utility. This shift ensures that agents are optimized not just for solving isolated issues, but for supporting complex, ongoing development workflows.