# Measuring AI’s Impact on Software Delivery Speed Rigorous experimental design is essential to distinguish AI-driven gains from process improvements. - **Source:** [O'Reilly Radar](https://www.oreilly.com/radar) - **Published:** 2026-11-04 - **Topic:** [careers](https://brifing.si/c/careers) - **Original article:** - **On Brifing:** https://brifing.si/s/b3b6ff83-802b-4bd5-b58c-d51cf426722d ## Delivery Speed Versus Productivity Delivery speed is a component of productivity but not its entirety. True software development productivity is defined as the value produced divided by the money spent. Going faster is only beneficial if the team is moving in the right direction, as speed without utility does not equate to actual business value. ## The Fallacy of Standardized Metrics Objective measurement of development speed is difficult because software outputs are not standardized. Lines of code, pull requests, and story points are arbitrary metrics that vary based on task complexity rather than team performance. Relying on estimates is also flawed, as teams are historically prone to being overly optimistic about their timelines. ## Comparing Teams Against Themselves To avoid noise from individual working styles, teams should be compared against their own historical performance. By analyzing the ratio of estimated time to actual time spent, organizations can identify shifts in speed. This internal benchmarking helps mitigate the impact of varying skill levels and different approaches to task estimation among developers. ## Addressing Cognitive Bias in Estimates Developers often adjust their estimates based on their intent to use AI, potentially skewing results. To maintain data integrity, estimates must be collected before the developer knows whether they will use AI for a specific task. Randomly assigning tasks ensures that the measurement remains objective and free from pre-existing performance expectations. ## Isolating AI from Process Changes AI adoption often coincides with broader process improvements, such as better communication or streamlined workflows. These changes can mask the true impact of AI tools. To isolate the effect of AI, organizations should interleave tasks performed with and without AI, ensuring that both approaches benefit from the same optimized development processes. ## Randomized Controlled Trial Methodology The most rigorous approach is a randomized within-subject repeated-measures trial. Each participant performs a mix of tasks with and without AI tools. By comparing each individual's performance against their own baseline, the number of required samples decreases, providing a clearer statistical view of how AI influences actual time spent on tasks. ## Accounting for Total Effort Speed increases must be calculated in the context of the full work week. If a team achieves a two-times improvement in coding speed but spends half their time on non-coding activities, the overall productivity gain is significantly lower. Leaders must track total effort and interruptions to understand the real-world impact. ## The Cost of AI Investment With AI spending reaching thousands of dollars per person monthly, engineering leaders must justify these costs through measurable outcomes. Without rigorous data, companies risk overestimating AI benefits, as seen in studies where engineers overestimated speed gains by nearly 50 percent. Measuring ROI is now a critical requirement for modern software management. ## Sustainability and Future Analysis Measuring delivery speed is only the first step in assessing AI's organizational impact. Future analysis must account for unintended consequences, such as code quality issues or long-term maintenance burdens. Understanding these trade-offs is essential for determining whether AI-driven speed improvements are sustainable and truly beneficial for the business's long-term goals. ## Sources covering this story - https://www.oreilly.com/radar/measuring-ais-impact-on-delivery-speed/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._