# Evaluating AI Agents' Testing Technique Effectiveness Study reveals how coding agents perform with different testing methodologies. - **Source:** [Dan Luu](https://danluu.com) - **Published:** 2026-09-07 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/04d33089-e104-4452-b588-a8eaa8f941a0 ## The Testing Paradox in AI Development While modern coding agents make it easier to achieve basic quality standards, overall software quality appears to be declining. This suggests default testing approaches may be inadequate. Researchers conducted experiments to measure how well AI agents implement various testing techniques when given simple instructions, without requiring deep testing expertise from the prompter. ## Experimental Setup The study reused the Zstd implementation evaluation framework to compare 26 different testing approaches. Agents were instructed to implement Zstd in Rust using specific techniques like TDD, property-based testing, or formal verification methods. Additional evaluations included IMAP RFC implementations. Each condition was tested with 80 runs to ensure statistical significance. ## Tested Methodologies The study examined diverse approaches including ACL2, Alloy, Creusot, differential testing, fuzzing, Hegel, Kani, Lean 4, metamorphic testing, mutation testing, QuickCheck, SMT solvers (Z3, cvc5, Yices), TLA+, and Verus. Four specialized testing skills were also evaluated: Hegel's official skill, ECC Rust test skill (250k GitHub stars), Trail of Bits property test skill, and a custom-written test skill. ## Pre-Registered Hypotheses Researchers made specific predictions before seeing results: TDD would underperform (55% confidence), formal methods wouldn't outperform (52% confidence), 'Make no mistakes' joke condition would fail (95% confidence), and the popular ECC test skill wouldn't help (65% confidence). These reflected skepticism about current trends in agent testing approaches. ## Skill Evaluation Concerns The study raised questions about skill effectiveness. The ECC skill's TDD focus might harm performance, while Hegel's 20k-token skill resembled a tutorial more than agent instructions. Trail of Bits' skill contained potentially useful information but was large. Researchers noted low confidence in skill predictions due to limited experience evaluating them. ## Visualization Approach Results were presented in intentionally dense graphs showing cost versus test pass rates, with bootstrap covariance and 50% uncertainty indicators. While challenging to read, these visualizations revealed subtle patterns that simpler graphs might miss. Color coding grouped similar approaches (blue for formal methods, green for property-based testing). ## Key Finding: Default Performance Surprisingly, the default condition (no special instructions) performed above average. This suggests that current agent training may already incorporate reasonable testing practices by default, or that additional instructions often introduce complexity without corresponding benefits. ## Formal Methods Results At higher effort levels, fuzzing and property-based testing conditions slightly outperformed formal methods on average. However, results varied significantly at medium effort levels, indicating that implementation difficulty might affect different approaches disproportionately. ## Skill Performance Testing-related skills recommended by Codex generally underperformed, though a custom quick skill showed some promise. This raises questions about the value of large, complex testing skills versus simpler, more targeted approaches in agent development. ## TDD Hypothesis Confirmed The prediction about TDD underperforming proved correct, supporting concerns that agents might not properly implement TDD principles when simply instructed to use the methodology without detailed guidance. ## Formal Methods Validation The hypothesis about formal methods not significantly outperforming held true, suggesting that while valuable, they don't provide decisive advantages over good testing techniques for relatively simple implementation tasks. ## Implications for AI Development The study highlights the need for better default testing approaches in coding agents and suggests that simply instructing agents to use specific techniques may not yield expected improvements. This has significant implications for how we train and utilize AI in software development workflows. ## Sources covering this story - https://danluu.com/agentic-testing/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._