Reviewing Agent-Generated Code Through Blast Radius Analysis
Shift from line-by-line inspection to evidence-based verification for AI-generated software changes.
1The Challenge of Cheap Code
As AI agents generate code rapidly, human reviewers face a bottleneck. Reading every line is unsustainable when output volume increases. The core problem is distinguishing between requested features and unintended side effects. We need a systematic approach to decide where human inspection is essential versus where automated evidence suffices for approval.
2Defining the Blast Radius
Blast radius measures how far a mistake can spread within a system. A small diff, such as adding a retry mechanism to a shared payment client, can impact every refund request. Reviewers must identify these boundaries before implementation to ensure that changes remain contained within the intended scope of the feature.
3Implementing Bidirectional Traceability
Bidirectional traceability connects requirements to implementation and verification. For every important requirement, developers must map the code and tests that support it. Conversely, any implementation change must be traceable back to a specific requirement. This process helps identify unexplained behavior that agents might introduce during the development of requested features.
4Detecting Unintended Behavior Drift
Agents often add functionality, like retries, that was not explicitly requested. While sometimes beneficial, these additions can introduce risks, such as duplicate payments. Reviewers should treat these as open decisions rather than assuming they are correct. If the agent modifies shared components, the impact must be evaluated beyond the primary feature request.
5The Risk of Shared Test Mistakes
Tests can pass while still being fundamentally wrong if the agent misinterprets a requirement. For example, if an agent interprets 'above 100' as '100 and above' in both code and tests, the system will appear correct. Reviewers must verify the meaning of assertions against agreed-upon boundary examples to prevent false confidence.
6Validating Through Controlled Failure
To ensure tests are effective, verify that they actually fail when the code is broken. Temporarily removing an approval check in a test run can confirm if the assertion is robust. If the test remains green, it provides no protection. Always check that assertions match the specific requirements of the business logic.
7Standardizing the Review Record
Every pull request should include a short review record. This document must state the agreed behavior, provide evidence from tests, define the blast radius, and list open questions. This record serves as the foundation for the review, allowing the human to focus on evidence rather than manually reconstructing the agent's logic.
8Comparing Against Agreed Baselines
Reviewers must compare new tests against the original requirements, not just the previous code state. If an agent updates a test to match new, unintended behavior, the reviewer must decide if that change is intentional. Quietly updating specs to match code hides drift and prevents necessary discussions about changing business requirements.
9Setting Review Depth by Impact
Review depth should be proportional to the potential impact of a mistake. If a change affects financial transactions, direct inspection is mandatory. For routine changes with established, reliable checks, reviewers can reduce line-by-line inspection. This risk-based approach allows teams to maintain high standards while scaling the volume of accepted code changes.
10Integrating Evidence into Development
Make the review record part of the definition of done. The developer submitting the code owns its accuracy, while the reviewer owns the acceptance decision. If evidence is inadequate or the blast radius is unclear, the change remains open. This practice builds on familiar development habits like acceptance testing and CI.
11Scaling Through Controlled Rollouts
Containment is vital for managing agent-generated code. Keep features separate from shared infrastructure redesigns to simplify reviews. Use controlled rollouts, such as A/B testing, to observe behavior in production before full deployment. This strategy ensures that even if a mistake slips through, its impact remains limited and manageable for the team.
Original article: O'Reilly Radar — Reviewing Agent-Generated Code Through Blast Radius Analysis