# Twinrun: Detect Unintended Code Behavior Changes Compare old and new code outputs to catch unintended regressions - **Source:** [Hacker News: Show HN](https://news.ycombinator.com) - **Published:** 2026-09-05 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/e930ffc2-4173-47a7-8297-cd834f55473e ## The Problem with Code Changes Traditional diffs show which lines changed but don't reveal behavioral differences. Tests only catch what developers explicitly assert. Twinrun addresses the gap between visible changes and actual behavior modifications, especially critical when AI-generated code produces more changes than humans can manually review. ## How Twinrun Works Twinrun runs both old (base) and new (head) code versions on identical inputs and compares outputs. It automatically detects where behavior differs without requiring manual specifications or assertions. The tool treats the old code as the oracle of correct behavior, showing exact calls where outputs diverge unintentionally. ## Practical Example In a billing system, Twinrun caught when Cart.total() changed from returning int to float. While seemingly minor, such changes could cause subtle bugs. Another example showed Cart.add() modifying different object attributes between versions. These findings emerged from actual calls rather than predefined tests. ## Validation Against Real Projects Tested against 509 commits in a project since 2019, Twinrun examined 1630 callables and identified 243 behavioral differences. Most importantly, it detected 5 out of 7 changes that maintainers later reverted - proving it catches real regressions that slip through other review methods. ## Integration Options Twinrun works as a standalone CLI tool or integrates with CI pipelines. The GitHub Action checks PRs against base branches automatically. It can also work with AI coding assistants to verify their changes before submission. The tool fails CI jobs when finding unintended changes by default. ## Technical Implementation The tool analyzes callables whose AST changed plus one level of callers. It checks signatures via interpreter queries and builds probes for each parameter. Twinrun runs code in separate worktree subprocesses, normalizes outputs, filters flaky results, and clusters related changes to identify root causes. ## Deterministic Testing Twinrun runs each probe twice per version to detect non-deterministic behavior. It uses configurable timeouts (default 20s per callable) and limits (24 probes/callable). A seed parameter ensures reproducible testing conditions, while repeats verify consistent behavior. ## Intentional Changes Workflow When behavioral changes are intentional, developers can use --accept to record them as new invariants. These become baseline expectations for future changes. The --note option adds human-readable explanations for why changes were approved. ## LLM Fallback Mechanism When Twinrun can't automatically generate test inputs, it can optionally query LLMs (requires API keys) to suggest parameters. This helps test code paths that are difficult to reach automatically while keeping the actual test execution deterministic. ## Dependency-Free Design Twinrun intentionally avoids dependencies to prevent conflicts with tested codebases. It can run directly from source without installation. The tool imports the code it measures, maintaining isolation from the tested project's environment. ## CI/CD Integration Features The GitHub Action handles shallow clone limitations by checking out base revisions automatically. It provides detailed reports in job summaries and configurable failure conditions. Teams can choose to receive reports without failing builds by setting fail-on-finding: false. ## Why This Matters As AI-assisted coding grows, the volume of changes exceeds human review capacity. Twinrun provides automated verification that behavioral changes match developer intent. It catches subtle bugs that static analysis misses and prevents regressions from reaching production, especially in critical systems. ## Sources covering this story - https://github.com/prian3003/twinrun — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._