Twinrun: Detect Unintended Code Behavior Changes
Compare old and new code outputs to catch unintended regressions
1The Problem with Code Changes
Traditional diffs show which lines changed but don't reveal behavioral differences. Tests only catch what developers explicitly assert. Twinrun addresses the gap between visible changes and actual behavior modifications, especially critical when AI-generated code produces more changes than humans can manually review.
2How Twinrun Works
Twinrun runs both old (base) and new (head) code versions on identical inputs and compares outputs. It automatically detects where behavior differs without requiring manual specifications or assertions. The tool treats the old code as the oracle of correct behavior, showing exact calls where outputs diverge unintentionally.
3Practical Example
In a billing system, Twinrun caught when Cart.total() changed from returning int to float. While seemingly minor, such changes could cause subtle bugs. Another example showed Cart.add() modifying different object attributes between versions. These findings emerged from actual calls rather than predefined tests.
4Validation Against Real Projects
Tested against 509 commits in a project since 2019, Twinrun examined 1630 callables and identified 243 behavioral differences. Most importantly, it detected 5 out of 7 changes that maintainers later reverted - proving it catches real regressions that slip through other review methods.
5Integration Options
Twinrun works as a standalone CLI tool or integrates with CI pipelines. The GitHub Action checks PRs against base branches automatically. It can also work with AI coding assistants to verify their changes before submission. The tool fails CI jobs when finding unintended changes by default.
6Technical Implementation
The tool analyzes callables whose AST changed plus one level of callers. It checks signatures via interpreter queries and builds probes for each parameter. Twinrun runs code in separate worktree subprocesses, normalizes outputs, filters flaky results, and clusters related changes to identify root causes.
7Deterministic Testing
Twinrun runs each probe twice per version to detect non-deterministic behavior. It uses configurable timeouts (default 20s per callable) and limits (24 probes/callable). A seed parameter ensures reproducible testing conditions, while repeats verify consistent behavior.
8Intentional Changes Workflow
When behavioral changes are intentional, developers can use --accept to record them as new invariants. These become baseline expectations for future changes. The --note option adds human-readable explanations for why changes were approved.
9LLM Fallback Mechanism
When Twinrun can't automatically generate test inputs, it can optionally query LLMs (requires API keys) to suggest parameters. This helps test code paths that are difficult to reach automatically while keeping the actual test execution deterministic.
10Dependency-Free Design
Twinrun intentionally avoids dependencies to prevent conflicts with tested codebases. It can run directly from source without installation. The tool imports the code it measures, maintaining isolation from the tested project's environment.
11CI/CD Integration Features
The GitHub Action handles shallow clone limitations by checking out base revisions automatically. It provides detailed reports in job summaries and configurable failure conditions. Teams can choose to receive reports without failing builds by setting fail-on-finding: false.
12Why This Matters
As AI-assisted coding grows, the volume of changes exceeds human review capacity. Twinrun provides automated verification that behavioral changes match developer intent. It catches subtle bugs that static analysis misses and prevents regressions from reaching production, especially in critical systems.
Original article: Hacker News: Show HN — Twinrun: Detect Unintended Code Behavior Changes