Shopify's Security Harness: AI-Powered Code Review
Automated scanning system enhancing software security with artificial intelligence.
1Shopify's Security Harness
Shopify's Application Security team developed an AI-powered code review and test harness to protect merchant and customer data. This system automatically detects vulnerabilities in software, validates them with real tests, and provides Shopify-specific fixes to developers. Tests over the past five months using frontier models proved the system more effective than traditional methods.
2Multi-Agent Architecture
The Ruby-based orchestrator named Dispatch manages large-scale agent scans. The system uses specialized agents for different tasks: hunter agents find vulnerability candidates, verifiers test them. These parallel-working agents can scan massive Rails monoliths like Shopify Core. To date, the system has scanned over 80 applications and produced more than 300 security findings.
3Cost-Effective Scanning
A full application scan costs between $50-$300, while incremental diff scans range from $5-$50. This is significantly below the potential cost of prevented security issues. Running 24/7, the system has saved over $400,000 in equivalent bug bounties. Two findings would have been rated Critical according to the company's severity calculator.
4Code Partitioning and Context Management
During initial scans, the system partitions code based on size and scope. It creates documentation of APIs and data models for each partition, reducing rework in subsequent scans. These documents enable agents to make more informed decisions while keeping most of their context window available for primary tasks.
5Test Oracle and PR Automation
The system automatically writes tests and opens GitHub PRs for every vulnerability found. Developers receive ready PRs complete with context, tests, and suggested fixes. This enables quick closure of the feedback loop. The system automates the challenging confirmation of exploitability in web application testing.
6Results and Impact
Over six weeks, thousands of scans produced more than 300 findings ranging from defense-in-depth improvements to resolved security incidents. The system's greatest success was catching complex vulnerabilities human reviewers might miss. In one audit, the model found over 30 candidate issues, though validation downgraded all to low/medium or identified them as false positives.
7Model Evolution and System Resilience
The best tested models discovered findings better than baseline frontier models but also produced more potential issues. As models improve, the harness's permanence emerges as the system's most crucial component. Shopify continues enhancing system effectiveness by regularly updating security models.
8Industry Implications
Shopify's innovation proves the practicality of AI-powered security scans for large codebases. The system's success may encourage similar solutions, particularly in the Rails ecosystem. Features like automatic PR generation and test oracles have potential to set new standards in software security practices.
Original article: Shopify Engineering — Shopify'nin Güvenlik Aracı: Yapay Zeka Destekli Kod Denetimi