# Scaling Just-in-Time Test Generation for Proactive Bug Detection Meta implements LLM-driven testing to identify production bugs before code deployment. - **Source:** [Engineering at Meta](https://brifing.si/source/28383801-e365-4f15-8f20-49d8c0635946) - **Published:** 2026-10-29 - **Topic:** [Software & Engineering](https://brifing.si/c/engineering) - **Original article:** - **On Brifing:** https://brifing.si/s/e5bd84c4-37f4-49cd-ad0c-8cf8dd8da492 ## Proactive Bug Detection Strategy Meta has transitioned from traditional testing to Just-in-Time Tests (JiTTests) to manage massive code complexity. By leveraging large language models, the system generates executable tests that identify behavioral differences in code changes. This proactive approach catches bugs before they land in production, addressing limitations inherent in conventional automated testing methods. ## Dual Workflow Implementation The system utilizes two primary workflows to maximize bug detection efficiency. The intent-aware workflow infers the purpose of a code change to target failure points, while the dodgy diff workflow treats every change as a potential mutant. These tandem processes ensure comprehensive coverage by analyzing both developer intent and potential code regressions. ## Filtering False Positives Distinguishing true bugs from false positives is managed through an ensemble approach. The LLM-as-Judge system aggregates assessments from multiple state-of-the-art models to classify weak catches. This is complemented by RubFake, a deterministic rule-based assessor that provides verified judgments, ensuring that only high-probability bug candidates proceed to the final review stage. ## Systematic Workflow Stages The five-stage workflow begins with targeting risky diffs using a Diff Risk Score. After generating tests through the two workflows, the system employs the LLM-as-Judge ensemble and RubFake to rank candidates. This structured pipeline allows for automated filtering, ensuring that only the most critical issues are escalated for further human consideration. ## Human-on-the-Loop Integration Unlike traditional human-in-the-loop models, Meta employs a human-on-the-loop approach to minimize engineering friction. Engineers define success criteria and monitor outcomes at an aggregate level rather than supervising every individual output. This shift allows developers to intervene at the policy level, significantly reducing the manual review load for individual test cases. ## Operational Impact Metrics The deployment has yielded measurable improvements in software reliability. During the first half of 2026, the system identified 74 serious privacy bugs at the diff stage. This intervention contributed to a 20% reduction in the overall occurrence of privacy-related SEVs, demonstrating the effectiveness of AI-generated testing in maintaining large-scale platform stability. ## Engineer Engagement Results Adoption rates indicate strong utility among the engineering team, with 589 engineers engaging with the system. Participants reported that 64% of the outreach resulted in actionable findings, maintaining a 60% true-positive rate. These metrics confirm that the system provides genuine value by surfacing critical issues without overwhelming developers with false alarms. ## Scalability and Future Outlook The JiTTests approach proves that LLMs can provide a scalable safety layer for massive codebases. By automating test generation and focusing human oversight on aggregate policy, Meta has demonstrated a generalizable model for modern software development. This methodology effectively bridges the gap between rapid deployment cycles and the necessity for rigorous, reliable testing. ## Sources covering this story - https://engineering.fb.com/2026/10/29/security/catching-bugs-before-they-land-jittestings-human-on-the-loop-approach-at-full-scale/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._