Meta Details Catching JiTTests: Tests Built to Fail for Each Code Change, Credited With a Key Role in Catching 74 Serious Privacy Bugs in H1 2026

On October 29, 2026, Meta's engineering team described how "Catching JiTTests" (just-in-time catching tests) now runs in production: a large language model generates executable tests for each proposed code change, tests designed to fail so that the failure exposes a bug the change introduced. Meta says 589 engineers had engaged with the system at the time of writing, and that engineers reported a 60% true-positive rate. In the first half of 2026 it played a key role in catching 74 serious (SEV) privacy bugs at the diff stage.

Meta separates the approach from common AI code review. Most LLM-based methods read code the way a human reviewer would and judge from experience where a fault might be. Catching JiTTests instead generates tests that demonstrate a concrete behavioral difference introduced by the change. Meta admits this takes more computational steps, but says it catches classes of bugs that LLM or agent reviews might miss. The difficulty is scale: with hundreds of millions of lines of changing code, the tests include "strong catches" that reveal real bugs and "weak catches" that may be false positives, and the system must tell them apart automatically without slowing engineers down.

The system runs in five stages. Stage 1 is targeting and triggering: it runs overnight on spare capacity and only handles diffs submitted the previous day that are judged risky, using a mechanism similar to Diff Risk Score and trained on earlier code changes. Stage 2 is two parallel test-generation workflows. The "dodgy diff" workflow treats each diff as a mutant, a version with a bug, and uses mutation-guided LLM test generation to produce tests that separate the diff's behavior from its parent's, without inferring intent; the aim is a larger pool of weak catches. The "intent-aware" workflow first has an LLM infer the diff's intent from the code changes, title and summary, then describe the risks, meaning ways an attempt to implement that intent could go wrong. It builds mutants from those risks and generates tests to kill each one. Those tests are then run on the real diff, and the ones that fail are harvested as weak catches.

Stage 3 is the LLM-as-Judge ensemble, several state-of-the-art models that review the weak-catch test code, the execution result (failure code, message, trace), the parent-to-child diff and the inferred intent, and judge whether it is more likely a false positive or a true bug. Stage 4 is RubFake, a rule-based assessor that gives verified, deterministic judgments to offset the non-determinism and hard-to-verify nature of LLM judgments; together they rank the weak catches. Stage 5 is human-on-the-loop: the system decides from the ranking whether to reach out to an engineer, and engineers set criteria and review outcomes in aggregate instead of supervising every output.

Several of the reported figures need their limits kept in view. The 589 engineers are 64% of Meta's outreach, but the post gives no total outreach count. The 60% true-positive rate is what engineers reported. Catching JiTTests is said to have played a key role in catching the 74 serious privacy bugs, and the post puts that in the same sentence as a 20% fall in overall privacy SEVs in that period, but it gives no counterfactual and does not say the system was the only factor. The comparison tables and the impact chart are images in the original, and no value from them is used here.

What other teams can borrow is the shape of the pipeline: pick a few risky diffs with a risk score, generate tests that prove a behavioral difference, filter false positives with both models and rules, and interrupt a person only when a case ranks high. The limits are plain too. The numbers come from Meta's own deployment, the cost is extra compute, and the results concern privacy bugs, so they should not be stretched to every kind of defect.