Netflix Pensive AI Diagnoses Failed Data Jobs by Returning Log Line Numbers

Netflix wraps AI around fixed rules to diagnose failed data jobs

On October 26, the Netflix Technology Blog published a long account of Pensive AI, a system that helps engineers understand why data and machine-learning workloads failed. Netflix says that on peak days roughly 100,000 workloads fail across engines such as Titus, Spark, Trino and Druid. The earlier Pensive read the logs of a failed job, used regular expressions and hand-written Java rules to classify the error, and passed retry advice to tools such as the Maestro workflow scheduler. That made about half of failures self-service. Over one three-month period it handled roughly seven million classifications, and about half landed in an unknown bucket. Pensive AI was built to cover that gap without discarding the rules that already worked.

Extraction by line number, not by copying

The first feature, error extraction, runs on every failed step. The design choice is how the model answers. Pensive numbers the lines of each log and shows the model records of line number and text. The model returns only two integers, the first and last line of the error, and Pensive rebuilds the text from its own copy of the log. Netflix gives two reasons: producing output tokens is slower than reading input, and the displayed text now comes straight from the original log, which reduces invented content. The model can still choose the wrong range, so extraction quality is part of the evaluations. The call shares one deadline inside Maestro's hard 90-second limit for post-processing a failed step, and a log call that misses the deadline is dropped so the response still returns on time. Netflix kept the old error field and added the new one beside it. The migration is unfinished: the Data Portal view for failed Maestro steps reads the new field, while the query editor, notebooks and some Slack clients still use the old path.

Debugging on demand, cached per failure

The second feature, Pensive AI Debugging, produces root-cause analysis and suggested fixes. It is available through APIs, MCP and the Data Portal interface. A response takes tens of seconds or a minute, so Netflix runs it only when someone asks, because most failed steps are never opened. The failed step attempt is the unit of work. Its result is cached, a second request becomes a key-value read, a conditional write stops two callers generating the same answer at once, and completed responses expire after 90 days. The service streams the answer, and each event carries the full text so far, which keeps the client simple after a dropped event. Generation continues if the client disconnects.

How Netflix checks quality

Netflix uses the old rules as a frozen source of labels. One model judge checks whether the known error is still represented, and another checks whether each extracted error is meaningful on its own. A script flags lines that appear in more than about 30% of workflows, and latency is recorded for every call. Logs expire after seven to 30 days, so production requests archive them to S3 and evaluations replay the archive. Each pull request deploys to a preview environment, precision and inclusion thresholds gate the merge, and the same checks run after promotion and on a schedule. Each thumbs-down on a debugging answer sends a Slack alert during business hours.

What the figures do and do not show

Netflix says AI extraction now covers 100% of failures, including those the old rules labeled unknown. Coverage means an extraction is produced, not that each one is right. The post also reports that failed attempts where nobody used Pensive AI averaged 321% more manual debug-log actions than attempts where someone did. Netflix calls this an observational comparison, not a causal claim. The post gives no accuracy rate, no time-to-resolution figure and no cost, so other teams should measure those on their own failures before copying the design.