ORCA-bench Released: Exposing the Capability Gap of LLM Agents in Real-World Operations Root Cause Analysis

Background and Context

As large language models (LLMs) increasingly demonstrate capabilities in writing, patching, and searching code, a critical yet frequently overlooked domain remains production-level Root Cause Analysis (RCA). Traditional code agents are typically designed to handle well-defined programming tasks, whereas real-world operations on-call scenarios present a fundamentally different challenge. Engineers often must start from vague user fault reports and, hours after an incident occurs, navigate through massive volumes of noisy metrics, logs, distributed traces, and extensive source code repositories to perform complex causal reasoning. Existing benchmarks predominantly focus on code generation accuracy or simple error correction, lacking the capacity to evaluate this high-noise, multi-modal data fusion reasoning capability.

To address this gap, the ORCA-bench benchmark has been introduced, specifically designed to test general-purpose code agents within high-fidelity, production-grade operations environments. The core contribution of this work lies in constructing a testbed that closely mirrors real production settings. It provides rich telemetry data interfaces while ensuring the reliability of ground truth through expert annotation. This initiative fills a significant void in evaluating the reasoning capabilities of agents in complex operational scenarios, moving beyond static code evaluation to dynamic system diagnostics.

Deep Analysis

Technically, ORCA-bench constructs a microservice system based on OpenTelemetry, exposing six days of real-world telemetry data. Users and agents can access data through authentic telemetry interfaces, including retrieving metrics via Prometheus, viewing distributed traces through Jaeger, and searching logs via OpenSearch in Grafana, all while maintaining full access to the complete source code. This design simulates the toolchain used by real operations engineers, ensuring ecological consistency in the testing environment. The benchmark comprises 1,079 RCA tasks, which vary systematically in report specificity, fault detection time windows, and concurrent fault scenarios to test model robustness across different complexity levels.

To ensure evaluation fairness, ground truth symptoms were meticulously curated and signed off by expert Site Reliability Engineers (SREs). Furthermore, the study employed LLMs as judges for scoring, complemented by independent human re-scoring. The calculated Cohen's kappa coefficient of 0.90 demonstrates a high degree of consistency between automated and manual assessments. This rigorous data construction and evaluation process provides a reproducible and credible benchmark for subsequent research, establishing a new standard for reliability in AI-driven operations testing.

Industry Impact

Experimental results from evaluating five frontier code agents reveal significant deficiencies in current technology within production operations scenarios. In medium-difficulty tasks, which simulate real-world inputs, the best-performing model achieved an RCA accuracy of only 25.3%. In more difficult scenarios, this accuracy dropped further to 10.0%. This performance gap persists even in the latest models such as Claude Fable 5, indicating that the issue is not merely a matter of model scale. More concerning is that the weakest models generated untrustworthy root cause hallucinations in 40% of fault reports, showing that models are prone to fabricating plausible but entirely incorrect explanations when lacking sufficient constraints.

Ablation studies further revealed that removing access to source code led to a significant decline across all metrics, proving that code context is crucial for accurate reasoning. These experiments were conducted on a carefully curated 50GB, six-day testbed with tasks investigated in isolated systems. Since real production systems are larger, more dynamic, and more specific, the reported performance gap represents a lower bound. This implies that the engineering investment required to safely delegate frontier code agents to production reliability efforts will far exceed the levels suggested by current benchmarks.

Outlook

The release of ORCA-bench holds profound implications for both the open-source community and industrial implementation. Firstly, it provides a publicly available dataset and testing platform, enabling researchers to standardize the evaluation of agent capabilities in the operations domain and promoting comparable research. Secondly, it highlights the shortcomings of current large model agents in complex reasoning and hallucination suppression, pointing directions for future model optimization, particularly in multi-modal data fusion and causal reasoning mechanisms.

For the industry, this benchmark serves as a reminder that while LLMs excel in code generation, significant engineering customization and human supervision are still required in critical operations safety domains to ensure security. Finally, ORCA-bench emphasizes the importance of real data distribution and toolchain integration, encouraging the community to build evaluation frameworks that are closer to production environments. This work drives the evolution of agents from "code assistants" to "reliable operations partners," offering an honest assessment of current AI boundaries and setting higher standards for future research in autonomous system reliability.