Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

Automated root cause analysis (RCA) with large language models (LLMs) has drawn growing attention. Today, SREs typically automate RCA with LLMs in one of two ways: directly using a general-purpose agent (e.g., Codex or Claude Code) for diagnosis, or building a specialized RCA agent from scratch. As mainstream general agents grow more capable and iterate quickly, our quantitative study finds that the former now often surpasses the latter. Its accuracy, however, still falls short of production needs, and this gap stems mainly from the external adaptation layer outside the agent's general capabilities, namely the harness. We therefore argue that LLM-based RCA should focus on this external harness, reusing the strong general capabilities of a modern agent rather than rebuilding an agent from scratch. A key capability of such a harness is to self-evolve, accumulating system-specific experience from past diagnoses so that it gets better the more it is used. We introduce OpsHarness, a self-evolving RCA harness that turns diagnosis experience into reusable expertise. Its data plane combines layered operational knowledge with an idea-card tool library, while its control plane coordinates setup, diagnosis, evolution, and verification. During evolution, OpsHarness contrasts successful and failed trajectories, converts their evidence into atomic proposals, and admits updates only through a dual-gate verification process designed to prevent overfitting and regression. Across two public benchmarks and an industrial deployment, OpsHarness achieves 59.0% top-1 accuracy, improving over a bare general agent by 63.4% and over baseline RCA agents by 4.02$\times$.

Root cause analysis (RCA) of microservice failures requires reasoning over telemetry far larger than a model's context, so a language agent reasons over extracted evidence. We start from an error analysis of such an agent on OpenRCA. A reverse reasoning agent reconstructs an evidence chain to each labelled cause and compares it against the agent's own trace, locating failures where a grounded chain existed but was not followed; its labels agree with two site reliability engineers. Being label-conditioned, it audits evidence rather than testing forward identifiability, leaving open what the agent lacked: procedure or telemetry-specific knowledge. Existing systems cannot separate them, extracting evidence inside the reasoning loop. We compute the evidence once, offline, and supply it identically to every arm. Rules mined from the audit reports of one Market cloudbed raise Full Score on a held-out cloudbed from 14.10 to 35.90, surviving correction for multiple testing, against 24.36 for manual rules and 19.23 for a label-and-schema control. The rules add 17 solved cases on top of the five-step procedure but 11 without it, the repairs overlapping. Removing the procedure raises Full Score in all three conditions tested, though no contrast survives correction. Deleting only its earliest-onset criterion yields a net gain of 8 cases against 11 for the whole procedure, which we report descriptively. Mining is label-supervised, and all results are specific to OpenRCA.