cs.AIOct 15, 2026
SaveDiviner: Agentic RCA with Human-Governed Adaptation for Large-Scale Infrastructure
Organizations: Meta, Inc., USA
Abstract
Root-cause analysis (RCA) in large-scale distributed systems is challenging because a single symptom can cross service, data, configuration, compute, and organizational boundaries. Responders must assemble incomplete evidence and coordinate experts who hold different parts of the causal path. Generic language-model agents do not solve this problem through tool access alone: they can use stale or mistimed context, interpret failed queries as absent symptoms, report unsupported system state, and overload observability services designed for human-paced use. We present Diviner (Domain-Informed Verifiable Incident Navigation through Evidence and Review), an agentic RCA system operating at Meta and organized around a four-stage reliability lifecycle. Encode combines a large corpus of reviewed incidents with service-owner expertise to produce scoped, versioned skills, workflows, typed tools, and an incident knowledge graph. Execute uses an adaptive Lead and context-isolated workers to apply relevant knowledge to point-in-time evidence and maintain evidence-linked hypotheses. Evaluate compares saved investigation trajectories with responder-reviewed outcomes to measure exact cause, useful direction, and Groundedness. Adapt uses recurring gaps and on-call feedback to derive focused operational- knowledge updates, admitting them only after replay, quality checks, and owner approval.
Figures & tables
Figure 1. Anatomy of a cross-service incident investigation. Solid arrows show representative paths from candidate causes through failure mechanisms and local effects to downstream impact. The dashed path above shows responders tracing the incident in the reverse direction, while service experts examine different parts of the system in parallel.
Figure 2. Relative mitigation time by cause in reviewed ranking-infrastructure incidents. Values are normalized to the corpus-wide median; bars span Q1–Q3 and dots mark category medians on a logarithmic axis.
| Requirement | Why it is needed | Design obligation |
|---|---|---|
| Grounding | Plausible but absent, stale, or unsupported details can misdirect responders. | Link findings to current, attributed evidence; distinguish empty data from tool failure; abstain when support is insufficient. |
| Control and load | Adaptive planning and machine-speed queries can amplify a wrong branch, retry loop, or backend load. | Keep the plan revisable while bounding capabilities, concurrency, retries, context, deadlines, and resource use. |
| Knowledge | Models lack local state, while procedures, ownership, and interfaces drift. | Encode owned, versioned artifacts and track their applicability and quality. |
| Evaluation | A fluent answer can hide a lucky guess, unsupported reasoning, or wasted work. | Evaluate outcome and trajectory, including direction, grounding, failed checks, and efficiency. |
| Performance | Assistance must arrive before responders find the same evidence. | Measure time to supported findings and final conclusions, including tail latency. |
| Action boundary | Tools access sensitive systems, but the model has no authority to change them. | Preserve service authorization and auditability; leave mitigation to responders. |
Table 1. Requirements for AI-assisted operational investigation.
Figure 3. Diviner’s four-stage lifecycle. Encode builds governed diagnostic knowledge; Execute combines it with live evidence; Evaluate identifies gaps in the saved trajectory; and Adapt publishes replayed, owner-approved updates.
Figure 4. IKG construction and matching. Reviewed incidents are distilled into semantic views and normalized factors, which form a versioned dual index. At runtime, preliminary symptoms and incident context form a query; semantic retrieval supplies candidates and shared factors constrain and explain the matches. Returned precedents remain hypotheses until verified with live tools.
Figure 5. Executing an investigation. The Lead converts incident context and governed knowledge into focused worker goals. Workers gather live observations through registered tools; an evidence layer preserves their source and status, and the runtime Review Agent checks the result before it reaches a human responder.
| Stage | Action | Recorded state |
|---|---|---|
| Discover | Search compact metadata; rank relevance first | Query and candidate identifiers |
| Bind | Resolve an enabled artifact version | Identifier and version |
| Dispatch | Construct a scoped worker context and tool view | Worker, plan step, and actual tool view |
| Update | Replay and review before publication | Proposal lineage and new version |
Table 2. Registry lifecycle and recorded state.
| Outcome | Interpretation |
|---|---|
| Data returned | Candidate observation after scope and provenance checks |
| Valid empty result | Potential counter-evidence only for the requested scope |
| Invalid or denied | No evidence; correct the request or report the authorization failure |
| Timeout, throttle, or backend failure | No evidence; retain the operational failure |
Table 3. Normalized tool outcomes and their evidentiary meaning.
| Measure | Result |
|---|---|
| Observed TDM difference | Approximately 52% lower (95% CI: 45.7–57.9%) |
| Observed pre-mitigation effort | Approximately 15% lower (95% CI: 11.0–18.1%) |
| Investigation quality | Full scored operational set; 45% Exact RCA, 60% useful Direction, and 80% mean Groundedness |
| Operational agent latency | Final conclusion: approximately 15 min p50 and 24.0 min p95; first useful, medium-confidence hypothesis: 7.0 min p50 and 13.7 min p95 |
| Early-trigger advantage | Agent-specific alerts launch investigation before responder-facing incident thresholds and provide roughly a half-hour head start |
| Knowledge-update activity | cycles; about 30 proposed new skills; all adopted changes passed replay and human approval |
Table 4. Operational summary
| Measure | Included records | Definition |
|---|---|---|
| TDM | Incidents with valid detection and mitigation times | Detection to mitigation, including investigation, coordination, and recovery |
| Final latency | Completed agent runs | End-to-end agent runtime from trigger acceptance to the final conclusion |
| First direction | Runs with fine-grained checkpoints | Trigger acceptance to the first useful, medium-confidence hypothesis |
| Exact RCA | Investigations with reviewed outcomes | Specific causal mechanism identified |
| Direction | Investigations with reviewed outcomes | Useful narrowing toward the correct subsystem or mechanism |
| Groundedness | Investigations with reviewable evidence | Degree to which claims remain supported by cited evidence |
Table 5. Evaluation measures, included records, and definitions.
| Dimension | Positive test | Does not receive credit |
|---|---|---|
| Exact RCA | Supported conclusion matches the reviewed causal mechanism. | Restating symptoms, impact, or only the affected component. |
| Direction | Findings narrow the search toward the correct subsystem, failure family, or mechanism. | Broad guesses or elimination unsupported by current evidence. |
| Groundedness | Claims and hypothesis transitions follow cited evidence with valid source, scope, time, and outcome. | Unsupported detail or transitions, contradiction, failed-query misuse, unreliable elimination, repeated work, or inconsistent time ranges. |
Table 6. DivinerEval dimensions and principal scoring tests.
| Root-cause category | RCA | Dir. | Ground. |
|---|---|---|---|
| Stronger observed performance | |||
| Infrastructure, hardware, and network | 55% | 65% | 80% |
| Traffic, scaling, and load management | 42% | 65% | 74% |
| Moderate performance | |||
| Latency and performance regression | 41% | 55% | 61% |
| Capacity and quota deficit | 40% | 55% | 69% |
Table 7. Category-level operational performance across all available Diviner evaluations. Scores are incident-weighted percentages. Green, amber, and red cells denote scores of at least 60%, 40–59%, and below 40%, respectively.
Figure 6. Assisted and unassisted TDM in a fixed operational cohort. Calendar dates are omitted. The top panel recomputes cumulative p50 TDM and normalizes both series to the final unassisted median; the bottom panel shows cumulative cohort share.
| Element | Design choice |
|---|---|
| Baseline | Claude vanilla with native capabilities and without the Diviner harness |
| Harness comparison | Diviner + Claude adds the Diviner harness while retaining the baseline model |
| Diviner harness | Governed skills and workflows, historical retrieval, Lead–worker orchestration, tool and evidence contracts, and checkpointing |
| Platform comparison | Diviner + Codex applies the same harness through a different agentic platform |
| Operational context | An additional operational configuration is reported for context |
| Pairing | Every configuration receives the same alert-time input for each incident |
Table 8. Controlled-study protocol.
Figure 7. Controlled results for the generic-agent baseline, the Diviner harness on two agentic platforms, and the operational configuration. Panel (a) reports mean Exact RCA, useful Direction, and Groundedness scores. Panel (b) reports wall time, with bars showing p50 and markers showing p90. Paired differences and confidence intervals are reported in the text.
| Observed gap | Target | Candidate change |
|---|---|---|
| Missing or incorrect diagnostic procedure | Skill | Add checks, decision rules, or interpretation guidance |
| Wrong ordering, dependency, or coverage | Workflow | Revise initial steps, branches, or skill bindings |
| Evidence unavailable, malformed, or repeatedly misused | Tool contract | Add a tool or repair inputs, outputs, and failure semantics |
| Stale scope, ownership, or eligibility | Registry | Correct metadata, filtering, enablement, or version selection |
| Missing or misleading historical precedent | IKG | Add, relink, rescope, or retire a failure signature |
| Systematic judge disagreement | Evaluation rubric | Refine scoring criteria without changing agent behavior |
Table 9. How DivinerEval findings route to adaptation targets.
| Gate | Check |
|---|---|
| Artifact validation | Schema parses; referenced artifacts resolve; owner, scope, and version are present; requested capabilities remain allowed. |
| Point-in-time replay | Baseline and candidate receive the same incident-time inputs, registry cutoff, and available tool snapshots. |
| Quality and regression | Target cases improve without material regressions in Exact RCA, Direction, Groundedness, latency, or tool behavior on relevant cases. |
| Leakage control | Runtime inputs exclude reviewed outcomes and later discussion; the candidate artifact is the only intended post-incident change. |
| Owner decision | The owner reviews the artifact diff, motivating evidence, replay comparison, and recorded regressions before approval. |
| Release control | An approved change receives a new version and retains monitoring, disablement, rollback, and proposal lineage. |
Table 10. Checks applied before an adaptation can enter a registry.
More in this topic
Root Cause Analysis
Root cause analysis identifies the initiating faults behind observed failures by tracing symptoms through causal and system dependencies.
See all 80 papers