Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Harsh Raj; David Lee; Anas Mahmoud; Renxiong Wang; Razvan-Gabriel Dumitru; Chenguang Wang; Tong Zhao; Yunzhong He

cs.AISep 11, 2026

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Authors: Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang, Razvan-Gabriel Dumitru, Chenguang Wang, Tong Zhao, Yunzhong He, +2 more

Organizations: Scale AI · Scale AI Research

Abstract

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from $0.471$ to $0.608$ . More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.

Figures & tables

Figure 1 : Comparison of root-cause attribution strategies. Standard baselines like one-shot judging, self-consistency, and a heterogeneous judge panel perform a single pass over the evidence. Passive Continuation prompts the judge to repeatedly reconsider its diagnosis within an ongoing session but does not mandate further exploration. Conversely, Continual Search asks the judge to challenge its standing answer at each successive turn and search beyond the evidence that supported it.

Figure 2 : Continual Search and Passive Continuation start from the same turn 1 (T1) attribution. Passive Continuation only asks the judge to reconsider and reverify its answer, whereas Continual Search prompts it to challenge its current diagnosis by searching for evidence it has not yet examined.

Dataset	Description	$n$	Median tokens	Median size
MegaRCA-Mix	First consequential error in Harbor Index evaluation records ( Shi et al., 2026 ) , labeled with the interaction-centric taxonomy ( Raj et al., 2026 ) . The mixture spans GAIA ( Mialon et al., 2024 ) , SWE-Bench ( Jimenez et al., 2024 ; Deng et al., 2025 ) , HLE ( Phan et al., 2025 ) , OpenRCA ( Xu et al., 2025 ) , and 15 additional benchmarks.	50	286K	1.05 MiB
TRAIL ( Deshpande et al., 2025 )	Error location and category in GAIA ( Mialon et al., 2024 ) and SWE-Bench ( Jimenez et al., 2024 ) trajectories	148	100K	430 KiB
TELBench ( Wang et al., 2026a )	Earliest harmful error span in deep-research trajectories from GAIA ( Mialon et al., 2024 ) , BrowseComp ( Wei et al., 2025 ) , and xbench ( Chen et al., 2025 )	33	39K	144 KiB
AgentRx ( Barke et al., 2026 )	Critical failure step in retail $\tau$ -bench ( Yao et al., 2024 ) trajectories	29	7.2K	23.9 KiB
Who&When ( Zhang et al., 2025 )	Responsible agent and decisive error step in multi-agent trajectories over GAIA ( Mialon et al., 2024 ) and AssistantBench ( Yoran et al., 2024 )	126	2.4K	8.9 KiB

Table 1 : Overview of the root-cause attribution benchmarks. The table details each benchmark’s attribution target, number of annotated failure trials (

n

), and the median token count and raw file size of its execution evidence. Native evaluation metrics are defined in Appendix B.3 .

	GPT-5.5			Opus-4.8
Dataset	Single-turn	Passive	Continual Search	Single-turn	Passive	Continual Search
MegaRCA-Mix	0.356	0.569	0.569	0.471	0.479	0.608
TRAIL (Joint Acc.)	0.143	0.170	0.207	0.133	0.135	0.152
TRAIL (Weighted F1)	0.429	0.459	0.494	0.482	0.488	0.543
TELBench	0.182	0.212	0.242	0.091	0.091	0.152

Table 2: Attribution performance on MegaRCA-Mix, TRAIL, and TELBench after four continuation turns. Each benchmark is evaluated using its native metric, with TRAIL reporting both Joint Accuracy and Weighted F1. The highest score for each judge is bolded.

Figure 3 : Opus-4.8 attribution performance against cumulative observation tokens, reasoning tokens, and cost across four continuation turns. Cumulative observation tokens represent the total token ingested across all read operations executed by the agentic judge until that turn. MegaRCA-Mix is scored by F1, TRAIL by Weighted F1, and TELBench by Span Accuracy.

Figure 4 : Impact of model tier on attribution performance. Weighted F1 across turns 1–4 of Continual Search on TRAIL plotted against cumulative per-trajectory inference cost for GLM-4.7, Sonnet-5, Opus-4.8, and Fable-5 at default reasoning settings. Cost shows the mean price, and the boxed value marks the best Turn 4 score.

Figure 5 : Impact of reasoning effort on attribution performance. Weighted F1 across turns 1–4 of Continual Search on TRAIL plotted against cumulative per-trajectory inference cost, ablating low, high, and max reasoning settings for Opus-4.8 (left) and Sonnet-5 (right). Cost shows the mean price, and the box in each panel marks that model’s best Turn 4 score.

	GPT-5.5			Opus-4.8
Dataset	Single-turn	Passive	Continual Search	Single-turn	Passive	Continual Search
AgentRx	0.310	0.345	0.241	0.414	0.379	0.379
Who&When	0.516	0.492	0.437	0.373	0.349	0.349

Table 3: Attribution performance on AgentRx and Who&When after four continuation turns. In contrast to MegaRCA-Mix, TRAIL, and TELBench, Continual Search does not consistently improve over the single-turn attribution.

	Weighted F1		Cost $
Method	Opus-4.8	GPT-5.5	Opus-4.8	GPT-5.5
Single-turn	0.482	0.429	1.59	2.22
Self-consistency	0.430	0.383	6.52	8.65
Judge panel	0.431	0.431	4.73	4.73
Continual Search	0.543	0.494	3.43	5.83

Table 4: Comparison of Continual Search against standard judge baselines on TRAIL across pooled GAIA and SWE splits, using Opus-4.8 and GPT-5.5 as underlying judge models. Self-consistency uses four samples per judge, while the judge panel combines GPT-5.5, Opus-4.7, Opus-4.8, and Grok-4.6. Cost reflects the mean price per trajectory at published API rates.

Figure 6 : Attribution performance of Opus-4.8 across evaluation log size terciles within each benchmark. Solid lines denote Continual Search, while dotted lines represent the corresponding single-turn baseline. Each data point is plotted at the median execution-log size of its tercile and evaluated using the native benchmark metric.

Figure 7 : Opus-4.8 attribution performance versus cumulative deduplicated observation tokens across continuation turns. Deduplicating repeated 13-gram spans ensures that horizontal movement strictly reflects the acquisition of novel evidence from the execution-log.

Figure 8 : Benchmark audit performance on MegaRCA-Mix using Opus-4.8. Comparison of the taxonomy classifier under Continual Search against the specialized ABA baseline across Turns 1 through 4. Left:

F_{1}

score plotted against mean trajectory cost, showing that Continual Search surpasses ABA by turn 2. Right: Detailed metric breakdown, demonstrating that performance gains are driven by a sharp rise in Recall across turns. The highest score per column is bolded.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9 : Attribution performance versus evidence coverage. Mean

F_{1}

score plotted against mean artifact coverage for Opus-4.8 on MegaRCA-Mix. Continual Search progressively inspects novel artifacts across successive turns to improve diagnostic performance, while Passive Continuation remains restricted to its initial evidence subset.

Panel of Figure 10 — Figure 10 : Qualitative example from GAIA2 demonstrating how Continual Search uses non-trajectory artifacts to resolve misattributions. At turn 2, the judge relies solely on the agent trajectory, concludes that the agent terminated early, and assigns the failure to model Satisficing . At turn 3, the judge inspects the previously unread config.json file and console log. These records reveal that the environment failed to respond during the full wait window, correcting the initial assumption to Stale State Delivery . Excerpts are quoted verbatim.

	Joint Accuracy			Weighted F1
Run	Single-turn	Passive	Continual Search	Single-turn	Passive	Continual Search
1	0.133	0.135	0.152	0.482	0.488	0.543
2	0.152	0.154	0.197	0.465	0.484	0.505
3	0.129	0.135	0.189	0.439	0.452	0.516
4	0.138	0.145	0.182	0.475	0.484	0.531

Appendix

Table 5: Performance stability across four independent Opus-4.8 judge runs on TRAIL with

n=148

. Each run is independently sampled from the first turn onward, and the highest score per metric is bolded.

Figure 11 : The decision tree that assembles every Continual Search prompt for turns 2–4. Each prompt starts empty and gains one sentence at each step. The numbered badges mark the five parts of every prompt, namely the opening line (1), trust line (2), search action (3), change rule (4), and answer format (5). The full prompts are given below.

Explore similar work

Aug 5, 2026cs.AI

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.

Zhixiang Liang, Yifei Liu, Yidan Huang +5

University of Illinois Urbana-Champaign · 2Joy Future Academy, JD

Search Agents Model Auditing Root Cause Analysis

Ask PDF

Jun 25, 2026cs.AI

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

Aoyang Fang, Yifan Yang, Jin'ao Shang +7

The Chinese University of Hong Kong, Shenzhen · Lero · The Chinese University of Hong Kong

Root Cause Analysis Real-World Cloud Fault Injection Dataset

Ask PDF

Feb 2, 2026cs.AI

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an

\textit{automated diagnostic framework}

that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.

Shraddha Barke, Arnav Goyal, Alind Khare +3

1Microsoft Research · 2Microsoft · 3UIUC

Artificial Intelligence Agents Latent Failure Patterns Object Localization

Ask PDF