cs.AISep 11, 2026
SaveRoot-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
Organizations: Scale AI · Scale AI Research
Abstract
The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from to . More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.
Figures & tables
Figure 1 : Comparison of root-cause attribution strategies. Standard baselines like one-shot judging, self-consistency, and a heterogeneous judge panel perform a single pass over the evidence. Passive Continuation prompts the judge to repeatedly reconsider its diagnosis within an ongoing session but does not mandate further exploration. Conversely, Continual Search asks the judge to challenge its standing answer at each successive turn and search beyond the evidence that supported it.
Figure 2 : Continual Search and Passive Continuation start from the same turn 1 (T1) attribution. Passive Continuation only asks the judge to reconsider and reverify its answer, whereas Continual Search prompts it to challenge its current diagnosis by searching for evidence it has not yet examined.
| Dataset | Description | Median tokens | Median size | |
| MegaRCA-Mix | First consequential error in Harbor Index evaluation records ( Shi et al., 2026 ) , labeled with the interaction-centric taxonomy ( Raj et al., 2026 ) . The mixture spans GAIA ( Mialon et al., 2024 ) , SWE-Bench ( Jimenez et al., 2024 ; Deng et al., 2025 ) , HLE ( Phan et al., 2025 ) , OpenRCA ( Xu et al., 2025 ) , and 15 additional benchmarks. | 50 | 286K | 1.05 MiB |
| TRAIL ( Deshpande et al., 2025 ) | Error location and category in GAIA ( Mialon et al., 2024 ) and SWE-Bench ( Jimenez et al., 2024 ) trajectories | 148 | 100K | 430 KiB |
| TELBench ( Wang et al., 2026a ) | Earliest harmful error span in deep-research trajectories from GAIA ( Mialon et al., 2024 ) , BrowseComp ( Wei et al., 2025 ) , and xbench ( Chen et al., 2025 ) | 33 | 39K | 144 KiB |
| AgentRx ( Barke et al., 2026 ) | Critical failure step in retail -bench ( Yao et al., 2024 ) trajectories | 29 | 7.2K | 23.9 KiB |
| Who&When ( Zhang et al., 2025 ) | Responsible agent and decisive error step in multi-agent trajectories over GAIA ( Mialon et al., 2024 ) and AssistantBench ( Yoran et al., 2024 ) | 126 | 2.4K | 8.9 KiB |
Table 1 : Overview of the root-cause attribution benchmarks. The table details each benchmark’s attribution target, number of annotated failure trials ( ), and the median token count and raw file size of its execution evidence. Native evaluation metrics are defined in Appendix B.3 .
| GPT-5.5 | Opus-4.8 | |||||
| Dataset | Single-turn | Passive | Continual Search | Single-turn | Passive | Continual Search |
| MegaRCA-Mix | 0.356 | 0.569 | 0.569 | 0.471 | 0.479 | 0.608 |
| TRAIL (Joint Acc.) | 0.143 | 0.170 | 0.207 | 0.133 | 0.135 | 0.152 |
| TRAIL (Weighted F1) | 0.429 | 0.459 | 0.494 | 0.482 | 0.488 | 0.543 |
| TELBench | 0.182 | 0.212 | 0.242 | 0.091 | 0.091 | 0.152 |
Table 2: Attribution performance on MegaRCA-Mix, TRAIL, and TELBench after four continuation turns. Each benchmark is evaluated using its native metric, with TRAIL reporting both Joint Accuracy and Weighted F1. The highest score for each judge is bolded.
Figure 3 : Opus-4.8 attribution performance against cumulative observation tokens, reasoning tokens, and cost across four continuation turns. Cumulative observation tokens represent the total token ingested across all read operations executed by the agentic judge until that turn. MegaRCA-Mix is scored by F1, TRAIL by Weighted F1, and TELBench by Span Accuracy.
Figure 4 : Impact of model tier on attribution performance. Weighted F1 across turns 1–4 of Continual Search on TRAIL plotted against cumulative per-trajectory inference cost for GLM-4.7, Sonnet-5, Opus-4.8, and Fable-5 at default reasoning settings. Cost shows the mean price, and the boxed value marks the best Turn 4 score.
Figure 5 : Impact of reasoning effort on attribution performance. Weighted F1 across turns 1–4 of Continual Search on TRAIL plotted against cumulative per-trajectory inference cost, ablating low, high, and max reasoning settings for Opus-4.8 (left) and Sonnet-5 (right). Cost shows the mean price, and the box in each panel marks that model’s best Turn 4 score.
| GPT-5.5 | Opus-4.8 | |||||
| Dataset | Single-turn | Passive | Continual Search | Single-turn | Passive | Continual Search |
| AgentRx | 0.310 | 0.345 | 0.241 | 0.414 | 0.379 | 0.379 |
| Who&When | 0.516 | 0.492 | 0.437 | 0.373 | 0.349 | 0.349 |
Table 3: Attribution performance on AgentRx and Who&When after four continuation turns. In contrast to MegaRCA-Mix, TRAIL, and TELBench, Continual Search does not consistently improve over the single-turn attribution.
| Weighted F1 | Cost $ | |||
| Method | Opus-4.8 | GPT-5.5 | Opus-4.8 | GPT-5.5 |
| Single-turn | 0.482 | 0.429 | 1.59 | 2.22 |
| Self-consistency | 0.430 | 0.383 | 6.52 | 8.65 |
| Judge panel | 0.431 | 0.431 | 4.73 | 4.73 |
| Continual Search | 0.543 | 0.494 | 3.43 | 5.83 |
Table 4: Comparison of Continual Search against standard judge baselines on TRAIL across pooled GAIA and SWE splits, using Opus-4.8 and GPT-5.5 as underlying judge models. Self-consistency uses four samples per judge, while the judge panel combines GPT-5.5, Opus-4.7, Opus-4.8, and Grok-4.6. Cost reflects the mean price per trajectory at published API rates.
Figure 6 : Attribution performance of Opus-4.8 across evaluation log size terciles within each benchmark. Solid lines denote Continual Search, while dotted lines represent the corresponding single-turn baseline. Each data point is plotted at the median execution-log size of its tercile and evaluated using the native benchmark metric.
Figure 7 : Opus-4.8 attribution performance versus cumulative deduplicated observation tokens across continuation turns. Deduplicating repeated 13-gram spans ensures that horizontal movement strictly reflects the acquisition of novel evidence from the execution-log.
Figure 8 : Benchmark audit performance on MegaRCA-Mix using Opus-4.8. Comparison of the taxonomy classifier under Continual Search against the specialized ABA baseline across Turns 1 through 4. Left: score plotted against mean trajectory cost, showing that Continual Search surpasses ABA by turn 2. Right: Detailed metric breakdown, demonstrating that performance gains are driven by a sharp rise in Recall across turns. The highest score per column is bolded.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9 : Attribution performance versus evidence coverage. Mean score plotted against mean artifact coverage for Opus-4.8 on MegaRCA-Mix. Continual Search progressively inspects novel artifacts across successive turns to improve diagnostic performance, while Passive Continuation remains restricted to its initial evidence subset.
Figure 10 : Qualitative example from GAIA2 demonstrating how Continual Search uses non-trajectory artifacts to resolve misattributions. At turn 2, the judge relies solely on the agent trajectory, concludes that the agent terminated early, and assigns the failure to model Satisficing . At turn 3, the judge inspects the previously unread config.json file and console log. These records reveal that the environment failed to respond during the full wait window, correcting the initial assumption to Stale State Delivery . Excerpts are quoted verbatim.
| Joint Accuracy | Weighted F1 | |||||
| Run | Single-turn | Passive | Continual Search | Single-turn | Passive | Continual Search |
| 1 | 0.133 | 0.135 | 0.152 | 0.482 | 0.488 | 0.543 |
| 2 | 0.152 | 0.154 | 0.197 | 0.465 | 0.484 | 0.505 |
| 3 | 0.129 | 0.135 | 0.189 | 0.439 | 0.452 | 0.516 |
| 4 | 0.138 | 0.145 | 0.182 | 0.475 | 0.484 | 0.531 |
Table 5: Performance stability across four independent Opus-4.8 judge runs on TRAIL with . Each run is independently sampled from the first turn onward, and the highest score per metric is bolded.
Figure 11 : The decision tree that assembles every Continual Search prompt for turns 2–4. Each prompt starts empty and gains one sentence at each step. The numbered badges mark the five parts of every prompt, namely the opening line (1), trust line (2), search action (3), change rule (4), and answer format (5). The full prompts are given below.