When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents

Chiming Duan; Tong Jia; Yingxi Chen; Minghua He; Lingzhe Zhang; Ying Li

cs.AIOct 27, 2026

When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents

Authors: Chiming Duan, Tong Jia, Yingxi Chen, Minghua He, Lingzhe Zhang, Ying Li

Organizations: Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China

Abstract

LLM agents interact with Web services to perform tasks on users' behalf. Errors during these interactions can disrupt execution and prevent task completion, motivating external intervention. However, agents may self-recover through continued execution, raising the question of when to intervene and when to wait for self-recovery. We present an empirical study of errors and self-recovery in LLM agents. We construct ReInBench, a dataset containing 1,259 agent traces and 1,403 error events across computer-use and Web tasks. Our study examines how often and how quickly agents self-recover, how self-recovery relates to subsequent execution and task completion, and whether self-recovery can be predicted when an error occurs. We find that 725 error events self-recover, often within a few steps, with self-recovery performance differing across Execution LLMs. Self-recovery can also help agents recover from subsequent errors of the same type and support overall task completion. Intervention should therefore prioritize errors unlikely to self-recover, while allowing agents to continue when self-recovery is likely. Building on these empirical findings, we propose ReCoF, a method that predicts whether an error will self-recover to inform whether additional intervention is needed. ReCoF achieves an average relative improvement of 9.02% in Macro-F1 over the strongest baseline on ReInBench. Our code and data can be found at https://method-notes.github.io/ReCoF/.

Figures & tables

Figure 1. Agent self-recovery and error detection. An LLM agent exchanges actions and feedback with an environment. An internal loop represents self-recovery. An external monitor receives the current trace and detects errors. A separate response can add guidance to the agent or wait without adding guidance.

Figure 2. A Web self-recovery example. A two-column montage of a Delta flight-query trace. The left column shows step 3 error onset and step 4 continued error, with LLM output, actions and screenshots. The right column shows the agent's recognition at step 5, omitted steps 6 and 7, and local self-recovery at step 8. Ctrl+A selects the address bar, SEA is entered there instead of the origin field, and the agent later reopens Delta to continue querying.

OSWorld-Verified 358 tasks
Execution LLM	Traces	Errors	Recovered
Claude 3.7 Sonnet	304	588	268
Claude Sonnet 4.5	339	343	226
Kimi K2.5	343	271	165
Online-Mind2Web 273 tasks
GPT-5 (2025-08-07)	273	201	66

Table 1. ReInBench overview.

Figure 3. ReCoF overview. Overview of ReCoF. Text, image, and joint image-text embeddings feed Recovery Pattern Experts that produce predictions p. An Assessment LLM provides a contextual verdict a. Conditional interaction computes r from p and a, and learned logistic fusion combines p, a, r, and Execution LLM identity to predict self-recovery or non-recovery. Historical execution contexts and recovery outcomes supply training supervision.

Figure 4. Self-recovery across Execution LLMs. Two side-by-side panels. Panel (a) shows shares of error events that recovered or did not recover within the observed trace. Recovery shares in OSWorld tasks are 45.6 percent for Claude 3.7 Sonnet, 65.9 percent for Claude Sonnet 4.5, and 60.9 percent for Kimi K2.5. The separate GPT-5/SeeAct row for Web tasks has 32.8 percent recovered. Panel (b) shows task-level overlap on shared OSWorld tasks with errors in all three LLMs. Each set contains tasks where that LLM recovers all annotated errors. Percentages use the union of these sets as 100 percent. Exclusive shares are 3.9, 27.5, and 21.6 percent. Pair-only shares are 7.8, 5.9, and 25.5 percent, and the three-way share is 7.8 percent. Circle areas are approximate.

Figure 5. Self-recovery timing. Three side-by-side percentage-frequency histograms with median markers. Panel (a) compares end-to-end onset-to-recovery duration for 600 events in OSWorld tasks and 66 in Web tasks, including recoveries without observable recognition. Panel (b) compares onset-to-recognition and recognition-to-recovery intervals within 509 events in OSWorld tasks. Panel (c) compares these two stages within 44 events in Web tasks. Each series is normalized separately, and the final bin contains intervals of 20 or more steps. Steps use each task group's recorded execution units in ReInBench.

Method / variant	Claude 3.7 Sonnet					Claude Sonnet 4.5					Kimi K2.5
	M-F1	F1	Prec.	Rec.	Acc.	M-F1	F1	Prec.	Rec.	Acc.	M-F1	F1	Prec.	Rec.	Acc.
Direct LLM	$68.75$	$62.50$	$69.77$	$56.60$	$70.00$	$72.88$	$63.41$	$72.22$	$56.52$	$76.19$	$61.69$	$47.06$	$50.00$	$44.44$	$67.27$
AgentForesight	$61.49$	$47.37$	${\color[rgb]{0.7109,0.1211,0.1406}\mathbf{78.26}}$	$33.96$	$66.67$	$68.89$	$55.56$	${\color[rgb]{0.7109,0.1211,0.1406}\mathbf{76.92}}$	$43.48$	$74.60$	$47.78$	$17.39$	$40.00$	$11.11$	$65.45$
Sentry	$61.10$	$48.78$	$68.97$	$37.74$	$65.00$	$72.88$	$63.41$	$72.22$	$56.52$	$76.19$	$50.82$	$24.00$	$42.86$	$16.67$	$65.45$
PrefixGuard	$58.59$ $\pm 3.19$	$59.40$ $\pm 1.77$	$52.96$ $\pm 3.40$	$68.30$ $\pm 6.86$	$58.83$ $\pm 3.15$	$47.50$ $\pm 6.74$	$22.34$ $\pm 8.39$	$41.19$ $\pm 17.81$	$15.65$ $\pm 5.83$	$59.68$ $\pm 7.24$	$55.43$ $\pm 5.72$	$43.85$ $\pm 8.22$	$42.90$ $\pm 8.71$	$50.00$ $\pm 17.12$	$59.64$ $\pm 8.47$
Wink	$66.26$	$59.79$	$65.91$	$54.72$	$67.50$	$66.09$	$52.63$	$66.67$	$43.48$	$71.43$	$41.56$	$18.18$	$20.00$	$16.67$	$50.91$

Table 2. Self-recovery prediction and core ablations on ReInBench.

Figure 6. Subsequent error recovery. Three side-by-side panels pooling ReInBench tasks and LLMs. Panel (a) shows later-event recovery proportions of 56.8, 76.0 and 47.5 percent among all 613, 200 same-type and 413 different-type pairs. Pairs may share a later event. Panels (b) and (c) use donut charts to compare later versus earlier recovery duration for same-type and different-type pairs, respectively. Among 126 same-type timing pairs, 79.4 percent are faster or equal and 20.6 percent slower. Among 181 different-type timing pairs, 52.5 percent are faster or equal and 47.5 percent slower. Each ring represents all valid timing pairs of its type.

Figure 7. Task completion and self-recovery. Two panels pooling ReInBench tasks and LLMs. Panel (a) shows task completion rates of 20.6, 18.0 and 60.0 percent for 286 traces with no recovery, 161 with partial recovery and 280 with complete recovery. Panel (b) shows the mean within-trace recovered and unrecovered error shares. The 256 successful traces average 71.9 percent recovered and 28.1 percent unrecovered. The 471 unsuccessful traces average 37.8 percent recovered and 62.2 percent unrecovered. These are averages of per-trace proportions, rather than pooled event counts.

Figure 8. Parameter sensitivity. Three side-by-side single-column panels show test Macro-F1 for Claude 3.7 Sonnet, Claude Sonnet 4.5, and Kimi K2.5. Panel (a) varies the inverse L2 regularization strength C, panel (b) the task-fold count K, and panel (c) an absolute offset from the validation-calibrated threshold. Curves are five-seed means and shaded bands are one sample standard deviation. Dashed vertical lines mark C=0.01, K=3, and zero threshold offset. Other parameters, Recovery Pattern Experts, Assessment LLM outputs and original task partitions remain fixed.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 705 papers

Sentry: Learning to Recover from LLM Agent Failures at Test Time
Oct 2, 2026Changxiu Ji, Amy Lu, Qizheng Zhang +1LLM Agent Reliability Memory-Augmented Language Model Agents
Department of Computer Science, Stanford University
AgentRewind: Recoverable Execution for Long-Horizon LLM Agents
Aug 14, 2026Yu Zhuang, Kefei Chen, Yitong Duan +3Long-Horizon LLM Agents Long-Horizon Agent Tasks
University of Chinese Academy of Sciences · Zhongguancun Academy, Beijing, China · Institute of Automation, Chinese Academy of Sciences +1
When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents
Sep 30, 2026Shuyao Xiao, Shengling Wang, Xuan Chen +7Long-Horizon Agent Evaluation Causal Interventions in Language Models
School of Artificial Intelligence, Beijing Normal University · Ke Holdings

Oct 2, 2026J/K move · Enter open · S save

Sentry: Learning to Recover from LLM Agent Failures at Test Time

Changxiu Ji, Amy Lu, Qizheng Zhang, Kunle Olukotun

Department of Computer Science, Stanford University

LLM Agent Reliability Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

Abstract

LLM Agents

Explore similar work

Sentry: Learning to Recover from LLM Agent Failures at Test Time

AgentRewind: Recoverable Execution for Long-Horizon LLM Agents

When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents