Sentry: Learning to Recover from LLM Agent Failures at Test Time

Changxiu Ji; Amy Lu; Qizheng Zhang; Kunle Olukotun

cs.LGOct 2, 2026

Sentry: Learning to Recover from LLM Agent Failures at Test Time

Authors: Changxiu Ji, Amy Lu, Qizheng Zhang, Kunle Olukotun

Organizations: Department of Computer Science, Stanford University

Abstract

LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

Figures & tables

Figure 1: Two ways failure knowledge goes wrong (Qwen3.5-9B on WebShop). (a) Always exposed: recovery rules learned by ACE stay in the agent’s context; after finding a matching item, the agent keeps comparing alternatives and takes no action. (b) Never learned: AgentGuard’s fixed no-progress warning resolves a search loop, but no lesson is stored; in a later task, the same warning fires after valid option selections, and the agent leaves the page and loses them.

Figure 2: Sentry’s framework. Sentry routes invalid actions to hard repair and progress or reasoning–grounding failures to soft repair with retrieved playbook guidance. Hard repairs are checked immediately; soft repairs undergo delayed recovery verification. Only verified soft-repair recoveries are added to the external playbook for use in future interventions.

Method	WebShop Avg. reward $\uparrow$	AppWorld Test pass (%) $\uparrow$	SWE-bench Lite Accuracy $\uparrow$	Mind2Web Replay Completion $\uparrow$
Base agent	0.170 $\,\pm\,$ 0.016	25.59 $\,\pm\,$ 0.88	0.20 $\,\pm\,$ 0.01	0.420 $\,\pm\,$ 0.030
AgentGuard	0.258 $\,\pm\,$ 0.019	25.09 $\,\pm\,$ 0.91	0.22 $\,\pm\,$ 0.02	0.450 $\,\pm\,$ 0.019
AgentForesight	0.168 $\,\pm\,$ 0.011	24.17 $\,\pm\,$ 0.91	0.24 $\,\pm\,$ 0.01	0.430 $\,\pm\,$ 0.016
AgentFixer	0.171 $\,\pm\,$ 0.017	26.69 $\,\pm\,$ 0.80	0.22 $\,\pm\,$ 0.01	0.687 $\,\pm\,$ 0.027
Wink	0.264 $\,\pm\,$ 0.015	27.56 $\,\pm\,$ 0.98	0.18 $\,\pm\,$ 0.02	0.370 $\,\pm\,$ 0.020
Sentry	0.468 $\,\pm\,$ 0.017	38.53 $\,\pm\,$ 0.87	0.30 $\,\pm\,$ 0.01	0.736 $\,\pm\,$ 0.019

Table 1: Comparison with runtime-intervention baselines. All methods use Qwen3.5-9B and the online-from-scratch protocol (§ 4.3 ); we report mean

\pm

std over five runs. Bold : best per column; underline : second best. The last row gives Sentry’s relative gain over the strongest baseline in each column.

Method	WebShop Avg. reward $\uparrow$	Mind2Web Replay Completion $\uparrow$
Base agent	0.186 $\,\pm\,$ 0.014	0.421 $\,\pm\,$ 0.026
Reflexion	0.295 $\,\pm\,$ 0.025	0.531 $\,\pm\,$ 0.023
ACE	0.335 $\,\pm\,$ 0.019	0.562 $\,\pm\,$ 0.020
AWM	0.270 $\,\pm\,$ 0.021	0.470 $\,\pm\,$ 0.018
ReasoningBank	0.250 $\,\pm\,$ 0.015	0.483 $\,\pm\,$ 0.020
Sentry	0.495 $\,\pm\,$ 0.017	0.729 $\,\pm\,$ 0.019

Table 2: Comparison with context-evolution baselines. All methods use Qwen3.5-9B and the held-out protocol (§ 4.3 ): methods with memory initialize it on 50 tasks and keep updating it by their own rules while being evaluated on 50 disjoint tasks. Only held-out tasks are scored, so Base-agent scores differ from Table 1 . Mean

\pm

std over five runs; bold : best per column; underline : second best. The last row gives Sentry’s relative gain over ACE, the strongest context-evolution baseline.

Figure 3: Local recovery rates by failure category and benchmark. Recovery rates measure the fraction of detected failures judged resolved within each category and benchmark. Action-validity recovery requires a corrected action accepted by the environment; progress and reasoning–grounding recovery is assessed by Sentry’s verifier after (H=10) reasoning–action–observation cycles.

	WebShop		AppWorld
Variant	Avg. reward $\uparrow$		Test pass (%) $\uparrow$
Full Sentry	0.468		38.53
Detection and repair
w/o hard repair	0.297	$-$ 36.5%	31.20	$-$ 19.0%
w/o soft repair	0.272	$-$ 41.9%	26.85	$-$ 30.3%
w/o intervention-need check	0.136	$-$ 70.9%	23.70	$-$ 38.5%

Table 3: Component ablations. Each variant removes one component from Sentry, using the online-from-scratch protocol and the same five runs as Table 1 . Entries are mean primary metrics; red values give the relative drop from Full Sentry. Variant definitions and secondary metrics are in Appendix A.6.2 .

	Playbook			WebShop		Mind2Web Replay
Variant	Retrieval	Updates	Persistent	Avg. reward $\uparrow$		Completion $\uparrow$
Base agent				0.186 $\,\pm\,$ 0.014		0.421 $\,\pm\,$ 0.026
No external playbook	✗	✗	✗	0.251 $\,\pm\,$ 0.017		0.533 $\,\pm\,$ 0.019
Frozen playbook	✓	✗	✗	0.364 $\,\pm\,$ 0.019		0.651 $\,\pm\,$ 0.023
+ persistent exposure	✓	✗	✓	0.296 $\,\pm\,$ 0.017	$-$ 0.068	0.599 $\,\pm\,$ 0.018	$-$ 0.052
Full Sentry	✓	✓	✗	0.495 $\,\pm\,$ 0.017		0.729 $\,\pm\,$ 0.019

Table 4: Reuse and exposure of the recovery playbook on held-out tasks. Variants with a playbook start from the same one, initialized on 50 tasks; all methods are evaluated on the same 50 disjoint tasks (§ 4.3 ). Sentry variants keep the detector and both repair routes and differ only in the playbook settings marked. Persistent : the full playbook is also placed in the task agent’s background context. The two frozen variants differ only in this setting; red values give the change from Frozen playbook. Mean

\pm

std over five runs; bold : best per column.

Method	WebShop	AppWorld	SWE-bench Lite	Mind2Web Replay	Total	Rel.
(a) Tokens, input + output (millions) $\downarrow$
Base agent	$6.19$	$4.22$	$1.06$	$0.86$	$12.32$	1.00 $\times$
AgentGuard	$7.53$	$5.39$	$1.34$	$1.01$	$15.26$	1.24 $\times$
AgentForesight	$15.32$	$10.06$	$2.48$	$2.22$	$30.08$	2.44 $\times$
AgentFixer	$8.20$	$5.61$	$1.41$	$1.11$	$16.33$	1.33 $\times$
Wink	$6.24$	$4.23$	$1.07$	$0.88$	$12.42$	1.01 $\times$

Table 5: Token and latency overhead. (a) Input plus output tokens, in millions; Rel. is each method’s total divided by the base agent’s. (b) Change in mean per-task wall-clock time relative to the base agent, in seconds; negative values mean faster than the base agent.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Detailed examples underlying Figure 1 . Trajectories from Qwen3.5-9B on WebShop. (a) ACE learns recovery rules from task 6594 and retains them in the agent’s context for task 4494. Despite identifying a matching item, the agent continues comparing alternatives and emits no action. (b) AgentGuard’s fixed warning helps the agent recover from repeated searches in task 3980, but no recovery lesson is stored. In task 4585, the same warning fires after valid option selections leave the page unchanged. The agent then navigates away and purchases the item without those options.

Method	Avg. reward	Full succ.	Bought	# Interv.	# Guided tasks	Guid. lift
Base agent	$0.170\pm 0.016$	6	23	0	0	–
AgentGuard	$0.258\pm 0.019$	7	37	533	65	0.134
AgentForesight	$0.168\pm 0.011$	4	25	15	14	-0.018
AgentFixer	$0.171\pm 0.017$	5	24	236	46	0.002
Wink	$0.264\pm 0.015$	10	33	137	60	0.156
Sentry	$\mathbf{0.468\pm 0.017}$	14	79	227	93	0.320

Appendix

Table 6: Full WebShop results using Qwen3.5-9B.

Method	Test pass (%)	# Interv.	# Guided tasks	Guid. lift
Base agent	$25.59\pm 0.88$	0	0	–
AgentGuard	$25.09\pm 0.91$	630	54	-0.93
AgentForesight	$24.17\pm 0.91$	25	12	-11.83
AgentFixer	$26.69\pm 0.80$	14	8	13.75
Wink	$27.56\pm 0.98$	24	14	14.07
Sentry	$\mathbf{38.53\pm 0.87}$	187	66	19.61

Appendix

Table 7: Full AppWorld results using Qwen3.5-9B. Guidance lift is in percentage points of test pass rate.

Method	Acc.	# Interv.	# Guided tasks	# Recovered
Base agent	$0.20\pm 0.01$	0	0	–
AgentGuard	$0.22\pm 0.02$	6	3	1
AgentForesight	$0.24\pm 0.01$	22	15	3
AgentFixer	$0.22\pm 0.01$	75	38	4
Wink	$0.18\pm 0.02$	4	3	3
Sentry	$\mathbf{0.30\pm 0.01}$	90	9	5

Appendix

Table 8: Full SWE-bench Lite results using Qwen3.5-9B.

Method	Mean replay completion	Fully completed tasks	# Interv.	# Guided tasks	Guid. lift
Base agent	$0.420\pm 0.030$	32	0	0	–
AgentGuard	$0.450\pm 0.019$	37	448	94	0.032
AgentForesight	$0.430\pm 0.016$	34	470	93	0.011
AgentFixer	$0.687\pm 0.027$	50	517	70	0.382
Wink	$0.370\pm 0.020$	32	7	6	-0.840
Sentry	$\mathbf{0.736\pm 0.019}$	55	435	77	0.410

Appendix

Table 9: Results on Mind2Web Replay using Qwen3.5-9B. Mean replay completion is the average fraction of reference steps completed in order. Fully completed tasks counts reference trajectories completed in their entirety. The protocol is specified in Appendix A.3.1 .

Method	WebShop reward	WebShop succ./bought	WebShop guid. lift	Mind2Web Replay completion	Mind2Web Replay completed tasks	Mind2Web Replay guid. lift
Base agent	0.192	7/28	–	0.550	43	–
AgentGuard	0.246	8/35	0.092	0.740	68	0.194
AgentForesight	0.091	4/12	-0.186	0.582	52	0.033
AgentFixer	0.273	5/44	0.112	0.753	62	0.328
Wink	0.197	7/33	0.008	0.580	47	0.111
Sentry	0.531	12/92	0.357	0.815	75	0.294

Appendix

Table 10: Results using GPT-OSS-120B on WebShop and Mind2Web Replay. WebShop reports average reward and full-success/purchase counts; Mind2Web Replay reports mean normalized trajectory completion and the number of fully completed tasks.

	No intervention	Sentry
Local recovery (of 200)	67 (33.5%)	121 (60.5%)
Recovered only in this branch	12	66
Higher final task score	25	74

Appendix

Table 11: Paired counterfactual evaluation at 200 soft-repair checkpoints. Local recovery is judged after

H=10

steps. The final row counts pairs in which each branch obtains a higher final score under the task-level metric defined in Section 4.1 ; the remaining 101 pairs are tied.

Variant	Avg. reward	Full succ.	Bought	# Interv.	# Guided tasks	Guid. lift
Full Sentry	$\mathbf{0.468}$	$\mathbf{14}$	$\mathbf{79}$	227	93	$\mathbf{0.320}$
w/o hard repair	$0.297$ $(\downarrow 0.171)$	5	51	59	53	0.240
w/o soft repair	$0.272$ $(\downarrow 0.196)$	9	63	159	75	0.136
w/o playbook failure-specific retrieval	$0.316$ $(\downarrow 0.152)$	6	45	102	93	0.156
w/o playbook self-evolution	$0.295$ $(\downarrow 0.173)$	4	38	198	89	0.140
w/o intervention-need check	$0.136$ $(\downarrow 0.332)$	1	19	347	99	$-0.035$

Appendix

Table 12: Component-ablation results on WebShop. Each variant removes one component from Sentry. Downward arrows indicate drops relative to Full Sentry for the primary task-level metric. Guidance lift is computed over guided tasks, following the definition in Appendix A.3.1 .

Variant	Test pass (%)	# Interv.	# Guided tasks	Guid. lift
Full Sentry	$\mathbf{38.53}$	187	66	$\mathbf{19.61}$
w/o hard repair	$31.20$ $(\downarrow 7.33)$	67	44	12.75
w/o soft repair	$26.85$ $(\downarrow 11.68)$	120	54	2.33
w/o playbook failure-specific retrieval	$31.05$ $(\downarrow 7.48)$	118	66	8.27
w/o playbook self-evolution	$28.90$ $(\downarrow 9.63)$	166	62	5.34
w/o intervention-need check	$23.70$ $(\downarrow 14.83)$	310	92	$-2.05$

Appendix

Table 13: Component-ablation results on AppWorld. Each variant removes one component from Sentry; arrows give the drop in test pass rate relative to Full Sentry. Guidance lift is in percentage points and follows the definition in Appendix A.3.1 .

Retrieval variant	WebShop avg. reward	Mind2Web Replay completion
Random retrieval	$0.33\pm 0.03$	$0.55\pm 0.02$
Unfiltered retrieval	$0.39\pm 0.01$	$0.61\pm 0.02$
Failure-type-only retrieval	$0.43\pm 0.01$	$0.66\pm 0.01$
Label-filtered, $k=3$	$0.47\pm 0.01$	$0.70\pm 0.01$
Label-filtered, $k=5$	$\mathbf{0.50\pm 0.02}$	$\mathbf{0.73\pm 0.02}$
Label-filtered, $k=10$	$0.46\pm 0.02$	$0.69\pm 0.02$

Appendix

Table 14: Playbook retrieval ablations on held-out WebShop and Mind2Web Replay tasks. Only the soft-repair retrieval rule is varied; all other Sentry components are fixed. We report mean

\pm

standard deviation over five repeated runs. Label-filtered retrieval with

k=5

is the Full Sentry configuration and uses the same runs as Table 2 (

0.495

and

0.729

at three decimals); relative decreases in the text are computed against those means.

Context window	Avg. reward	Full succ.	Bought	# Interv.	# Guided tasks	Guid. lift
1	0.352	8	60	165	78	0.233
3	0.431	12	73	205	88	0.296
5	0.468	14	79	227	93	0.320
7	0.456	13	78	238	94	0.304
10	0.421	12	72	255	95	0.264
15	0.384	10	66	275	97	0.220

Appendix

Table 15: Effect of Sentry context-window size on WebShop performance.

$H$	Avg. reward $\uparrow$	Guidance lift $\uparrow$	Recovery-judge acc. (%) $\uparrow$
3	0.432	0.283	76.0
5	0.458	0.307	82.0
10	0.468	0.320	84.0
20	0.461	0.312	83.0

Appendix

Table 16: Sensitivity to the soft-repair verification horizon

H

on WebShop. Performance and recovery-judge accuracy are stable across moderate verification horizons, while verifying too early reduces both end-to-end task performance and agreement with human recovery judgments.

Label	Criterion
Intervention needed	The agent shows an invalid action, repeated no-progress behavior, poor grounding, or another failure that requires intervention.
Recovered	After Sentry’s intervention, the agent stops the triggering failure pattern and resumes valid, task-relevant progress.
Helpful	The intervention text addresses the observed local failure and gives task-relevant guidance without unsupported assumptions or objective drift.

Appendix

Table 17: Human annotation rubric for auditing Sentry trajectory windows.

Benchmark	$N$	Intervention acc.	Recovery-judge acc.	Helpful rate
WebShop	50	86.0	84.0	86.2
AppWorld	50	86.0	81.0	85.2
SWE-bench Lite	50	82.0	76.9	82.1
Mind2Web Replay	50	90.0	85.2	89.7
Overall	200	86.0	81.8	85.8

Appendix

Table 18: Human validation of Sentry’s intervention decisions, recovery judgments, and intervention helpfulness.

N

is the number of audited windows per benchmark. Intervention accuracy is computed over all windows, recovery-judge accuracy over soft-repair windows (99 in total), and helpful rate over issued interventions (113 in total). All values except

N

are percentages.

Label	$N$	Percent agreement	Cohen’s $\kappa$
Intervention needed	200	94.5	0.89
Recovered	99	90.9	0.82
Helpful	113	95.6	0.82

Appendix

Table 19: Inter-annotator agreement before consensus discussion for the human audit.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 832 papers

Real-Time Detection and Repair of LLM Agent Failures
Aug 3, 2026Sunny DubeyAI Agent Reliability AI Agent Monitoring
Structural Baselines Matter for Agent Failure Detection and Attribution
Oct 12, 2026Ramesh Arvind Naagarajan, Stefan StreifAgent Failure Analysis Failure Attribution for LLM Agents
Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany
SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
Aug 10, 2026You Lu, Xinyu Huang, Bihuan Chen +1LLM Agent Skill Learning LLM Agents
College of Computer Science and Artificial Intelligence, China

Aug 3, 2026J/K move · Enter open · S save

Real-Time Detection and Repair of LLM Agent Failures

Sunny Dubey

AI Agent Reliability AI Agent Monitoring LLM Agent Reliability Runtime Verification

Open paper AskPDF arXiv

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

Abstract

LLM Agent Reliability

Explore similar work

Real-Time Detection and Repair of LLM Agent Failures

Structural Baselines Matter for Agent Failure Detection and Attribution

SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance