Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

Siddharth Jain; Venkat N. Vedam

cs.AIOct 27, 2026

Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

Authors: Siddharth Jain, Venkat N. Vedam

Organizations: Intuit

Abstract

Classification accuracy and true-positive and false-positive rates alone do not determine whether restarting an AI agent improves success. Using an exact alarm-to-action identity for an independent clean restart with zero monitoring overhead, we show analytically that a monitor with fixed 90% accuracy can swing from a five-percentage-point loss to a 35-point gain in success rate, depending solely on the replacement attempt's residual-budget success probability. We introduce two techniques for evaluating monitors and progress signals in decision terms. First, a decision-sufficiency criterion, a binary-action specialization of Blackwell's comparison of experiments, identifies which progress representations or context compressions preserve the information needed to choose optimally between the specified actions, independent of trace reconstruction fidelity. Second, a finite-sample, model-free decision audit uses simultaneous binomial intervals over existing agent traces, without additional model calls, to certify a preference for "restart" or "continue" under stated sampling assumptions, or return "abstain" when evidence is insufficient. In a study of 194 externally evaluated model executions on 19 selected SymPy issues from SWE-bench Verified, with a retrospective audit of 130 of these traces, we find a 2.29-point apparent restart gain traced to a single surviving failed attempt, a 42.29-point loss on a second model, no positive replication of the initial gain on prospective task cohorts, and zero of 103 live comparisons meeting the specified certification bar. These results motivate evaluating agent monitors through recovery opportunity, decision value, and statistical support, rather than classification performance alone.

Figures & tables

Line of work	Observed quantity	Primary question
Inference-time search	Candidate scores or votes	Which candidate should receive compute?
Dynamic restart theory	Run features and runtime beliefs	Which restart policy minimizes expected runtime?
Learned termination and recovery	Prefix alarms and retained edits	Does a trained controller save cost or improve resolution?
Semantic stopping	Change between successive outputs	Has an iterative answer stabilized?
Trajectory evaluation	Task-specific subgoal state	How much of the task is complete?
Context management	Structured or compressed history	What information should the agent retain?

Table 1: The object measured by adjacent lines of work. A task-grounded state can support progress decisions; a trace-only controller must infer it.

Figure 1: Observational aliasing in black-box control. Distinct latent states can induce the same observable history while requiring opposite actions. A useful progress sensor must separate histories according to continuation advantage, not surface activity.

Figure 2: Analytical controls for monitor evaluation. Panel A keeps the failure detector’s true-positive and false-positive rates fixed while varying continuation success and restartability. The black curve is the zero-gain boundary. Panel B gives the number of zero-success observations required by the specified simultaneous interval procedure to bound a success probability. Neither panel reports newly collected agent performance.

Stage	Question	Attempts	Frozen comparison
Initial restart	Does a short cutoff beat one 30-call attempt?	70	Cutoffs and universal schedules
Prospective transfer	Does the selected cutoff transfer to unseen tasks?	60	Fixed 16 versus fixed 30
Horizon extension	Is completion mass hidden after call 30?	20	One 60 versus repeated 30
Context recovery	Does fresh context recover protocol collapse?	40	Baseline versus rejuvenation
Patch algebra	Do clean unions of failed patches solve?	0	19 frozen composites
Semantic resolver	Can one agent synthesize failed hypotheses?	4	Four bounded portfolio runs

Table 2: Experimental ladder. Model attempts count actual agent or resolver executions. Patch algebra used existing failed workspaces and no model calls.

Component	Frozen value
Benchmark	SWE-bench Verified, selected sympy/sympy tasks
Agent	mini-swe-agent 2.4.6
Models	openai/gpt-5.6-luna ; historical comparator deepseek/deepseek-v4-flash-0731
Sampling	Temperature 0.7; separate provider draws
Primary budget	Model calls, with stage horizons of 12, 30, or 60
Verification	External fail-to-pass evaluator after every environment action

Table 3: Runtime controls shared across model-driven experiments unless a stage states otherwise.

Comparison	Observed result	Interpretation
Fixed 16 versus 30, Luna training tasks	$+2.29$ points $[0.00,6.86]$	Weak in-sample gain
Luna-selected fixed 16 on DeepSeek	$-42.29$ points $[-69.14,-17.71]$	Selected scale does not transfer across models
Fixed 16 versus 30, all 19 Luna tasks	$-0.21$ points $[-4.21,2.53]$	Pooled estimate includes initial and prospective tasks
First completion after call 30	0/20; one-sided upper bound $13.9\%$	No observed useful late tail; rare mass remains possible
Fresh context versus baseline	0/20 versus 1/20; mean cost $+56.7\%$	Protocol label improves, task outcome does not
Failed-branch reuse	0/19 composites; 0/4 resolvers	Textual diversity does not supply verified closure

Table 4: Main empirical results. Intervals resample tasks and attempts unless noted.

Task	Pass	Calls: min/mean/max	Pass	Calls: min/mean/max
	Luna		DeepSeek
23413	4/5	5/7.0/9	4/5	21/23.8/27
23824	5/5	4/5.2/6	5/5	11/12.8/15
23950	0/5	none	1/5	29/29.0/29
24066	5/5	5/7.0/11	3/5	20/25.0/29
24213	5/5	4/4.6/5	5/5	13/19.8/28

Table 5: Initial externally verified outcomes. Call statistics are conditional on successful attempts.

Figure 3: Model-specific completion scale and restart-policy value on the initial seven tasks. Panel A reports the task-macro probability that one attempt has verified by call

c

. Panel B reports the exact plug-in success of repeated cutoff

c

under 30 total calls. Shading gives pointwise 95% hierarchical bootstrap intervals. Enlarged points mark the Luna-selected cutoff of 16 calls.

Cohort	Tasks / attempts	Fixed 16	Fixed 30	Difference and 95% interval
Initial	7 / 35	85.14%	82.86%	$+2.29$ [ $0.00,6.86$ ]
Prospective high ID	7 / 35	100.00%	100.00%	$0.00$ [ $0.00,0.00$ ]
Difficulty enriched	5 / 25	0.00%	4.00%	$-4.00$ [ $-16.00,0.00$ ]
Pooled	19 / 95	68.21%	68.42%	$-0.21$ [ $-4.21,2.53$ ]

Table 6: Transfer of the Luna-selected 16-call cutoff. The interval resamples tasks and attempts.

Cohort	Attempts	Live	Restart all	Perfect alarm
Initial Luna	35	6	$+2.29$	$+2.29$
Initial DeepSeek	35	27	$-42.29$	$0.00$
Prospective high ID	35	0	$0.00$	$0.00$
Prospective stress	25	25	$-4.00$	$0.00$

Table 7: Decision opportunities at call 16 with total budget 30. Changes are task-macro percentage points relative to continuation. The oracle knows future continuation failure and uses the specified independent clean restart. Its values are plug-in diagnostics, not statistically validated upper bounds. No live cell receives a sign certificate.

Figure 4: Conditional restart advantage in the 13 task-model cells with a live attempt at call 16. Dots are plug-in estimates. Lines are simultaneous difference bounds formed from binomial intervals across all 156 audited comparisons. Positive values favor restarting. Every interval includes zero despite substantial heterogeneity in point estimates.

Task	Pass	First pass	Format fail	Limit	Mean cost
16597	0/5	none	4	1	$0.05716
17630	0/5	none	5	0	$0.03577
21612	1/5	call 17	4	0	$0.03126
21930	0/5	none	4	1	$0.04996
Total	1/20	no pass after 30	17	2

Table 8: Fresh 60-call outcomes on tasks previously unsolved in five 30-call attempts.

Figure 5: Context rejuvenation changes the observed failure mode without improving task success. Panel A shows terminal outcomes, with counts and within-condition percentages. Panel B shows the corresponding increase in mean metered cost and wall time relative to baseline.

Outcome	Baseline	Rejuvenation	Treatment difference
Verified success	5.0%	0.0%	$-5.0$ points
Repeated-format terminal	90.0%	0.0%	$-90.0$ points
Limit terminal	5.0%	100.0%	$+95.0$ points
Mean cost	$0.03786	$0.05934	+$0.02148
Mean wall time	309.5 s	512.8 s	+203.3 s

Table 9: Context-rejuvenation outcomes under the shared 60-call budget.

Task	Sources	Unique	Pairs	Clean novel	Evaluated
16597	10	9	45	35	8
17630	10	10	45	0	0
21612	9	9	36	13	8
21930	9	9	36	3	3
Total	38	37	162	51	19

Table 10: Verifier-blind patch composition from failed terminal branches.

Figure 6: Five bounded tests of restart and recovery. Outcomes identify limitations of the tested interventions, not impossibility results for richer controllers. The task populations and uncertainty differ across stages.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$X_{t}$	Latent agent and environment state after call $t$
$O_{t}$ , $\mathcal{F}_{t}$	Observation at call $t$ and the observable history filtration
$T$	First call whose resulting state passes the external verifier
$E$ , $H$	Explicit protocol-terminal time and configured attempt horizon
$B$ , $b$	Total policy budget and residual budget
$F(t)$ , $\lambda_{t}$	Completion-time CDF and discrete completion hazard

Appendix

Table 11: Principal notation.

Cohort	Task IDs
Initial	23413, 23824, 23950, 24066, 24213, 24443, 24661
Prospective high ID	22456, 22714, 22914, 23262, 23534, 24539, 24562
Difficulty enriched	16597, 17630, 21612, 21930, 22080
Horizon 60	16597, 17630, 21612, 21930
Context A/B	16597, 17630, 21612, 21930
Patch and resolver	16597, 17630, 21612, 21930

Appendix

Table 12: Task IDs by cohort. IDs omit the common sympy__sympy- prefix.

Stage	Attempts	Metered cost	Wall hours
Initial Luna	35	$0.38327	0.771
Initial DeepSeek	35	$0.38023	3.644
Prospective Luna	60	$1.14379	2.596
Horizon 60	20	$0.87077	2.207
Context A/B	40	$1.94403	4.568
Semantic resolver	4	$0.05132	0.145

Appendix

Table 13: Collection cost and accumulated wall time. Wall time sums parallelizable attempts and is not elapsed study duration.

Artifact cohort	Scope	Primary use
Initial paired-model cohort	70 attempts, 7 tasks	Completion curves and restart replay
Prospective transfer cohorts	60 attempts, 12 tasks	Out-of-sample cutoff transfer
Horizon extension	20 attempts, 4 tasks	Late completion and protocol termination
Context intervention trial	40 attempts, 4 tasks	Terminal outcomes, cost, and latency
Patch-composition screen	19 verifier-only candidates	Failed-branch compositionality
Semantic resolver screen	4 attempts, 4 tasks	Semantic aggregation

Appendix

Table 14: Accompanying analysis artifacts and their role in the study.

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 281 papers

Oct 12, 2026

Structural Baselines Matter for Agent Failure Detection and Attribution

An LLM agent using tools can fail silently. It stops before the task is done, raises no error, and answers fluently. Two kinds of monitor address this, one deciding whether a run failed and the other naming the step where failure became unavoidable. Much of the reported performance on both tasks comes from regularities that need no model. Failing runs are longer, and the step labelled decisive is usually the last. A rule that always names the last step is correct

67\%

96\%

of the time here, without reading the run at all. That ceiling reflects how the labels were written, not how agents break. On failures whose decisive step replay can confirm, it falls to near chance. Reporting gain over that rule therefore changes what a score means. Under that correction, methods tested here gain nothing on give-up failures, while several published systems keep a clear advantage on other splits. For detection, the agent's internal states beat transcript length and text similarity on the agent benchmark, and hold up where the text baseline does not. Precision stays low when failures are rare, so such a monitor should rank runs for review, not replace it.

Ramesh Arvind Naagarajan, Stefan Streif

Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany

Agent Failure Analysis Failure Attribution for LLM Agents LLM Agent Evaluation LLM Agent Reliability

Ask PDF

Oct 13, 2026

Executable Prefix Values for Locating Agent Failures

Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.

Bingyu Shen, Jiaqi Tang, Lan Wei +3

Independent Researcher · Department of Computer Science and Technology, Kean University, Union, NJ, USA · Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, FL, USA

Agent Failure Analysis LLM Agent Evaluation Bias in LLM-as-a-Judge

Ask PDF

Sep 15, 2026

Locating Hidden Failures Makes Long-Horizon Agents More Reliable

As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed. An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. We study

2518

agent trajectories across software engineering, computer use, and science, close to real deployment, and classify

6967

mistakes into

78

failure types. Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. Yet Scout, a

4

B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.

Salman Rahman, Yubin Kim, Mihir Parmar +15

University of California, Los Angeles · New York University · Google Research +1

Agent Failure Analysis AI Agent Reliability Long-Horizon Agent Evaluation Fault Detection AI Agent Benchmarks

Ask PDF

Abstract

AI Agent Monitoring

Explore similar work

Structural Baselines Matter for Agent Failure Detection and Attribution

Executable Prefix Values for Locating Agent Failures

Locating Hidden Failures Makes Long-Horizon Agents More Reliable