StopLoss: Teaching Agents to Predict Failure and Stop Early

Jiarong Liang; Xueguang Ma; Dongfu Jiang; Yuxuan Zhang; Ping Nie; Wenhu Chen

cs.AIOct 18, 2026

StopLoss: Teaching Agents to Predict Failure and Stop Early

Authors: Jiarong Liang, Xueguang Ma, Dongfu Jiang, Yuxuan Zhang, Ping Nie, Wenhu Chen

Organizations: University of Waterloo

Abstract

Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5% on Verified while preserving 99% of successful runs, and by up to 31% on the costlier Pro subset, reaching 48% with a Qwen3-8B backbone, and in all four settings its cost per retained solve is below that of every judge policy. Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.

Figures & tables

Figure 1: StopLoss stops more failing runs than LLM judges and spends far less on monitoring. Results on SWE-bench Pro tasks executed by GPT-5.6 Luna. StopLoss is the Qwen2.5-Coder-7B-Instruct . (a) Failure capture versus cost per retained solve. StopLoss stops 35 of the 69 failing runs (50.7%), while no judge stops more than 20. (b) Cumulative monitoring cost under each frozen policy:

0.15 for StopLoss across all 213 runs, versus

2.03–$15.36 for the judges. Table 1 reports details.

Figure 2: StopLoss turns final verifier outcomes into early-stopping decisions. Offline, the outcome of each completed run labels its earlier windows, 7B–8B models are adapted with LoRA on these windows, and the stopping threshold

\tau

is calibrated on successful runs and then frozen. Online , at tool steps

5,8,11,\ldots

, each ensemble member reads the step count and the three most recent action–observation pairs in one forward pass; the run stops once the averaged failure score reaches

\tau

, and the executor itself is unchanged.

SWE-bench Verified / GPT-5.6 Luna 425 stopping runs; 306 successes; 119 failures
Method	Cost		Prediction	Stopping outcomes
Method	$\Delta$ CPS (%) $\downarrow$	Monitoring ( $)$ \downarrow$	Tier AUC $\uparrow$	Success retained (%) $\uparrow$	Failure capture (%) $\uparrow$
StopLoss	−4.5	0.44	0.622	99.0	5.0
Qwen3-8B	−1.8	0.43	0.605	97.7	5.0
Qwen2.5-Coder-7B-Instruct	−5.2	0.35	0.612	99.0	5.9
GPT-5.6 Sol	+116.8	33.20	0.633	91.8	20.2

Table 1: StopLoss achieves lower estimated cost per retained solve than LLM-judge policies across all four settings and all three backbones.

\Delta

CPS reports the percentage change in cost per retained solve relative to full execution without monitoring, including monitoring overhead; negative values indicate savings. Monitoring is the estimated cumulative cost of monitor calls during execution. Tier AUC measures how well the monitor assigns higher failure scores to runs that eventually fail than to those that succeed (0.5 = chance; 1.0 = perfect ranking). Success retained is the percentage of otherwise successful runs preserved by the policy; failure capture is the percentage of otherwise failed runs correctly stopped. Within each setting, bold marks the best value in each cost and stopping-outcome column. StopLoss denotes the primary Qwen2.5-7B-Instruct ensemble.

		GPT-5.6 Luna runs		Claude Sonnet 4.6 runs
Backbone	Configuration	SWE-bench Verified	SWE-bench Pro	SWE-bench Verified	SWE-bench Pro
	Untuned base	0.491	0.452	0.512	0.511
Qwen2.5-7B-Instruct	Trained ensemble	0.622 (+0.131)	0.614 (+0.162)	0.561 (+0.049)	0.579 (+0.068)
	Untuned base	0.504	0.507	0.481	0.516
Qwen3-8B	Trained ensemble	0.605 (+0.100)	0.621 (+0.114)	0.563 (+0.082)	0.608 (+0.092)
	Untuned base	0.437	0.433	0.455	0.478

Table 2: Trained ensembles achieve higher failure-prediction AUC than their untuned backbones in all four settings. Parentheses give absolute AUC gains over the corresponding untuned base. Step count only uses the current tool-step index as the failure score, without reading trajectory content.

Figure 3: StopLoss provides an early failure signal from the first scheduled check. At tool step 5, the primary StopLoss ensemble (blue) achieves higher ordinary AUC than its untuned base (gray) in all four settings. Higher AUC indicates better separation of eventual failures from successes; the dotted line marks chance-level ranking (AUC = 0.5). This post-hoc evaluation uses one observation per original run reaching each checkpoint, without applying the stopping policy. Shaded bands show pointwise 95% repository-bootstrap intervals. Numbers below each checkpoint report total runs / failed runs. Later checkpoints contain different subsets of runs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Model(s)	Reasoning setting	Limits and interface
Executor	GPT-5.6 Luna	Effort unset; upstream default not recorded	100 turns; 7,200 s agent timeout; $9 spend valve. Output-token caps stripped by the GPT gateway.
Executor	Claude Sonnet 4.6	Thinking enabled, 8,000-token budget	100 turns; 7,200 s agent timeout; $26 spend valve. Gateway default output limit 16,000 tokens.
Judge	GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna	High reasoning effort	Native Responses API; output-token caps stripped, so no enforced per-call output cap.
Judge	Claude Opus 5, Claude Opus 4.6, Claude Sonnet 4.6	Adaptive thinking, high effort	Native Messages API; output limits 4,000, 8,000, 16,000 on truncation retries.

Appendix

Table 3: Recorded generation and judging configurations. Model identifiers are listed in Table 5 . Collection spend valves use the collection-time accounting schedule; reported costs are recomputed under the frozen full-price schedule.

Benchmark / executor	Ranking runs	Solved	No checkpoint	Windows	Stopping runs
Verified / Luna	499	355	2	2307	425
Verified / Sonnet	499	377	9	2215	425
Pro / Luna	264	177	3	1569	213
Pro / Sonnet	261	132	2	1960	213

Appendix

Table 4: Detailed evaluation coverage. Solved runs, runs without checkpoints, and checkpoint windows refer to the full ranking cohort. Every checkpoint window is scored by all three same-family judges and all small monitors. The last column gives the calibration-disjoint stopping cohort.

Model identifier	Input	Output
gpt-5.6-sol	4.00	20.00
gpt-5.6-terra	2.00	12.00
gpt-5.6-luna	0.20	1.20
claude-opus-5	5.00	25.00
claude-opus-4-6	5.00	25.00
claude-sonnet-4-6	3.00	15.00

Appendix

Table 5: Frozen September 18 full-price schedule in dollars per million tokens. Every input token is charged at the input rate, including cache reads and writes; output includes reasoning/thinking tokens.

Cell	Judge	Windows	$/check	Judge / ensemble
Verified / Luna	GPT-5.6 Sol	2307	0.01949	82.3
Verified / Luna	GPT-5.6 Terra	2307	0.00774	32.7
Verified / Luna	GPT-5.6 Luna	2307	0.00248	10.5
Verified / Sonnet	Claude Opus 5	2215	0.07722	326.3
Verified / Sonnet	Claude Opus 4.6	2215	0.02015	85.1
Verified / Sonnet	Claude Sonnet 4.6	2215	0.02114	89.3

Appendix

Table 6: Mean per-prediction judge costs on the same full-cohort windows, using recorded attempts at full token prices. Ratios divide by the primary ensemble proxy of $0.000237 per check. This is independent of which checks a stopping policy visits.

Judge	Covered windows	New windows	Known new $	Unknown usage
GPT-5.6 Sol	3876	2751	48.64	3
GPT-5.6 Terra	3876	2751	19.71	2
GPT-5.6 Luna	3876	2751	6.12	7
Claude Opus 5	4175	2592	146.36	47
Claude Opus 4.6	4175	2592	49.07	0
Claude Sonnet 4.6	4175	2592	49.68	1

Appendix

Table 7: Collection accounting for the full-coverage extension. Covered windows include reused valid scores; new windows were scored once successfully, with any recorded failed attempts charged separately. Known dollars are the newly collected calls at list prices, excluding reused historical scores and calibration expenditure. Unknown-usage calls are additional failed attempts whose prices cannot be reconstructed; they are not priced at zero. The total is rounded after summing unrounded call costs.

Cell	$1\times$	$2\times$	$5\times$	$10\times$	FK%
Verified / Luna	-4.5	-2.9	+1.8	+9.7	1.0
Verified / Sonnet	+0.2	+0.3	+0.8	+1.5	0.0
Pro / Luna	-24.2	-23.0	-19.5	-13.6	27.8
Pro / Sonnet	-30.6	-30.5	-30.3	-29.9	8.4

Appendix

Table 8: Stopping-cohort cost-per-solve change under inflated primary-ensemble monitoring charges. Scores, frozen nominal-5% thresholds, visited checks, and false-kill rates remain fixed. Negative values are savings.

Qwen2.5-7B-Instruct
Seed	GPT-5.6 Luna runs		Claude Sonnet 4.6 runs
Seed	Verified	Pro	Verified	Pro
Seed 0	0.615	0.607	0.578	0.548
Seed 1	0.608	0.599	0.522	0.564
Seed 2	0.605	0.586	0.531	0.586
Seed 3	0.598	0.610	0.545	0.582

Appendix

Table 9: Complete training-seed results. Single-model tier AUC of every trained seed on the full ranking cohort.

Early Stopping

Early stopping uses signals during training or inference to halt computation before a fixed budget, balancing generalization, quality, and cost.

See all 44 papers

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
Aug 4, 2026Chenyu Wang, Yunbo Lyu, Junda He +4Software Engineering Agents AI Agent Monitoring
Singapore Management University · University of Alberta · Nanjing University of Science and Technology +1
Structural Baselines Matter for Agent Failure Detection and Attribution
Oct 12, 2026Ramesh Arvind Naagarajan, Stefan StreifAgent Failure Analysis Failure Attribution for LLM Agents
Professorship for Automatic Control and System Dynamics Chemnitz University of Technology, Chemnitz, Germany · Department of Bioresources Fraunhofer Institute for Molecular Biology and Applied Ecology Giessen, Germany
Real-Time Detection and Repair of LLM Agent Failures
Aug 3, 2026Sunny DubeyAI Agent Reliability AI Agent Monitoring

Aug 4, 2026J/K move · Enter open · S save

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks

Chenyu Wang, Yunbo Lyu, Junda He, Zhou Yang+3

Singapore Management University · University of Alberta · Nanjing University of Science and Technology+1

Software Engineering Agents AI Agent Monitoring LLM Agent Reliability

Open paper AskPDF arXiv

Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion. Early termination, however, risks interrupting trajectories that would otherwise succeed; conversely, an unsuccessful trajectory may still contain useful repository edits. We present FailFast-RestartSmart, a two-stage controller for a single active trajectory. FailFast is a lightweight 0.6B monitor trained with terminal and dense fail-to-pass supervision to predict failure from observable prefixes without policy logits or hidden states. Upon an alarm, RestartSmart launches a fresh same-policy rollout without prior prompt history and offers the interrupted repository diff as an optional overlay that the agent may inspect, apply, or discard. On SWE-bench Verified, a monitor trained solely on Qwen3.6-27B trajectories transfers to three other policies, including a closed-API model, and saves 14.6%-20.4% of execution tokens at a target 5% false-positive rate; on Qwen3.6-27B, its 20.4% saving exceeds the 12.5% achieved by our per-step AgentStop adaptation. At a target 25% false-positive rate, RestartSmart raises Qwen3.6-27B resolution from 66.6% to 71.8%, whereas cold restart reaches only 66.8%. Together, these results support early termination with sequential same-policy recovery.

Abstract

Early Stopping

Explore similar work

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks

Structural Baselines Matter for Agent Failure Detection and Attribution

Real-Time Detection and Repair of LLM Agent Failures