Web navigation agents built on open-weight language models fail often and fail late, with most of the tokens spent after the trajectory has become unrecoverable. Recent work has shown that early termination is a viable cost-reduction strategy, but existing approaches produce opaque termination decisions that do not support diagnosis or human oversight. In this work we apply closed sequential pattern mining (BIDE+) to K-step prefixes of agent execution traces to produce a small, human-reviewable failure-signature library that organizes into a four-category failure taxonomy with associated repair recommendations. We use a multi-variant evaluation protocol that explicitly removes contamination artifacts (zero-step crashes and trivially separable single-step successes) so that reported detection accuracy reflects behavioral discrimination rather than trace-length distributions. We evaluate our methodology on 1,544 MiniWoB++ traces collected from three open-weight language models, achieving the best F1 at K=3 on the Exclude-errors configuration across eight methods (0.717), recovering 88% of test-set failures within three steps, and supporting an early-termination policy that saves 26.3% of tokens at 92% precision.
Figures & tables
Method
F1@3
F1@5
F1@8
F1@10
AUC-PR@5
BIDE Coverage (ours)
0.661
0.683
0.678
0.678
0.729
Bi-LSTM [ 9 ]
0.630
0.654
0.648
0.631
0.741
Frequency Vector
0.596
0.667
0.649
0.649
0.716
n -gram
0.562
0.568
0.571
0.571
0.736
TaSPM [ 10 ]
0.618
0.618
0.618
0.618
0.591
DeepLog [ 8 ]
0.430
0.500
0.535
0.529
0.610
Table 1: Variant C balanced (natural failures vs. successes). BIDE Coverage leads from K=5 . Step Count is a length-only control: methods that fail to exceed it are not extracting behavioral signal.
Threshold
Terminated
TP
FP
Precision
Recall
Success Kill Rate
Overall Savings
0.05
129
91
38
0.705
0.752
0.306
54.5%
0.10
103
75
28
0.728
0.620
0.226
46.2%
0.15
50
46
4
0.920
0.380
0.032
26.3%
0.20
30
29
1
0.967
0.240
0.008
16.2%
0.25
9
9
0
1.000
0.074
0.000
4.4%
Table 2: Online detection operating points on Exclude-errors at K=5 . The boldface row is the recommended operating point: 92% precision, 3.2% impact on successful traces, 26.3% overall token savings.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
F1@3
F1@5
F1@8
F1@10
AUC-PR@10
BIDE Coverage (ours)
0.717
0.677
0.683
0.612
0.757
Frequency Vector
0.712
0.717
0.730
0.730
0.822
n -gram
0.690
0.706
0.731
0.744
0.815
Bi-LSTM [ 9 ]
0.657
0.713
0.731
0.765
0.861
TaSPM [ 10 ]
0.676
0.650
0.656
0.656
0.622
Step Count (control)
0.667
0.667
0.667
0.667
0.500
Appendix
Table 3: Exclude-errors balanced: F1 at each prefix length and AUC-PR at K=10 . BIDE Coverage leads at K=3 ; neural and feature methods overtake at longer prefixes.
Reliable web-agent monitoring is difficult when model-internal uncertainty signals such as token logits are unavailable. In this work, we study prefix-level risk prediction for web agents using observable trajectory signals: given an evolving prefix, estimate whether the current execution remains on track or is tending toward failure. We derive two observable trajectory representations: Macro features summarize cross-step agent--environment behavior and feedback, while Micro features measure the consistency of intention, action, and anticipated state change through repeated black-box queries. Instead of inheriting the final result label, we label the first critical error that remains uncorrected in the observed continuation and is associated with final failure as a key-step boundary, preserving valid early prefixes of failed trajectories as on track. Across WebArena-Lite and Online Mind2Web web agent benchmarks with five open- and closed-source backbones, observable trajectory signals are competitive with internal-signal baselines. The resulting predictors also support early intervention under fixed false-cut budgets and transfer across held-out website categories. These findings show that observable trajectory signals support valuable risk prediction abilities.
Sitong Pan, Yipeng Shen, Yilin Lu +3
University of Minnesota · 2Purdue University · 3The Pennsylvania State University
Long-horizon web agents often fail in ways hidden by final-answer evaluation: they may visit useful pages, produce a well-formed answer, and terminate confidently while still missing fields, over-including unsupported items, or relying on stale evidence. We study these failures with Parallel WebBench, a parallel web-exploration benchmark containing 1,679 verified records: 350 manually curated parallel tasks and 1,329 reconstructed records with verified URL-based trajectories. We train WebExplorer-style agents with GRPO under human-only, balanced human-synthetic, and synthetic-heavy data mixtures. At 16k context and 16 interaction rounds, the best GRPO model improves completion over WebExplorer-8B from 50.7% to 96.0% and GPT-4.1-mini-judged element-wise F1 from 0.2489 to 0.4529, but binary accuracy remains far below completion. Trace-level analysis identifies three persistent failure modes: context-bound search loops, premature termination on partial answers, and synthesis collapse after relevant evidence has already been retrieved. These results show that synthetic-data GRPO reduces abstention and improves partial correctness, but leaves a completion-correctness gap that requires evidence-grounded coverage and synthesis diagnostics.
Aagam Sogani, Botao Rui, Swetha Vaidyanathan +3
University of Wisconsin–Madison, Madison, WI, USA.
Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5% on Verified while preserving 99% of successful runs, and by up to 31% on the costlier Pro subset, reaching 48% with a Qwen3-8B backbone, and in all four settings its cost per retained solve is below that of every judge policy. Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.