Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents (CUAs) it fails precisely where oversight matters. On 361 OSWorld tasks our pipeline, a read-only feasibility gate, a planner, and a GUI executor, reaches a mean task score of 82.9, above the 72.4 human reference, yet 64 of its 71 failures (90%) end with a success claim, 61 acknowledging no blocker, and the explicit failure affordance is never used in roughly 9,100 calls. We introduce CURA (Certified Runtime Alarms for Computer-Use Agents), an external monitor that reads only harness-visible telemetry, with no model internals, extra LLM calls, or prompt changes, and turns the running trajectory into a sequential test with certified false-alarm control. At alpha = 0.10 its CUSUM alarm detects 42.3% of failures a median of 31 steps before termination at a realized false-alarm rate of 0.066, and risk is partly resolvable before the first action (gate probe, 0.69 AUROC). Retrospectively the composite reaches 0.828 AUROC (fold-internal floor 0.802), but its margin over a total-token baseline is not significant (Delta = +0.026, p = 0.101); the separation is online, where CURA recalls more at matched certified budgets: 0.41 versus 0.34 at alpha = 0.10, 0.56 versus 0.38 at alpha = 0.20. Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on 38, giving a deployable cascade at mean score 86.8 and 84.5% full-solve (305 of 361). The certificate bounds false alarms only. We also report where behavioral monitoring is uninformative.
Figures & tables
Figure 1: Verbatim executor reasoning from six failed runs, showing internal recognition of difficulty followed by a completion claim. Across all 361 tasks, 83% of failures (59/71) contain pre-registered pivot-language statements versus 45% of successes (131/290), averaging 25.6 versus 3.5 per run. Failed runs are 2.4× longer, so part of the gap is length. Pivot rate alone predicts failure at 0.760 AUROC [0.695, 0.822].
Figure 2: The frozen agent pipeline (left) and the CURA read-only trust layer (right), over 361 OSWorld tasks on an Ubuntu desktop. Despite the gate refusing 20 of 27 infeasible tasks with no false refusals, 90% of failed executions still claim success and none self-declare failure. CURA reads telemetry without modifying the pipeline, fuses four task-agnostic signal families into a running CUSUM risk process, and applies a Learn-then-Test-calibrated threshold with certified false-alarm control.
Agent / System
Steps
Score
General models, vendor harness
qwen3.7-plus ( OSWorld Team 2026 ) †
100
73.3
claude-opus-5 ( OSWorld Team 2026 ) †
100
83.4
claude-fable-5 ( OSWorld Team 2026 ) †
100
86.0
Agentic frameworks
UI-TARS-1.5-7B ( Qin et al. 2025 )
100
27.4
Table 1: OSWorld-Verified mean task score (%). Published rows report no strict full-solve rate; ours are 80.3/84.5/86.7% (290/305/313 of 361 at ≥0.99 ). † : repository- or leaderboard-reported. Framework scores are as tabulated by Yang et al. (2026) , differ in backbone and step budget, and are context only; Agent S2.5 has no paper. The qwen3.7-plus vendor row is the controlled comparison.
Signal stack
n
AUROC
95% CI
Length only (effort)
2
0.794
[0.734, 0.851]
+ reasoning semantics
5
0.797
[0.736, 0.853]
+ execution physiology
7
0.803
[0.745, 0.857]
+ visual dynamics (lean)
8
0.828
[0.775, 0.877]
Fold-internal floor
–
0.802
[0.742, 0.858]
No-length composite
12
0.808
[0.750, 0.860]
Table 2: Build-up of the composite (task-level, out-of-fold, fixed-monitor). Visual dynamics is the significant increment ( Δ=+0.025 , p=0.043 ). The fold-internal floor refits success-only models and re-selects signals inside every fold. Tokens alone reach 0.802; the composite’s edge over them is not significant ( p=0.101 ).
Figure 3: The resolvable-risk curve. AUROC of the cumulative composite (vermillion) at each pipeline stage against the best single signal available there (blue, drawn from the composite’s own pool). Risk is measurable before the first action (gate probe, 0.69); the composite matches or leads at every stage. Fixed-monitor protocol; 95% bootstrap CIs.
Figure 4: Accumulated risk Wt for failed (vermillion) and successful (blue) trajectories over the first 40 steps: cohort medians with interquartile bands, individual runs behind. Medians are conditional on trajectories still active at step t , so short runs leave the cohort as they terminate. Dashed line: the certified α=0.10 threshold ( θ=16.97 ), reached by 42.3% of failures overall.
A. Pre-execution gate
Refuse infeasible
20/27
Precision 1.00
Recall 0.74
B. Certified alarm thresholds (LTT, δ=0.05 )
α
FPR
Recall
Lead
0.05
0.024
0.282
42
0.10
0.066
0.423
31
0.20
0.159
0.606
19
Table 3: Oversight decisions and outcomes. Block C matches false-alarm budgets across CURA, token, and step alarms; CURA’s rows are held out, the baselines’ are in-calibration, so the comparison is conservative. Block E reports retained full-solve rate; “cascade” is the 313/361 escalated arm. † All 48 residual failures are rejected in-sample, not a zero-error guarantee. In Block F only the alarm-gated cascade uses no oracle.
Figure 5: Per-domain detection at the certified α=0.10 threshold: bars are the share of failures detected, dots the false-alarm rate on successes, whiskers 95% CIs, ( n ) the failure count. Detection is high where failures grind (multi-app) and absent where they end quickly and without irregularity (browser, media player, document).
Attempt
Outcome
Token log-probability
No signal beyond length
Reasoning embeddings
0.748→0.738 ; encoder overfits 71 positives
Single-screenshot LLM judge
Chance (0.497)
Failure-type prediction
Near chance (0.44–0.59)
Per-step weak labels
Worse than trajectory fusion
Escape-key counters
Uninformative ( ≈0.5 )
Table 4: Negative results on the OSWorld corpus. Common confidence and monitoring signals, including token log-probabilities and single-screenshot LLM judging, do not reliably predict trajectory failure.
12 GUI, 7 code and file, 6 plan management, 2 terminal
Table 5: Tool inventory, counted from the tool-registration lists in the shipped source rather than from documentation.
Component
Model
Access
Pixels?
Inner budget
Notes
The monitored pipeline, frozen throughout
Feasibility gate
qwen3.7-plus
provider API
yes
20 turns
reasoning effort high ; 5 recent images
Planner
qwen3.7-plus
provider API
yes
20 turns
effort max ; 3 recent images
GUI executor
qwen3.7-plus
provider API
yes
20 turns
effort max ; 5 recent images
Context summariser
qwen3.7-plus
provider API
no
n/a
fires only above 80% of the input threshold
Escalation leg, second attempt only
Table 6: Every model in the study and where it sits. The pipeline runs one backbone in three roles; the escalation leg swaps only the planner slot; the 7B agent is a separate, self-hosted open-weight system used as a cross-model check; and CURA itself is a frozen encoder plus two linear fits.
Role
Calls
Uncached in
Cached in
Out
Gate
20
157,996
134,656
3,583
Planner
1
5,846
0
705
Executor
29
277,298
331,008
2,999
Total
50
441,140
465,664
7,287
Table 7: Token accounting for one solved task (17 harness steps, score 1.0). Input dominates output by roughly 60 × because every call re-sends the whole conversation. Cache hits covered 51.4% of input tokens, cutting this task from about 0.258toabout0.143 with no configuration.
Figure 6: Per-domain outcome for the pipeline and for the open-weight 7B baseline on identical tasks, machines, and step budget. Task counts in parentheses. The pipeline leads in every domain, but the ordering of difficulty is largely shared: multi-application tasks are hardest for both, and both are strongest on code-editor work. The 7B series is recomputed at score ≥0.99 from the banked task table (Section E.1).
Figure 7: Fraction of the 361 tasks fully solved as a function of the harness step budget. More than two thirds of the eventual total is reached by step 30; the last 25 steps of the budget buy 1.1 points.
Figure 8: Where trajectories end, split by outcome, for both agents. The pipeline’s solved tasks are sharply front-loaded (median 10 steps) while its failures run 2.4 × longer on average (median 23, mean 36.1); this is the length confound the main paper flags on the face of Figure 1. The 7B agent shows a different shape entirely: 147 of its trajectories reach the 100-step cap, including 15 that were nonetheless scored correct. A weak agent grinds until the budget stops it; a strong agent stops early and says it succeeded.
Figure 9: The 71 failures, partitioned. Buckets are mutually exclusive and sum to 71. The six upper categories are addressable by mechanisms outside the agent’s policy, such as better gate recall, a claim-time check, or a save-location convention, and together they bound a ceiling of 86.4% without touching the executor. Including the half-done tier reaches 87.0%.
Gate outcome
n
Mean cost
Mean steps
Correct refusal
20
$0.007
1.0
Missed infeasible
7
$0.369
31.0
Feasible task (normal)
333
$0.131
20.0
Table 8: Gate economics. A correct refusal terminates inside harness step 1, before the planner and executor are ever constructed, so the entire set of 20 refusals cost less than one average task. The seven misses cost 2.8 × an average task each and all ended in a false completion claim.
Figure 10: Precision of the agent’s completion claim, stratified by how long the trajectory ran. A claim made within ten steps is right 88% of the time; a claim made after fifty steps is right 41% of the time.
Pipeline
UI-TARS-1.5-7B
Backbone
qwen3.7-plus
UI-TARS-1.5-7B
Tasks fully solved
290 / 361
113 / 361
Steps extracted
6,883
19,128
Completion claims
331
214
of which false
64
118
claim precision
80.7%
44.9%
Table 9: The same benchmark under two very different agents. The weaker agent takes 2.8 × as many steps and is wrong about its own completion more often than it is right, but it does at least say so 142 times. The stronger agent never once declares failure.
Figure 11: Top single behavioural signals for predicting task failure, computed with identical code on both corpora. Signal directions were fixed by hypothesis before evaluation. The two agents fail in different ways and are therefore predicted by different signals, but in both regimes cheap external behaviour beats the agent’s own report.
Signal stack
7B
95% CI
Pipeline
Length only (effort)
0.784
[0.729, 0.834]
0.794
+ physiology
0.783
[0.730, 0.835]
0.803
+ visual dynamics
0.768
[0.709, 0.823]
0.828
Table 10: The composite build-up refit end to end on the 7B corpus, beside the pipeline column from the main paper. Individual signals replicate; the fusion does not. This is a negative result and we report it as one.
Signal
Family
Definition
out_tokens_total
effort
total executor output tokens
n_steps
effort
executed GUI actions
pivot_rate
semantics
steps whose preceding reasoning matches pivot
saydo_rate
semantics
fraction with a say–do gap
esc_rate
semantics
fraction issuing the Escape key
nll_mean
physiology
mean per-step vitals NLL
Table 11: The fourteen trajectory signals. The first eight are the lean composite (main paper, Table 2, n=8 ); the no-length composite drops out_tokens_total and n_steps and adds four further gate and plan features to reach n=12 .
α
θ^α
realized FPR
recall
median lead
0.05
42.70
0.024
0.282
42
0.10
16.97
0.066
0.423
31
0.20
5.47
0.159
0.606
19
Table 12: Learn-then-Test thresholds at δ=0.05 , replayed on the calibration successes. These are the main paper’s Table 3B.
α=0.05
α=0.10
α=0.20
mean test FPR
0.019
0.056
0.140
mean test recall
0.24
0.41
0.56
replicates exceeding α
2.0%
4.5%
6.5%
Table 13: Held-out validation over 200 replicates, against the certificate’s nominal δ=5% tolerance. Detection drops three to five points relative to in-calibration replay.
Alarm
α
θ
FPR
recall
token CUSUM
0.05
8.71
0.028
0.268
0.10
5.85
0.069
0.338
0.20
3.50
0.159
0.380
step count
0.05
74
0.028
0.183
0.10
40
0.069
0.338
0.20
26
0.145
0.465
Table 14: Baseline alarms, calibrated by the same Learn-then-Test procedure. Protocol note: these are single-calibration numbers, directly comparable to Table 12 (CURA 0.282 / 0.423 / 0.606) and not to the held-out row of Table 13 . Block C of the main paper’s Table 3 places CURA’s held-out row beside these, which makes the comparison conservative in CURA’s favour at α∈{0.10,0.20} and slightly against it at α=0.05 .
Stack
AUROC
Δ
p (one-sided)
length only (effort)
0.7942
n/a
n/a
+ reasoning semantics
0.7971
+0.0028
0.445
+ execution physiology
0.8034
+0.0063
0.116
+ visual dynamics
0.8282
+0.0249
0.043
Table 15: Incremental fusion under the fixed-monitor protocol, with paired bootstrap deltas on identical resamples ( B=104 ). Only the visual-dynamics increment is significant. The fully fold-internal variant of the final row, which refits the success-only models and re-runs signal selection inside every training fold, gives 0.802 [0.742, 0.858], the floor quoted in the main paper.
Single signal
AUROC
out_tokens_total
0.8023
nll_max
0.7880
pivot_rate
0.7602
recon_max
0.7407
nll_mean
0.7138
n_steps
0.7022
Table 16: Every signal alone. Total output tokens is the strongest single predictor at 0.8023, which is why the main paper treats the online comparison, not the retrospective AUROC, as the evidence that CURA is more than a token counter.
Stage
Evidence
Composite
Best single
S0
gate probe
0.6883
0.6896
S1
+ plan
0.7364
0.7246
S2
+ steps 1–5
0.7444
0.7185
S2
+ steps 1–15
0.8087
0.7660
S2
+ steps 1–30
0.8258
0.7923
S3
full trajectory
0.8416
0.8023
Table 17: The resolvable-risk curve behind Figure 3, with the best single signal available at each stage drawn from the composite’s own pool. At the gate the best single ( g_out_tok , 0.6896) edges the composite (0.6883); from the plan stage onward the composite leads.
Domain
nfail
recall
npass
FPR
thunderbird (email)
2
1.000
13
0.077
vs_code (code editor)
1
1.000
22
0.045
multi_apps
28
0.607
65
0.108
gimp (image editor)
6
0.333
20
0.000
os
3
0.333
21
0.000
libreoffice_calc
10
0.300
37
0.081
Table 18: Per-domain detection at the certified α=0.10 threshold ( θ=16.97 ), the data behind Figure 5. Recomputed from the exported CUSUM paths; totals reproduce Table 12 . Note that an older banked per-domain file covers 9 domains and 70 failures at a raw dial rather than 10 domains and 71 at the certified threshold. This table, not that file, is canonical.
Arm
plan-fixable
residual-hard
total
alarm-gated escalation
17
6
23
blanket Opus planner
23
0
23
union of arms
23
6
29
Table 19: The two arms convert the same number of failures but not the same failures. Alarm-gated escalation reaches six replanning-resistant cases that blanket escalation never touches, because its interventions act mid-execution rather than only at planning time. Denominators: 70 reruns, 69 completed Opus-planner reruns.
Coverage
Retained
Capture
95% CI
base agent, ordered by ρ(Z)
100%
80.3%
0%
[76.2, 84.2]
95%
84.0%
23%
[79.6, 87.8]
90%
85.5%
34%
[81.5, 89.2]
80%
88.2%
52%
[84.1, 91.7]
70%
90.9%
68%
[86.6, 94.5]
Table 20: Retained full-solve rate and share of failures captured, by coverage. † In-sample saturation: all 48 residual failures are abstained. This is conditional on the stated coverage and is never a zero-error guarantee. Ignoring the escalation tier and mixing raw risk scores across regimes reaches only 94.8% at 80% coverage.
Target FPR
θ
tokens saved
full-solve
0.00 (no stopping)
∞
0.0%
80.3%
0.02
42.89
17.7%
78.7%
0.05
21.07
23.1%
76.2%
0.10
10.67
28.3%
72.3%
0.20
3.08
36.9%
64.3%
Table 21: Compute recovered by hard-stopping at the alarm, at raw quantile thresholds (descriptive, not certified). The main paper quotes the 10% row: 28.3% of output tokens and 27.0% of steps, at a cost of 8 points of full-solve rate (80.3% to 72.3%), not of mean task score. In the escalation policy flagged trajectories are handed off rather than killed, so that cost is not paid.
Domain / task
call
steps
terminated by
chrome/ae78f875
43
31
done
os/b3d4a89c
207
19
done
calc/1de60575
75
100
step cap
gimp/2a729ded
20
47
done
gimp/8ea73f6f
71
98
done
calc/0326d92d
13
73
done
Table 22: Provenance of the six quotes. Ellipses in the figure mark elided text between non-adjacent sentences; no quote is paraphrased or joined.
Population
n
Definition
evaluation corpus
361
sole denominator for accuracy
successes / failures
290 / 71
fail = score <0.99
rerun population
70
71 minus the case-study task
completed reruns
69
one rerun did not complete
residual failures
48
46 escalated-leg + 2 carried
step corpus
6,883
executed GUI actions
Table 23: Every denominator used anywhere in the paper.
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.
Zihan Dong, Zhiyuan Ma, Zekun Wang +5
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent monitors make thresholded alarm decisions under tight false-alarm budgets. These are different estimands. We introduce an Operational Validity Contract that fixes a monitor's target, observability, identity, timing, intervention unit, comparator, calibration, and cost. We formalize risk at the semantic request or trajectory level: when one task contains repeated alarm opportunities, row-level AUROC and false positive rate do not identify semantic-unit any-alarm risk. A confidence-certified threshold also requires enough independent negative units, a tie-safe rule, and transport to deployment. Across Models Under Pressure, LASR refusal prediction, immutable AgentDojo, and a prospectively protocol-frozen ST-WebAgentBench replication, joint activation-observable monitors reach AUROC 0.957 and 0.935 on the first two benchmarks, yet their locked 5%/10% detection rates are only .642/.742 and .719/.782, respectively. On AgentDojo, the secondary mean-activation rollout monitor reaches AUROC 0.922 but detects none of 38 positive semantic cases at the locked 5% operating point; thresholds intended for 10% false alarms realize 18.7-20.0% on test. Because the test misses its prospectively frozen 40-positive support gate, we label it support-insufficient. On ST-WebAgentBench, activation reaches AUROC .874, but 23 independent calibration negatives cannot identify even a 10% controller; the locked policy abstains rather than reporting its mechanical zero FPR as a success. An exploratory counterexample also lowers full AUROC while improving realized 10% utility. The fail-closed compiler caps MUP and LASR at restricted predictive value and AgentDojo and ST-Web at representation accessibility; no setting reaches alarm-policy validity.
Runtime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions. Yet an accurate alarm may arrive too late for recovery, while an unnecessary intervention may disrupt an execution that would otherwise succeed. We introduce RESCUE, a counterfactual framework for evaluating whether a monitor's alarms enable useful interventions. By restoring simulator and policy states at execution checkpoints, RESCUE compares alternative actions, identifies the latest checkpoint satisfying a specified recovery criterion, and evaluates whether acting on alarms reduces expected task-completion cost. We evaluate RESCUE with OpenVLA-OFT and π0.5 over LIBERO-Spatial and LIBERO-Plus using 128,340 counterfactual branches. Recoverability depends on the state--intervention pair rather than on robot state alone: at τ=0.5, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures contain a recoverable checkpoint under continuation, compared with 34 of 66 under continuation or scripted recovery, while recovery reduces recoverability in other conditions. On the primary strong-policy conditions, favourable detection scores do not establish lower intervention cost. Under the evaluated stop-and-reset model, even an idealised detector can afford only a 2.0--3.3% false-alarm rate before unnecessary resets outweigh the savings from stopping failures early. These results show that monitor value depends jointly on the intervention available and the workload in which it is deployed.
David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin +1