CURA: Certified Runtime Alarms for Computer-Use Agents

Divake Kumar; Sina Tayebati; Devashri Naik; Amanda Sofie Rios; Nilesh Ahuja; Omesh Tickoo; Ranganath Krishnan; Amit Ranjan Trivedi

cs.AIAug 28, 2026

CURA: Certified Runtime Alarms for Computer-Use Agents

Authors: Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi

Organizations: University of Illinois Chicago · Intel Labs · Capital One AI Labs

Abstract

Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents (CUAs) it fails precisely where oversight matters. On 361 OSWorld tasks our pipeline, a read-only feasibility gate, a planner, and a GUI executor, reaches a mean task score of 82.9, above the 72.4 human reference, yet 64 of its 71 failures (90%) end with a success claim, 61 acknowledging no blocker, and the explicit failure affordance is never used in roughly 9,100 calls. We introduce CURA (Certified Runtime Alarms for Computer-Use Agents), an external monitor that reads only harness-visible telemetry, with no model internals, extra LLM calls, or prompt changes, and turns the running trajectory into a sequential test with certified false-alarm control. At alpha = 0.10 its CUSUM alarm detects 42.3% of failures a median of 31 steps before termination at a realized false-alarm rate of 0.066, and risk is partly resolvable before the first action (gate probe, 0.69 AUROC). Retrospectively the composite reaches 0.828 AUROC (fold-internal floor 0.802), but its margin over a total-token baseline is not significant (Delta = +0.026, p = 0.101); the separation is online, where CURA recalls more at matched certified budgets: 0.41 versus 0.34 at alpha = 0.10, 0.56 versus 0.38 at alpha = 0.20. Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on 38, giving a deployable cascade at mean score 86.8 and 84.5% full-solve (305 of 361). The certificate bounds false alarms only. We also report where behavioral monitoring is uninformative.

Figures & tables

Figure 1: Verbatim executor reasoning from six failed runs, showing internal recognition of difficulty followed by a completion claim. Across all 361 tasks, 83% of failures (59/71) contain pre-registered pivot-language statements versus 45% of successes (131/290), averaging 25.6 versus 3.5 per run. Failed runs are

2.4\times

longer, so part of the gap is length. Pivot rate alone predicts failure at 0.760 AUROC [0.695, 0.822].

Figure 2: The frozen agent pipeline (left) and the CURA read-only trust layer (right), over 361 OSWorld tasks on an Ubuntu desktop. Despite the gate refusing 20 of 27 infeasible tasks with no false refusals, 90% of failed executions still claim success and none self-declare failure. CURA reads telemetry without modifying the pipeline, fuses four task-agnostic signal families into a running CUSUM risk process, and applies a Learn-then-Test-calibrated threshold with certified false-alarm control.

Agent / System	Steps	Score
General models, vendor harness
qwen3.7-plus ( OSWorld Team 2026 ) †	100	73.3
claude-opus-5 ( OSWorld Team 2026 ) †	100	83.4
claude-fable-5 ( OSWorld Team 2026 ) †	100	86.0
Agentic frameworks
UI-TARS-1.5-7B ( Qin et al. 2025 )	100	27.4

Table 1: OSWorld-Verified mean task score (%). Published rows report no strict full-solve rate; ours are 80.3/84.5/86.7% (290/305/313 of 361 at

\geq 0.99

\dagger

: repository- or leaderboard-reported. Framework scores are as tabulated by Yang et al. (2026) , differ in backbone and step budget, and are context only; Agent S2.5 has no paper. The qwen3.7-plus vendor row is the controlled comparison.

Signal stack	$n$	AUROC	95% CI
Length only (effort)	2	0.794	[0.734, 0.851]
+ reasoning semantics	5	0.797	[0.736, 0.853]
+ execution physiology	7	0.803	[0.745, 0.857]
+ visual dynamics (lean)	8	0.828	[0.775, 0.877]
Fold-internal floor	–	0.802	[0.742, 0.858]
No-length composite	12	0.808	[0.750, 0.860]

Table 2: Build-up of the composite (task-level, out-of-fold, fixed-monitor). Visual dynamics is the significant increment (

\Delta=+0.025

p=0.043

). The fold-internal floor refits success-only models and re-selects signals inside every fold. Tokens alone reach 0.802; the composite’s edge over them is not significant (

p=0.101

Figure 3: The resolvable-risk curve. AUROC of the cumulative composite (vermillion) at each pipeline stage against the best single signal available there (blue, drawn from the composite’s own pool). Risk is measurable before the first action (gate probe, 0.69); the composite matches or leads at every stage. Fixed-monitor protocol; 95% bootstrap CIs.

Figure 4: Accumulated risk

W_{t}

for failed (vermillion) and successful (blue) trajectories over the first 40 steps: cohort medians with interquartile bands, individual runs behind. Medians are conditional on trajectories still active at step

t

, so short runs leave the cohort as they terminate. Dashed line: the certified

\alpha=0.10

threshold (

\theta=16.97

), reached by 42.3% of failures overall.

A. Pre-execution gate
Refuse infeasible	20/27	Precision 1.00	Recall 0.74
B. Certified alarm thresholds (LTT, $\delta=0.05$ )
$\alpha$	FPR	Recall	Lead
0.05	0.024	0.282	42
0.10	0.066	0.423	31
0.20	0.159	0.606	19

Table 3: Oversight decisions and outcomes. Block C matches false-alarm budgets across CURA, token, and step alarms; CURA’s rows are held out, the baselines’ are in-calibration, so the comparison is conservative. Block E reports retained full-solve rate; “cascade” is the 313/361 escalated arm. † All 48 residual failures are rejected in-sample, not a zero-error guarantee. In Block F only the alarm-gated cascade uses no oracle.

Figure 5: Per-domain detection at the certified

\alpha=0.10

threshold: bars are the share of failures detected, dots the false-alarm rate on successes, whiskers 95% CIs, (

n

) the failure count. Detection is high where failures grind (multi-app) and absent where they end quickly and without irregularity (browser, media player, document).

Attempt	Outcome
Token log-probability	No signal beyond length
Reasoning embeddings	$0.748\to 0.738$ ; encoder overfits 71 positives
Single-screenshot LLM judge	Chance (0.497)
Failure-type prediction	Near chance (0.44–0.59)
Per-step weak labels	Worse than trajectory fusion
Escape-key counters	Uninformative ( $\approx 0.5$ )

Table 4: Negative results on the OSWorld corpus. Common confidence and monitoring signals, including token log-probabilities and single-screenshot LLM judging, do not reliably predict trajectory failure.

Role	Tools	Composition
Gate	19	3 read-only probes, 3 file and window, 2 web, 8 GUI navigation, 1 skill selector, 2 verdicts
Planner	1	submit_plan only
Executor	27	12 GUI, 7 code and file, 6 plan management, 2 terminal

Table 5: Tool inventory, counted from the tool-registration lists in the shipped source rather than from documentation.

Component	Model	Access	Pixels?	Inner budget	Notes
The monitored pipeline, frozen throughout
Feasibility gate	qwen3.7-plus	provider API	yes	20 turns	reasoning effort high ; 5 recent images
Planner	qwen3.7-plus	provider API	yes	20 turns	effort max ; 3 recent images
GUI executor	qwen3.7-plus	provider API	yes	20 turns	effort max ; 5 recent images
Context summariser	qwen3.7-plus	provider API	no	n/a	fires only above 80% of the input threshold
Escalation leg, second attempt only

Table 6: Every model in the study and where it sits. The pipeline runs one backbone in three roles; the escalation leg swaps only the planner slot; the 7B agent is a separate, self-hosted open-weight system used as a cross-model check; and CURA itself is a frozen encoder plus two linear fits.

Role	Calls	Uncached in	Cached in	Out
Gate	20	157,996	134,656	3,583
Planner	1	5,846	0	705
Executor	29	277,298	331,008	2,999
Total	50	441,140	465,664	7,287

Table 7: Token accounting for one solved task (17 harness steps, score 1.0). Input dominates output by roughly 60

\times

because every call re-sends the whole conversation. Cache hits covered 51.4% of input tokens, cutting this task from about

0.258 to about

0.143 with no configuration.

Figure 6: Per-domain outcome for the pipeline and for the open-weight 7B baseline on identical tasks, machines, and step budget. Task counts in parentheses. The pipeline leads in every domain, but the ordering of difficulty is largely shared: multi-application tasks are hardest for both, and both are strongest on code-editor work. The 7B series is recomputed at score

\geq 0.99

from the banked task table (Section E.1).

Figure 7: Fraction of the 361 tasks fully solved as a function of the harness step budget. More than two thirds of the eventual total is reached by step 30; the last 25 steps of the budget buy 1.1 points.

Figure 8: Where trajectories end, split by outcome, for both agents. The pipeline’s solved tasks are sharply front-loaded (median 10 steps) while its failures run 2.4

\times

longer on average (median 23, mean 36.1); this is the length confound the main paper flags on the face of Figure 1. The 7B agent shows a different shape entirely: 147 of its trajectories reach the 100-step cap, including 15 that were nonetheless scored correct. A weak agent grinds until the budget stops it; a strong agent stops early and says it succeeded.

Figure 9: The 71 failures, partitioned. Buckets are mutually exclusive and sum to 71. The six upper categories are addressable by mechanisms outside the agent’s policy, such as better gate recall, a claim-time check, or a save-location convention, and together they bound a ceiling of 86.4% without touching the executor. Including the half-done tier reaches 87.0%.

Gate outcome	$n$	Mean cost	Mean steps
Correct refusal	20	$0.007	1.0
Missed infeasible	7	$0.369	31.0
Feasible task (normal)	333	$0.131	20.0

Table 8: Gate economics. A correct refusal terminates inside harness step 1, before the planner and executor are ever constructed, so the entire set of 20 refusals cost less than one average task. The seven misses cost 2.8

\times

an average task each and all ended in a false completion claim.

Figure 10: Precision of the agent’s completion claim, stratified by how long the trajectory ran. A claim made within ten steps is right 88% of the time; a claim made after fifty steps is right 41% of the time.

	Pipeline	UI-TARS-1.5-7B
Backbone	qwen3.7-plus	UI-TARS-1.5-7B
Tasks fully solved	290 / 361	113 / 361
Steps extracted	6,883	19,128
Completion claims	331	214
of which false	64	118
claim precision	80.7%	44.9%

Table 9: The same benchmark under two very different agents. The weaker agent takes 2.8

\times

as many steps and is wrong about its own completion more often than it is right, but it does at least say so 142 times. The stronger agent never once declares failure.

Figure 11: Top single behavioural signals for predicting task failure, computed with identical code on both corpora. Signal directions were fixed by hypothesis before evaluation. The two agents fail in different ways and are therefore predicted by different signals, but in both regimes cheap external behaviour beats the agent’s own report.

Signal stack	7B	95% CI	Pipeline
Length only (effort)	0.784	[0.729, 0.834]	0.794
+ physiology	0.783	[0.730, 0.835]	0.803
+ visual dynamics	0.768	[0.709, 0.823]	0.828

Table 10: The composite build-up refit end to end on the 7B corpus, beside the pipeline column from the main paper. Individual signals replicate; the fusion does not. This is a negative result and we report it as one.

Signal	Family	Definition
out_tokens_total	effort	total executor output tokens
n_steps	effort	executed GUI actions
pivot_rate	semantics	steps whose preceding reasoning matches pivot
saydo_rate	semantics	fraction with a say–do gap
esc_rate	semantics	fraction issuing the Escape key
nll_mean	physiology	mean per-step vitals NLL

Table 11: The fourteen trajectory signals. The first eight are the lean composite (main paper, Table 2,

n=8

); the no-length composite drops out_tokens_total and n_steps and adds four further gate and plan features to reach

n=12

$\alpha$	$\hat{\theta}_{\alpha}$	realized FPR	recall	median lead
0.05	42.70	0.024	0.282	42
0.10	16.97	0.066	0.423	31
0.20	5.47	0.159	0.606	19

Table 12: Learn-then-Test thresholds at

\delta=0.05

, replayed on the calibration successes. These are the main paper’s Table 3B.

	$\alpha{=}0.05$	$\alpha{=}0.10$	$\alpha{=}0.20$
mean test FPR	0.019	0.056	0.140
mean test recall	0.24	0.41	0.56
replicates exceeding $\alpha$	2.0%	4.5%	6.5%

Table 13: Held-out validation over 200 replicates, against the certificate’s nominal

\delta=5\%

tolerance. Detection drops three to five points relative to in-calibration replay.

Alarm	$\alpha$	$\theta$	FPR	recall
token CUSUM	0.05	8.71	0.028	0.268
	0.10	5.85	0.069	0.338
	0.20	3.50	0.159	0.380
step count	0.05	74	0.028	0.183
	0.10	40	0.069	0.338
	0.20	26	0.145	0.465

Table 14: Baseline alarms, calibrated by the same Learn-then-Test procedure. Protocol note: these are single-calibration numbers, directly comparable to Table 12 (CURA 0.282 / 0.423 / 0.606) and not to the held-out row of Table 13 . Block C of the main paper’s Table 3 places CURA’s held-out row beside these, which makes the comparison conservative in CURA’s favour at

\alpha\in\{0.10,0.20\}

and slightly against it at

\alpha=0.05

Stack	AUROC	$\Delta$	$p$ (one-sided)
length only (effort)	0.7942	n/a	n/a
+ reasoning semantics	0.7971	+0.0028	0.445
+ execution physiology	0.8034	+0.0063	0.116
+ visual dynamics	0.8282	+0.0249	0.043

Table 15: Incremental fusion under the fixed-monitor protocol, with paired bootstrap deltas on identical resamples (

B=10^{4}

). Only the visual-dynamics increment is significant. The fully fold-internal variant of the final row, which refits the success-only models and re-runs signal selection inside every training fold, gives 0.802 [0.742, 0.858], the floor quoted in the main paper.

Single signal	AUROC
out_tokens_total	0.8023
nll_max	0.7880
pivot_rate	0.7602
recon_max	0.7407
nll_mean	0.7138
n_steps	0.7022

Table 16: Every signal alone. Total output tokens is the strongest single predictor at 0.8023, which is why the main paper treats the online comparison, not the retrospective AUROC, as the evidence that CURA is more than a token counter.

Stage	Evidence	Composite	Best single
S0	gate probe	0.6883	0.6896
S1	+ plan	0.7364	0.7246
S2	+ steps 1–5	0.7444	0.7185
S2	+ steps 1–15	0.8087	0.7660
S2	+ steps 1–30	0.8258	0.7923
S3	full trajectory	0.8416	0.8023

Table 17: The resolvable-risk curve behind Figure 3, with the best single signal available at each stage drawn from the composite’s own pool. At the gate the best single ( g_out_tok , 0.6896) edges the composite (0.6883); from the plan stage onward the composite leads.

Domain	$n_{\mathrm{fail}}$	recall	$n_{\mathrm{pass}}$	FPR
thunderbird (email)	2	1.000	13	0.077
vs_code (code editor)	1	1.000	22	0.045
multi_apps	28	0.607	65	0.108
gimp (image editor)	6	0.333	20	0.000
os	3	0.333	21	0.000
libreoffice_calc	10	0.300	37	0.081

Table 18: Per-domain detection at the certified

\alpha=0.10

threshold (

\theta=16.97

), the data behind Figure 5. Recomputed from the exported CUSUM paths; totals reproduce Table 12 . Note that an older banked per-domain file covers 9 domains and 70 failures at a raw dial rather than 10 domains and 71 at the certified threshold. This table, not that file, is canonical.

Arm	plan-fixable	residual-hard	total
alarm-gated escalation	17	6	23
blanket Opus planner	23	0	23
union of arms	23	6	29

Table 19: The two arms convert the same number of failures but not the same failures. Alarm-gated escalation reaches six replanning-resistant cases that blanket escalation never touches, because its interventions act mid-execution rather than only at planning time. Denominators: 70 reruns, 69 completed Opus-planner reruns.

Coverage	Retained	Capture	95% CI
base agent, ordered by $\rho(Z)$
100%	80.3%	0%	[76.2, 84.2]
95%	84.0%	23%	[79.6, 87.8]
90%	85.5%	34%	[81.5, 89.2]
80%	88.2%	52%	[84.1, 91.7]
70%	90.9%	68%	[86.6, 94.5]

Table 20: Retained full-solve rate and share of failures captured, by coverage. † In-sample saturation: all 48 residual failures are abstained. This is conditional on the stated coverage and is never a zero-error guarantee. Ignoring the escalation tier and mixing raw risk scores across regimes reaches only 94.8% at 80% coverage.

Target FPR	$\theta$	tokens saved	full-solve
0.00 (no stopping)	$\infty$	0.0%	80.3%
0.02	42.89	17.7%	78.7%
0.05	21.07	23.1%	76.2%
0.10	10.67	28.3%	72.3%
0.20	3.08	36.9%	64.3%

Table 21: Compute recovered by hard-stopping at the alarm, at raw quantile thresholds (descriptive, not certified). The main paper quotes the 10% row: 28.3% of output tokens and 27.0% of steps, at a cost of 8 points of full-solve rate (80.3% to 72.3%), not of mean task score. In the escalation policy flagged trajectories are handed off rather than killed, so that cost is not paid.

Domain / task	call	steps	terminated by
chrome/ae78f875	43	31	done
os/b3d4a89c	207	19	done
calc/1de60575	75	100	step cap
gimp/2a729ded	20	47	done
gimp/8ea73f6f	71	98	done
calc/0326d92d	13	73	done

Table 22: Provenance of the six quotes. Ellipses in the figure mark elided text between non-adjacent sentences; no quote is paraphrased or joined.

Population	$n$	Definition
evaluation corpus	361	sole denominator for accuracy
successes / failures	290 / 71	fail $=$ score $<0.99$
rerun population	70	71 minus the case-study task
completed reruns	69	one rerun did not complete
residual failures	48	46 escalated-leg $+$ 2 carried
step corpus	6,883	executed GUI actions

Table 23: Every denominator used anywhere in the paper.

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 281 papers

How Benchmarks Mis-Score Computer-Use Agents
Jul 30, 2026Zihan Dong, Zhiyuan Ma, Zekun Wang +5Agent Failure Analysis Computer-Use Agent Benchmarks
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents
Oct 3, 2026Xueping GaoAI Agent Monitoring Language Model Safety Evaluation
Alibaba Cloud Computing Hangzhou, China
RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors
Oct 9, 2026David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin +1Robot Failure Detection Robot Failure Recovery
CSIRO, Australia

Jul 30, 2026J/K move · Enter open · S save

How Benchmarks Mis-Score Computer-Use Agents

Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li+4

Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center+3

Agent Failure Analysis Computer-Use Agent Benchmarks Benchmark Auditing Computer-Use Agents Benchmark Validity Automated Evaluation+1

Open paper AskPDF arXiv

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Abstract

AI Agent Monitoring

Explore similar work

How Benchmarks Mis-Score Computer-Use Agents

From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents

RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors