Organizations: University of Texas at Dallas · Independent Researcher · University of Manchester · Rice University · University of Texas at Austin · University of California, Davis
As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state and cause cascading failures in subsequent actions. Reliable error detection and recovery have become critical bottlenecks for building robust CUAs. However, existing benchmarks focus primarily on end-to-end task performance, providing limited insight into whether agents can detect and recover from errors during execution. The few benchmarks that evaluate recovery rely largely on manually injected errors, which may not reflect the types or horizons of failures induced by real agent policies. We introduce CUA-Recovery, a benchmark comprising test and training suites across multiple desktop applications. We construct it using ReRail, a framework for synthesizing long-horizon tasks and trajectories in a persistent desktop environment. ReRail composes MyPCBench tasks into new long-horizon workflows with executable ground truth, collects verifier-confirmed failures from multiple policies, and turns them into verified recovery trajectories. CUA-Recovery comprises a human-verified test suite of 2,250 erroneous states over 100 tasks and a training suite of 8.6K trajectories over 377 tasks. Evaluation on the test suite shows that agents struggle both to recognize inherited errors and to complete the task. Across six agents and six takeover depths, Error Awareness Rate (EAR) reaches at most 44.0% and Pass@3 at most 25.5%. After fine-tuning Qwen3.5-35B-A3B on the training suite, its average EAR rises from 19.9% to 35.2% on held-out recovery states, and its Rubric Score goes from 5.0% to 16.1%. It also gains 15.3 points on MyPCBench and 10.8 points on Odysseys for Pass@3, showing that the training trajectories help both with recovery and with completing tasks from a clean start. Our code is available at https://github.com/xu-hu-2002/CUA-Recovery.
Figures & tables
Figure 1: Case study: recovering from a real policy-induced error. The user asks for his usual Sweetgreen order. Top: Kimi-K3 fails to notice and recover from the error. Bottom: ReRail-35B-A3B takes over at the root-cause step, detects the error, and recovers from it.
Benchmark
Platform
Traj.
Mean steps
Multi-app
Interface
Error source
Recovery
AndroidControl ( Li et al., 2024 )
Mobile
15,283
5.5
–
GUI
Only perfect traj.
×
AgentTrek ( Xu et al., 2025 )
Web
10,398
12.1
–
GUI
Only perfect traj.
×
ScaleCUA ( Liu et al., 2026b )
Cross
19K
9.0
–
GUI
Only perfect traj.
×
VideoAgentTrek ( Lu et al., 2025a )
Cross
39K
39.3
–
GUI
Only perfect traj.
×
AgentNet ( Wang et al., 2025 )
Desktop
22,532
18.8
7.2%
GUI
Human demo errors
×
GUI-Reflection ( Wu et al., 2025 )
Mobile
8,640
3.8
–
GUI
Injected errors
Single-step
Table 1: Comparison with existing CUA training sets. Recovery indicates whether the data contains error corrections: Single-step corrections immediately follow an injected error; Verified recoveries start from real failure prefixes and are kept only if the verifier confirms task completion.
Figure 2: Overview of our methodology and benchmark.
Figure 3: Error types and the delay before an error becomes explicit.
Error Awareness (%)
Rubric Score / Pass@3 (%)
Agent d=
0
5
10
15
20
25
0
5
10
15
20
25
API-based
GPT 5.5
20.1
26.7
32.3
28.6
16.2
21.0
65.3/20.1
67.7/24.3
61.2/20.9
60.8/20.1
68.5/25.5
54.9/18.9
Opus 4.8
18.9
44.0
39.1
22.7
18.8
27.3
35.3/0.0
24.0/0.0
29.5/4.5
38.5/5.3
48.1/7.7
36.4/0.0
Kimi-K3
13.9
7.8
10.0
11.1
18.5
15.0
39.2/11.6
41.7/8.6
36.4/7.4
33.1/11.8
41.5/23.1
36.9/9.1
↪ ReRail
25.0
15.7
17.5
22.9
14.8
15.0
36.4/ 16.6
34.8/ 12.6
31.7/ 10.1
32.4/11.6
36.8/15.2
30.0/8.8
Table 2: Recovery on CUA-Recovery-Test by takeover depth d (steps after the root cause). Each agent takes over states from its own failures without an error warning. ↪ ReRail: ReRail-35B-A3B takes over the same states as the agent above. Opus 4.8 = Claude Opus 4.8; Qwen3.5-35B = Qwen3.5-35B-A3B, the base model of ReRail. Bold : higher than the corresponding source agent.
MyPCBench
Odysseys
Agent
Pass@3 (%)
Rubric Score (%)
Pass@3 (%)
Rubric Score (%)
API-based
GPT 5.5
85.0
92.2
82.1
94.2
Claude Opus 4.8
68.9
84.6
81.7
95.6
Kimi-K3
66.9
83.6
77.9
94.3
Open-source
Table 3: Completion performance on MyPCBench and Odysseys. Bold and underline indicate the best and second-best results within API-based and open-source agents.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Unit
Unit cost (USD)
Count
Total (USD)
Clean-start collection
API agent rollouts (Claude Opus 4.8, GPT 5.5; 3 runs)
Table 4: API cost. A judge group is one run of one agent’s clean-start rollouts ( 3 runs per agent) or one agent–horizon cell of takeover episodes. Red values are estimates.
Agent
Episodes
Min./episode
GPU-hours
Sandbox-hours
Clean-start collection
Claude Opus 4.8
552
37.6
–
345
GPT 5.5
552
9.2
–
84
Kimi-K3
552
30.5
–
282
EvoCUA-32B
552
25.1
1,872
234
OpenCUA-72B
552
7.6
558
69
Appendix
Table 5: GPU and sandbox compute. Each locally served model runs one episode at a time on a server of eight AMD MI308X GPUs, so GPU-hours are eight times episode wall-clock time; API agents use sandbox time only. Clean-start times are measured; takeover times use each agent’s measured per-step latency over 58 post-takeover steps.
Stage
Count
Composed candidates reaching the executable gate
364
Rejected by a static gate
82
Candidates with a gold run (composed + source)
417
Mutation executions
17,857
Candidates with R(M)≥0.5
327
Selected tasks (composed / source)
277 / 10
Appendix
Table 6: The synthesis funnel on one generation run over the 135 accepted source IRs.
Train
Test
Tasks
377
100
Examples
8,577
2,250
Mean steps per trajectory
60.0
59.7
Multi-application trajectories (%)
67.5
68.4
Label checks
Automatic
Human
Appendix
Table 7: Statistics of CUA-Recovery. Training examples are verified recovery trajectories; test examples are erroneous takeover states.
Annotation
Description
Overall (%)
Planning
21.2
Fabricate Data
Reports specific facts, values, or records that the trajectory provides no evidence for.
9.1
Scope Error
Operates over the wrong time range, objects, records, population, or coverage.
8.1
Misunderstand Task Objective
Pursues the wrong end goal, deliverable, object, or key constraint.
3.9
Perception
36.1
Ineffective Action
The action neither advances the task nor produces the intended state change.
12.6
Appendix
Table 8: Error category distribution in CUA-Recovery-Test.
Figure 4: Clean-start performance of the six agents on the MyPCBench source tasks, by task category. Rows are ordered from lowest to highest Pass@3 and columns by overall Pass@3. (a) Pass@3 exposes categories in which several agents score zero. (b) The Rubric Score resolves partial progress within those zero-success cells. Each cell reports its sample size; row and column labels report marginal values.
Figure 5: Case 1: Fabricate Data. Columns show task evidence, root error, and consequence.
Figure 6: Case 2: Scope Error. Columns show task evidence, root error, and consequence.
Figure 7: Case 3: Detail Misperception. Columns show task evidence, root error, and consequence.
Figure 8: Case 4: State Misinterpretation. Columns show task evidence, root error, and consequence.
Figure 9: Case 5: Grounding Failure. Columns show task evidence, root error, and consequence.
Figure 10: Case 6: Incorrect UI Element. Columns show task evidence, root error, and consequence.
Figure 11: Case 7: Ineffective Action. Columns show task evidence, root error, and consequence.
Agent
{SCAFFOLD} (agent-native)
Blocks
Context
{TOOL_NOTE}
GPT 5.5
OpenAI CUA lead-in: tools computer , shell ; stop with DONE / FAIL / [INFEASIBLE] . Sent as the first user message.
S,V,P,N,K,G
shell
–
Claude Opus 4.8
Claude CUA <role> : tools computer , bash , str_replace_based_edit_tool ; same stop signal. Blocks in <instructions> / <safety> , context in <environment> .
S,V,P,N,K
shell
–
Qwen3.5-35B-A3B
Qwen tool-call prompt: computer_use schema (1000 × 1000), XML <tool_call> , “Action + tool call” format, terminate(status) .
–
shell
bash tool
Kimi-K3
General VLM tool agent (ours): 1280 × 800 absolute pixels, one tool call per step, answer(status) ; tools via function calling.
Table 9: Slot values per agent. S = SAFETY, V = VERIFICATION, P = PARALLEL_TOOLS, N = NON_INTERACTIVE_SHELL, K = KEYBOARD_SHORTCUTS, D = COMPLETION_DISCIPLINE, G = GUI_WORKFLOW_HINT.
Agent
Agree. (%)
κ
F1
Qwen3.5-35B-A3B
97.7
0.88
0.89
EvoCUA-32B
89.2
0.77
0.85
OpenCUA-72B
98.6
0.86
0.87
Appendix
Table 10: Human–human agreement on the trajectory-level full completion judgment, measured on the 30% double-annotated subsets.
Pass@3
LLM-as-Judge vs. Human-as-Judge
Agent
#Tasks
Human-as-Judge
LLM-as-Judge
Consistency
Precision
Recall
F1
κ
Qwen3.5-35B-A3B
43
11.6%
9.3%
0.977
1.000
0.800
0.889
0.876
EvoCUA-32B
37
40.5%
29.7%
0.892
1.000
0.733
0.846
0.766
Total
80
25.0%
18.8%
0.938
1.000
0.750
0.857
0.818
Appendix
Table 11: Agreement between the LLM-as-Judge (GPT 5.6-Terra) and the Human-as-Judge on Pass@3.
Agent
Weights / endpoint
Scaffold
GUI
CLI
Claude Opus 4.8
claude-opus-4-8 (API)
Claude Code
✓
✓
GPT 5.5
gpt-5.5 (API)
Codex
✓
✓
Kimi-K3
kimi-k3 (API)
general VLM tool agent (ours)
✓
✓
Qwen3.5-35B-A3B
Qwen/Qwen3.5-35B-A3B (vLLM)
Qwen CUA
✓
✓
EvoCUA-32B
meituan/EvoCUA-32B-20260105 (vLLM)
official EvoCUA
✓
–
OpenCUA-72B
xlangai/OpenCUA-72B (vLLM)
official OpenCUA
✓
–
Appendix
Table 12: Evaluated agents. GUI: screenshots with mouse and keyboard actions. CLI: shell commands. Temperature follows each model’s reported default.
Setting
Value
Base model
Qwen3.5-35B-A3B
Training data
CUA-Recovery-Train, 8,577 trajectories
Hardware
64× AMD MI308X, 18 hours
Epochs / batch size / learning rate
1 / 256 / 1×10−5 , cosine
Context length / image history
32k tokens / 20 screenshots
Appendix
Table 13: Training setting of Qwen3.5-35B-A3B.
Training data
Recovery from Qwen failures
MyPCBench
Odysseys
EAR
Rubric Score
Pass@3
Pass@3
Rubric Score
Pass@3
Rubric Score
None
19.9
5.0
0.2
9.2
33.4
24.1
66.5
Successful trajectories
23.6
10.7
0.3
13.8
37.4
28.4
70.2
CUA-Recovery-Train recovery trajectories
35.2
16.1
1.9
24.5
53.2
34.9
78.2
Appendix
Table 14: Effect of the training trajectory.
Failure source
Evaluated agent
d=0
d=5
d=10
d=15
d=20
d=25
Qwen3.5-35B-A3B
Source model
25.8
21.4
23.9
18.5
14.3
15.6
ReRail-35B-A3B
41.5
37.0
38.6
34.2
30.8
29.0
Kimi-K3
Source model
13.9
7.8
10.0
11.1
18.5
15.0
ReRail-35B-A3B
25.0
15.7
17.5
22.9
14.8
15.0
EvoCUA-32B
Source model
24.7
28.3
30.3
30.4
32.7
21.8
ReRail-35B-A3B
45.3
43.9
42.4
49.5
44.6
37.7
Appendix
Table 15: Error-Awareness Rate (%) across failure sources and takeover depths. “Source model” denotes the agent that produced the failures and is evaluated on its own states.
Rubric Score / Pass@3 (%) by takeover depth
Agent
d=0
d=5
d=10
d=15
d=20
d=25
Self-takeover
API-based
GPT 5.5
65.3 / 20.1
67.7 / 24.3
61.2 / 20.9
60.8 / 20.1
68.5 / 25.5
54.9 / 18.9
Claude Opus 4.8
35.3 / 0.0
24.0 / 0.0
29.5 / 4.5
38.5 / 5.3
48.1 / 7.7
36.4 / 0.0
Kimi-K3
39.2 / 11.6
41.7 / 8.6
36.4 / 7.4
33.1 / 11.8
41.5 / 23.1
36.9 / 9.1
Appendix
Table 16: Rubric Score / Pass@3 (%) for self-takeover, on the same runs as Table 2 . Each row uses the named agent's own failures.
Figure 12: Awareness and recovery across error depth (self-takeover). Top: Error Awareness Rate; bottom: Rubric Score. Right column: ReRail-35B-A3B against Qwen3.5-35B-A3B on the same failures (green fill: gain; gray fill: loss). The shaded region marks d≥20 , where the root-cause screenshot reaches or leaves the 20-screenshot context. Pass@3 is in Table 16 .
Agent
Δ EAR
Δ Rubric Score
Δ Pass@3
GPT 5.5
+4.8
−13.6
−6.6
Claude Opus 4.8
+8.5
−11.7
−7.7
Kimi-K3
−3.5
−4.6
−14.0
EvoCUA-32B
−10.9
−8.4
−2.3
Qwen3.5-35B-A3B
+1.3
−0.7
0.0
OpenCUA-72B
+0.4
+2.1
0.0
Appendix
Table 17: Changes from d=20 to d=25 in self-takeover, in percentage points. Each change is the score at d=25 minus the score at d=20 , using Tables 2 and 16 . Each depth uses its eligible test cases.
Figure 13: ReRail-35B-A3B recovers other agents’ failures better than the agents themselves while the root cause is in context. Each column fixes one source agent. Top: Error Awareness Rate; bottom: Pass@3. Dashed: self-takeover; blue: ReRail-35B-A3B on the same failures (blue fill: higher; gray fill: lower). The shaded region marks d≥20 , where the root-cause screenshot reaches or leaves the 20-screenshot context.
Agent
Pass ∣ Aware
Pass ∣ Unaware
GPT 5.5
34.7
31.1
Claude Opus 4.8
13.9
7.6
Kimi-K3
27.0
22.8
EvoCUA-32B
9.6
7.2
Qwen3.5-35B-A3B
3.2
2.8
OpenCUA-72B
4.9
1.5
Appendix
Table 18: Recovery conditioned on error awareness in self-takeover runs. Pass denotes acceptance by the final verifier for an individual rollout. Macro Avg. gives equal weight to each agent.
Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse operations. However, such agents remain fragile in long-horizon settings, where errors accumulate and executed actions leave unintended side effects, making the ability to recover from failures critical. Yet, existing remedies remain inadequate: environment resets are infeasible in proprietary settings, action replay is prohibitively expensive over long horizons, and API-based rollback is limited to web environments, leaving model-based recovery largely unexplored. To address this gap, we present ReTrace, a scalable evaluation framework for systematically studying recovery behavior in computer-use agents. We formalize Transition-Consistent Recovery: identical downstream actions from the recovered and target states must yield identical outcomes. We operationalize this criterion through goal-oriented action replay following controlled error injection. Using ReTrace, we present the first systematic study of autonomous, model-based recovery, revealing that (i) ideal recovery can unlock model's latent task completion capabilities, (ii) current computer-use agents fail to exploit their latent recovery capabilities, and (iii) accurate model-based recovery yields measurable end-to-end gains. Together, these findings establish recovery as a fundamental capability for building reliable computer-use agents. Our code and dataset is available at https://github.com/BangProx/ReTRACE.
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.
Zihan Dong, Zhiyuan Ma, Zekun Wang +5
Georgia Institute of Technology · North Carolina State University · Lenovo AI Technology Center +3
Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence-recovery separation. Evaluations across complementary execution boundaries show that recovery is phase-dependent: before mutation, methods perform similarly without duplicate effects among capable trials; during partial mutation, naive retry, per-call idempotency, and zero-privilege journaling collapse on the evaluated composite workflows; after commit but before acknowledgment, verification and server-side idempotency substantially improve safety. These findings demonstrate that evaluating nominal completion alone masks critical, phase-dependent recovery vulnerabilities in autonomous agents.