Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay

Byeonghun Bang; Sangwoo Park; Jinheon Baek; Kangsan Kim; Youngwan Lee; Sung Ju Hwang

cs.AIOct 20, 2026

Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay

Authors: Byeonghun Bang, Sangwoo Park, Jinheon Baek, Kangsan Kim, Youngwan Lee, Sung Ju Hwang

Organizations: KAIST · ETRI

Abstract

Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse operations. However, such agents remain fragile in long-horizon settings, where errors accumulate and executed actions leave unintended side effects, making the ability to recover from failures critical. Yet, existing remedies remain inadequate: environment resets are infeasible in proprietary settings, action replay is prohibitively expensive over long horizons, and API-based rollback is limited to web environments, leaving model-based recovery largely unexplored. To address this gap, we present ReTrace, a scalable evaluation framework for systematically studying recovery behavior in computer-use agents. We formalize Transition-Consistent Recovery: identical downstream actions from the recovered and target states must yield identical outcomes. We operationalize this criterion through goal-oriented action replay following controlled error injection. Using ReTrace, we present the first systematic study of autonomous, model-based recovery, revealing that (i) ideal recovery can unlock model's latent task completion capabilities, (ii) current computer-use agents fail to exploit their latent recovery capabilities, and (iii) accurate model-based recovery yields measurable end-to-end gains. Together, these findings establish recovery as a fundamental capability for building reliable computer-use agents. Our code and dataset is available at https://github.com/BangProx/ReTRACE.

Figures & tables

Figure 1: Comparison of error-handling strategies in computer-use agents. (A) Reflection proceeds from the erroneous state, (B) rollback requires often-infeasible environment resets, while (C) recovery restores a prior intact state and resumes the task.

Figure 2: Overview of ReTrace . (A) On-Policy Trajectory Rollout collects successful trajectories and replays each action sequence $N$ times to build a transition-consistent dataset. (B) Trajectory Drift via Perturbation selects a recovery target and perturbs the agent into an erroneous state. (C) Goal-Oriented Action Replay replays the remaining actions from the recovered state to test whether the agent reproduces the original task outcome.

Figure 3: Success@

\mathrm{k}

across different recovery steps

\mathrm{R}

under ideal recovery, using EvoCUA-32B as the forward policy on OSWorld.

Model	Setting	SR (%) $\uparrow$
Sonnet-4.6	Baseline	$70.27\pm 0.70$
Sonnet-4.6	w/ Recovery	$71.81\pm 1.51$
Qwen3.5-9B	Baseline	$38.76\pm 0.64$
Qwen3.5-9B	w/ Recovery	$41.56\pm 4.14$
UI-TARS-1.5-7B	Baseline	$17.25\pm 1.55$
UI-TARS-1.5-7B	w/ Recovery	$15.74\pm 0.65$

Table 3: End-to-end task success rate (SR) on OSWorld across different models with the Recovery Target Contextualization strategy, over two independent runs.

Error Type	Instruction	Target State	Restored State	Action	Problem
Spurious Perception	Align the text of the first textbox on slide 3 to the right , on slide 4 to the center, and on slide 5 to the left.			[click] on the hovering alignment button to align slide 3 the text to the right.	Fixates on to the hovering UI element in recovered state , skipping [drag] on the text before clicking the alignment button.
State Tracking Failure	Change the font to "Times New Roman" throughout the text.			[scroll down] to find Times New Roman, since font selection interface is opened.	Hallucinates the restored state due to failed state synchronization with its previous states.

Table 4: Qualitative analysis of erroneous agent behaviors induced by imperfect state restoration.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Web Tasks (%)	Non-Web Tasks (%)
OSWorld	12.66	87.34
WindowsAgentArena	21.28	78.72

Appendix

Table 5: Ratio of web and non-web tasks in the OSWorld and WindowsAgentArena benchmarks.

	Dataset Filtering Step	Claude-Sonnet-4.6	GPT-5.4-CUA	QWEN-3.5-9B	UI-Tars-1.5-7B	EvoCUA-32B
OsWorld	# of traj. after formatting	2733	2749	790	1277	1259
	# of traj. after transition-consistent filtering	1921	1912 (600)	574	227	924
	# of traj. after perturbation	1335	334	397	133	314
	# of traj. after final set construction	120	120	120	60	120
Windows AgentArena	# of traj. after formatting	919	867	211	N/A	280
	# of traj. after transition-consistent filtering	779	407	153	N/A	201

Appendix

Table 6: Dataset statistics after each filtering step for the short-horizon settings.

	Domain Composition (%)	Original	Claude-Sonnet-4.6	GPT-5.4-CUA	QWEN-3.5-9B	UI-Tars-1.5-7B	EvoCUA-32B
OsWorld	chrome	6.64	4.17	2.50	6.67	25.00	0.00
	gimp	8.23	0.83	6.67	8.33	1.67	6.67
	libreoffice_calc	14.87	26.67	14.17	7.50	0.00	17.50
	libreoffice_impress	14.87	29.17	15.83	21.67	13.33	38.33
	libreoffice_writer	7.28	5.83	8.33	5.83	13.33	14.17
	multi_apps	23.73	16.67	35.00	27.50	3.33	9.17

Appendix

Table 7: Domain Composition ratio per recovery evaluation dataset (short horizon).

Figure 9: Expanded Results on WindowsAgentArena from Figure 5 .

Model	Max 2	Max 4	Max 5	Max 10
Claude-Sonnet-4.6	26.98	13.34	8.76	6.37
GPT-5.4-CUA	17.85	6.30	5.81	5.15
Qwen3.5-9B	12.04	7.29	6.12	5.29
EvoCUA-32B	0.86	2.00	0.86	1.44
UI-TARS-1.5-7B	7.26	7.81	12.31	7.82

Appendix

Table 8: Implicit recovery success rate (%) across diverse max recovery step budgets in OSWorld benchmark. We report the mean success rate across three different trials from three independently sampled evaluation sets.

	Temperature	Max Tokens	Short Horizon	Long Horizon
EvoCUA -32B	1.0	32768	0.00	7.94
	1.0	16384	1.75	4.76
	0.0	32768	3.51	20.63
	0.0	16384	4.39	6.35
UI-Tars- 1.5-7B	1.0	1000	31.67	25.00
	1.0	500	26.67	26.67

Appendix

Table 9: Recovery Success Rate (%) across two different models with different hyper parameters using Recovery Target Contextualization in OSWorld benchmark.

	Temperature	Max Tokens	Top_P	Success Rate (%)
EvoCUA -32B	0.7	32768	0.95	18.00
	1.0	32768	0.95	14.00
	1.0	16384	0.95	20.00
	0.0	32768	0.95	20.00
	0.0	16384	0.95	18.00

Appendix

Table 10: Recovery Success Rate (%) of EvoCUA-32B with different hyper parameters using Recovery Target Contextualization in WindowsAgentArena benchmark.

Figure 10: Three-run average recovery success rates for each recovery variant across three randomly sampled recovery evaluation sets in ReTrace .

Oracle	Unaffected Rate (%)
ReTrace (All)	86.0 (%)

Appendix

Table 11: Analysis on the rate of oracle evaluations unaffected by stochastic environmental dynamics.

Oracle	Alignment Rate (%)
$\mathcal{R}_{\mathrm{TC}}$	38.8 (%)
$\mathcal{R}_{\mathrm{O}}$	61.2 (%)
ReTrace	81.2 (%)

Appendix

Table 12: Analysis on the alignment rate of oracle judgments with human annotations using samples under

\mathcal{R}_{\mathrm{O}}

and

\mathcal{R}_{\mathrm{TC}}

disagreement.

Threshold	Alignment Rate (%)
0.99	56.9 (%)
0.95	33.9 (%)
0.91	26.6 (%)

Appendix

Table 13: Analysis of the Effect of the SSIM Threshold on the Alignment Between

\mathcal{R}_{\mathrm{O}}

and Human Judgments.

Error Type	Instruction	Target State	Restored State	Action	Problem
Spurious Perception	Align the text of the first textbox on slide 3 to the right , on slide 4 to the center, and on slide 5 to the left.			[click] on the hovering alignment button to align slide 3 the text to the right.	Fixates on to the hovering UI element in recovered state , skipping [drag] on the text before clicking the alignment button.
Spurious Perception	Please help me change the color theme of VS Code to Visual Studio Dark.			[click] Activities element in the top left corner to launch vscode application	Fixates on to the UI element appearing in the restored state, unable to correctly click vscode application after restoration.
Spurious Perception	Could you help me remove the dock on the left side of the screen in the GIMP?			[click] on windows option in the right window to explore options for removing the dock	Fixates on to the new windows option irrelevant to removing the dock option , fails to generate correct actions due to incorrect state restoration.
State Tracking Failure	Change the font to "Times New Roman" throughout the text.			[scroll down] to find Times New Roman, since font selection interface is opened.	Hallucinates the restored state due to failed state synchronization with its previous states.
State Tracking Failure	Help me freeze the range A1:B1 on this sheet to keep the headers always visible.			[Drag] from column A1 to B1, in order to accurately select the area we want to freeze	Hallucinates that the restored state allows clicking the range A1 to B1, when in fact, cannot be dragged , failing to synchronize with the actual prior state.
State Tracking Failure	Please modify VS Code’s settings to disable error reporting for Python missing imports.			[hotkey(key=’ctrl shift p’)] I checked the results, but there’s nothing in the search box. I’m aware of a better solution.	Hallucinates that the restored state has blank results, just like the recovery target state , leading to completely novel behavior.

Appendix

Table 14: Additional Qualitative analysis of erroneous agent behaviors induced by imperfect state restoration.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents
Oct 20, 2026Xu Hu, Haoming Li, Yuxiang Wang +11Computer-Use Agents Computer-Use Agent Benchmarks
University of Texas at Dallas · Independent Researcher · University of Manchester +3
Human-Guided Harm Recovery for Computer Use Agents
Apr 20, 2026Christy Li, Sky CH-Wang, Andi Peng +1Computer-Use Agents AI Agent Security Benchmarks
MIT CSAIL · Abridge · humans&
Recoverability as a System Primitive for Long-Horizon AI Agents
Sep 12, 2026Zhihui Zhang, Wei LiuAI Agent Reliability Runtime Enforcement for AI Agents
Independent researcher · Beijing Information Science and Technology University

Oct 20, 2026J/K move · Enter open · S save

CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents

Xu Hu, Haoming Li, Yuxiang Wang, Jinxin Yang+10

University of Texas at Dallas · Independent Researcher · University of Manchester+3

Computer-Use Agents Computer-Use Agent Benchmarks AI Agent Reliability Agent Failure Analysis Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state and cause cascading failures in subsequent actions. Reliable error detection and recovery have become critical bottlenecks for building robust CUAs. However, existing benchmarks focus primarily on end-to-end task performance, providing limited insight into whether agents can detect and recover from errors during execution. The few benchmarks that evaluate recovery rely largely on manually injected errors, which may not reflect the types or horizons of failures induced by real agent policies. We introduce CUA-Recovery, a benchmark comprising test and training suites across multiple desktop applications. We construct it using ReRail, a framework for synthesizing long-horizon tasks and trajectories in a persistent desktop environment. ReRail composes MyPCBench tasks into new long-horizon workflows with executable ground truth, collects verifier-confirmed failures from multiple policies, and turns them into verified recovery trajectories. CUA-Recovery comprises a human-verified test suite of 2,250 erroneous states over 100 tasks and a training suite of 8.6K trajectories over 377 tasks. Evaluation on the test suite shows that agents struggle both to recognize inherited errors and to complete the task. Across six agents and six takeover depths, Error Awareness Rate (EAR) reaches at most 44.0% and Pass@3 at most 25.5%. After fine-tuning Qwen3.5-35B-A3B on the training suite, its average EAR rises from 19.9% to 35.2% on held-out recovery states, and its Rubric Score goes from 5.0% to 16.1%. It also gains 15.3 points on MyPCBench and 10.8 points on Odysseys for Pass@3, showing that the training trajectories help both with recovery and with completing tasks from a clean start. Our code is available at https://github.com/xu-hu-2002/CUA-Recovery.

Abstract

Computer-Use Agents

Explore similar work

CUA-Recovery: Benchmarking Error Recovery in Long-Horizon Computer-Use Agents

Human-Guided Harm Recovery for Computer Use Agents

Recoverability as a System Primitive for Long-Horizon AI Agents