UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

Zhengxi Lu; Fei Tang; Guangyi Liu; Kaitao Song; Xu Tan; Jin Ma; Wenqi Zhang; Weiming Lu

cs.LGApr 15, 2026

UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

Authors: Zhengxi Lu, Fei Tang, Guangyi Liu, Kaitao Song, Xu Tan, Jin Ma, Wenqi Zhang, Weiming Lu,

Organizations: Zhejiang University · Apple · Tencent

Abstract

MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.

Figures & tables

Figure 1: Left : Performance on dynamic GUI benchmarks. Right : Task distribution of these benchmarks.

Figure 2: MemGUI-Bench Inference Case. Our method successfully completes the task by invoking Copilot Model, whereas other models fail due to memory degradation, progress confusion, and math hallucinations.

Figure 3: Overview of TIPO Pipeline. Policy model jointly learns tool invocations and multi-turn action prediction.

Figure 4: Training Dataset Curation Pipeline.

		#Cross App								Difficulty Level
Models	Type	1 App		2 App		3 App		4 App		Easy		Med		Hard		Avg
		p@1	p@3	p@1	p@3	p@1	p@3	p@1	p@3	p@1	p@3	p@1	p@3	p@1	p@3	p@1	p@3
CogAgent	AO	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Qwen2.5VL-7B*	AT	14.3	17.9	1.8	1.8	0.0	0.0	0.0	0.0	10.4	12.5	0.0	0.0	0.0	0.0	3.9	4.7
UI-Venus-7B	AT	21.4	28.6	1.8	1.8	0.0	2.9	0.0	0.0	14.6	20.8	0.0	0.0	0.0	0.0	5.5	7.8
UI-S1-7B*	MC	17.9	21.4	3.6	3.6	0.0	0.0	0.0	0.0	12.5	14.6	2.6	2.6	0.0	0.0	5.5	6.2

Table 1: Results on MemGUI-Bench. p@k denotes pass@k. Results marked with * are evaluated by ourselves with tool usage, while the remaining results are reported from the benchmark paper, where pass@3 is tested with long-term memory. The best performance in each column is highlighted in bold , and the second best is underlined . AO denotes A ction- O nly rollout without history. AT denotes A ction- T hought history management. MC denotes M ulti-turn C ontext. TC denotes our T ool-Integrated multi-turn C ontext rollout. MW denotes M ulti-agent W orkflow, with Gemini-2.5 Pro ( Comanici et al., 2025 ) as planning agents.

Closed-source Models
Models	AC-High			GUI Odyssey			AC-Real		Wob	AW	Avg
	TM	GR	SR	TM	GR	SR	PG	TSR	SR	SR	SR
Claude-CU (SoM) ( Anthropic, 2024 )	63.7	0.0	12.5	60.9	0.0	3.1	–	–	–	27.9	–
GPT-4o (SoM) ( Hurst et al., 2024 )	66.3	0.0	20.8	34.3	0.0	3.3	–	–	62.0	34.5	–
Open-source Models
Qwen2VL-2B ( Wang et al., 2024b )	42.3	18.7	13.6	24.4	12.2	12.6	2.0	1.0	20.8	0.0	7.3

Table 2: Results on Other GUI Benchmarks. * shows the result with tool calling, and UI-Copilot-7B achieves 32.2% accuracy without tool usage. Wob denotes MiniWob++ and AW denotes AndroidWorld. Average SR is computed as average of AC-Real-TSR, Wob-SR and AW-SR. The highest value is in bold , the second is underlined .

Figure 5: Training Dynamics of UI-Copilot-7B on our selected AndroidWorld-Verified (60 tasks), MemGUI-Bench-Verified (55 tasks) subsets, Tool-call-Test (1000 tasks from

\mathcal{D}_{0}

) and AC-Real-Test (1536 tasks).

Table 8

Figure 8: Case Study. UI-Copilot-7B successfully completes a math-related task from AndroidWorld (top) and a memory-related task from MemGUI-Bench (bottom).

Figure 9: Error Type Analysis with Tool Usage. Errors are categorized into Memory Degradation, Progress Confusion, Math Hallucination and Other Fault.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Action Type	Description
click	Tap a specific coordinate $(x,y)$ on the screen.
long_press	Press and hold at $(x,y)$ for a specified duration.
swipe	Perform a swipe gesture from $(x_{1},y_{1})$ to $(x_{2},y_{2})$ .
answer	Output a textual answer to the task.
type	Enter text into the currently focused input field.
system_button	Trigger a system-level button (e.g., Home, Back).

Appendix

Table 3: Action space in AndroidWorld automation.

Figure 10: Interaction example for UI-Copilot-7B.

Figure 17: Golden steps comparison between AndroidWorld and MemGUI-Bench.

Closed-source Models
Models	ScreenSpot			AC-Low			AC-High			GUI Odyssey
	V2	Pro	Avg	TM	GR	SR	TM	GR	SR	TM	GR	SR
GPT-4o ( Hurst et al., 2024 )	18.3	0.8	9.6	74.3	0.0	19.4	66.3	0.0	20.8	34.3	0.0	3.3
Claude-CU ( Anthropic, 2024 )	83.0	17.1	50.1	74.3	0.0	19.4	63.7	0.0	12.5	60.9	0.0	3.1
Open-source Models
OS-Atlas-4B ( Wu et al., 2024 )	71.9	3.7	37.8	91.9	83.8	80.6	49.0	49.5	22.8	49.6	34.6	20.3

Appendix

Table 4: Model Comparison on Single-turn Benchmarks.

Figure 18: Pass@k validation on AndroidWorld-Verified (60 tasks) and MemGUI-Bench-Verified (55 tasks).

Panel of Figure 19 — Figure 19: Tool Type Distribution on AndroidWorld-Verified, MemGUI-Bench-Verified, and MiniWob++.

Figure 20: Error Type Analysis of Tool Invocation.

Figure 21: Successful Cases : Vanilla Rollout without tool usage.

Figure 22: Successful Cases : Tool-Integrated Rollout with Math Calculator.

Figure 23: Successful Cases : Tool-Integrated Rollout with Memory Retriever.

Figure 24: Bad Cases : Reasoning Hallucination.

Figure 25: Bad Cases : Progress Confusion.

Figure 26: Bad Cases : Action Inconsistency.

Panel of Figure 27 — Figure 27: Prompt for UI-Copilot-7B.

Panel of Figure 28 — Figure 28: Prompt for Retriever.

Panel of Figure 29 — Figure 29: Prompt for Calculator.

Long-Horizon LLM Agents

Long-horizon large language model agents plan, use tools, and preserve task state across extended, multi-step interactions.

See all 446 papers

Executable Agentic Memory for GUI Agent
May 12, 2026Zerui Qin, Sheng Yue, Xingyuan Hua +2GUI Agents LLM Agent Memory
1Tsinghua University, China · 2Sun Yat-sen University, China
MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
May 18, 2026Ziyun Zeng, Hang Hua, Bocheng Zou +3Multimodal Memory GUI Agents
University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison
Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
Jul 17, 2026Zedong Yu, Qianxing Li, Zhi Gao +8Computer-Use Agent Benchmarks GUI Agents
Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China · China University of Geosciences (Beijing), Beijing, China +1

May 12, 2026J/K move · Enter open · S save

Executable Agentic Memory for GUI Agent

Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu+1

1Tsinghua University, China · 2Sun Yat-sen University, China

GUI Agents LLM Agent Memory Mobile GUI Agents Graph-Based Agent Memory Long-Horizon LLM Agents GUI Automation

Open paper AskPDF arXiv

Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to

19.6\%

on AndroidWorld, while reducing token costs

6\times

relative to GPT-4o. With a

2.8

s average latency, EAM enables reliable, quick, and long-horizon GUI automation.

Abstract

Long-Horizon LLM Agents

Explore similar work

Executable Agentic Memory for GUI Agent

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents