Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents

Yuxiang Chai; Quyu Kong; Panrong Tong; Steven Hoi; Yue Wang

cs.AIOct 29, 2026

Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents

Authors: Yuxiang Chai, Quyu Kong, Panrong Tong, Steven Hoi, Yue Wang

Organizations: Alibaba Token Hub, Alibaba Group · MMLab, The Chinese University of Hong Kong

Abstract

Trajectory-level GRPO is a common approach for training multi-step GUI agents, but repeatedly rolling out complete trajectories from the task root can be inefficient: task-level rewards provide coarse supervision over which intermediate decisions require improvement, while already-mastered prefixes are repeatedly re-executed before reaching difficult states. We introduce a two-stage adaptive training framework that jointly addresses where to learn and how far to roll out. We first construct reusable intermediate checkpoints from successful expert trajectories and select informative pivots using action disagreement, which measures uncertainty directly from sampled GUI actions. Stage I performs one-step RL at these pivots to improve uncertain local decisions. Stage II then progressively expands policy responsibility through adaptive Local, Suffix, and Root training according to task progress. Recoverable checkpoints enable repeated training from intermediate states without replaying the preceding trajectory. Across MobileGym and CUA-Gym-Hub, our method reduces environment interaction by 56-61% relative to Vanilla GRPO while improving task performance, demonstrating that jointly allocating training across states and rollout ranges can substantially improve GUI-agent RL efficiency.

Figures & tables

Figure 1: Overview of standard trajectory-level GRPO and our efficient adaptive training framework. Vanilla GRPO repeatedly traverses mastered prefixes and provides sparse trajectory-level supervision. Our method instead combines reusable checkpoints, pivot selection, and adaptive training to reduce redundant interaction and focus learning on pivotal steps.

Figure 2: Training pipeline of our method. We construct reusable checkpoints from successful expert trajectories, select uncertain intermediate states using action disagreement, and perform a two-stage training procedure consisting of one-step local warm-up followed by adaptive hybrid training.

Environment	Checkpoint Restore	10-Step Replay	Time Reduction
MobileGym	2.97 s	15.7 s	81.1%
CUA-Gym-Hub	3.21 s	11.3 s	71.6%

Table 1: Checkpoint restoration latency compared with replaying a 10-step trajectory prefix.

Figure 3: One-step pivot training with different state-selection strategies on MobileGym and CUA-Gym-Hub. All methods use the same local RL procedure and differ only in the selected intermediate states. Solid and dashed curves denote Avg@4 and Pass@4, respectively; gray lines show the common base performance.

Method	MobileGym			CUA-Gym-Hub
Method	Samples	Train	Held-Out	Samples	Train	Held-Out
All Pivots	38.4K	61.14 / 74.70	50.00 / 63.28	16.4K	63.75 / 68.75	47.08 / 60.00
Reward Variance	8.2K	56.02 / 72.29	45.70 / 60.16	14.3K	67.44 / 73.75	47.50 / 57.87
Action Disagreement (Ours)	8.2K	58.73 / 74.70	48.83 / 62.50	15.4K	68.81 / 75.00	52.08 / 61.67

Table 2: Comparison of pivot-selection strategies on MobileGym and CUA-Gym-Hub. Each metric cell reports Avg@4 / Pass@4 at the checkpoint with the highest Train Avg@4.

Panel of Figure 4 — Figure 4: Full-training effectiveness and efficiency on MobileGym (top) and CUA-Gym-Hub (bottom). For each benchmark, the left two panels report Train and Held-out performance against cumulative training samples, while the right two report performance against cumulative rollout steps. Solid and dashed curves denote Avg@4 and Pass@4, respectively, and horizontal gray lines indicate base-policy performance.

Method	MobileGym		CUA-Gym-Hub
	Train	Held-Out	Train	Held-Out
Base	48.80 / 62.65	40.82 / 57.03	52.81 / 61.25	47.08 / 58.33
SFT	81.93 / 90.36	48.24 / 65.63	70.00 / 73.75	50.83 / 58.33
Base $\rightarrow$ Vanilla	56.33 / 71.08	47.66 / 62.50	61.88 / 68.75	45.83 / 55.00
Base $\rightarrow$ Hybrid	56.93 / 73.49	47.85 / 64.84	65.31 / 75.00	55.83 / 65.00
Action $\rightarrow$ Vanilla	62.65 / 74.69	49.61 / 62.50	65.63 / 68.75	53.33 / 63.33

Table 3: Summary of full-training performance on MobileGym and CUA-Gym-Hub. For each method, we select the checkpoint with the highest Train Avg@4 and report its corresponding Train and Held-Out performance. Each cell reports Avg@4 / Pass@4.

Figure 5: Evolution of training allocation in adaptive hybrid training on MobileGym and CUA-Gym-Hub. The four panels show how the proportions of Local, Root, and Suffix training change over cumulative training samples for Action $\rightarrow$ Hybrid and Base $\rightarrow$ Hybrid.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Action Counts	$\hat{D}(c_{t})$	Action Counts	$\hat{D}(c_{t})$
$8$	$0.000$	$5$ – $3$	$0.536$
$7$ – $1$	$0.250$	$4$ – $4$	$0.571$
$6$ – $2$	$0.429$	$5$ – $2$ – $1$	$0.607$
$6$ – $1$ – $1$	$0.464$	$4$ – $3$ – $1$	$0.679$
$4$ – $2$ – $2$	$0.714$	$3$ – $3$ – $2$	$0.750$
$2$ – $2$ – $2$ – $2$	$0.857$	$1$ – $1$ – $1$ – $1$ – $1$ – $1$ – $1$ – $1$	$1.000$

Appendix

Table 4: Representative action-count patterns and their empirical disagreement scores for

N=8

policy samples. We use

\tau_{\mathrm{dis}}=0.45

RL for GUI Agents

Reinforcement learning for GUI agents trains screen-based AI to complete multi-step tasks across mobile, web, and desktop interfaces.

See all 52 papers

Adaptive-Group Process Rewards for GUI Agents
Oct 13, 2026Zirui Shang, Yang Liu, Xin Shu +3RL for GUI Agents Process Reward Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University +1
GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis
Aug 24, 2026Long Zhang, Yuhan Chen, Chaoran Zhang +7RL for GUI Agents Reward Design for RL
Wuhan University · Xiaomi Inc.
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University

Oct 13, 2026J/K move · Enter open · S save

Adaptive-Group Process Rewards for GUI Agents

Zirui Shang, Yang Liu, Xin Shu, Zhi Gao+2

Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University+1

RL for GUI Agents Process Reward Models Step-Level Credit Assignment in RL Sparse-Reward RL

Open paper AskPDF arXiv

Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We further augment GRPO's outcome advantage with the proposed process rewards to provide finer-grained guidance for GUI agent training. Experiments on process reward benchmarks and downstream RL training demonstrate the effectiveness of the proposed framework.

Abstract

RL for GUI Agents

Explore similar work

Adaptive-Group Process Rewards for GUI Agents

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

GUI-C2^22: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning