Learn from Move: the Next Step for GUI Agents

Jiahao Lu; Yize Shen; Chuanyuan Tan; Mianqiu Huang; Shicheng Fang; Xinchi Chen; Yining Zheng; Yuxin Wang

cs.LGOct 12, 2026

Learn from Move: the Next Step for GUI Agents

Authors: Jiahao Lu, Yize Shen, Chuanyuan Tan, Mianqiu Huang, Shicheng Fang, Xinchi Chen, Yining Zheng, Yuxin Wang,

Organizations: Fudan University · Shanghai Innovation Institute · Soochow University

Abstract

Existing benchmarks for GUI agents primarily focus on increasing task difficulty by extending task horizons. Besides, enabling agents to explore unfamiliar environments and successfully complete tasks within them is also a critical yet challenging problem. We introduce LatentGUIWorld, a benchmark of six tasks in three matched pairs organized by the evidence needed for control: $I_0$ uses prior knowledge and available observations, $I_1$ requires interaction that directly reveals hidden semantics, and $I_2$ identifies dynamics from action-response relationships. Frontier models struggle especially on $I_2$ tasks. We introduce MOVE, which separates movement from button events and exposes intermediate visual feedback, together with demonstrations of iterative exploration and control. Ablations show that both action granularity and multi-step demonstrations improve performance. After SFT, LatentLearner-9B achieves 50.0% overall success on LatentGUIWorld, comparable to several frontier models. Joint RL further raises egocentric Drag and Ten-Choice success to 92.0% and 75.3%. Behavioral analysis reveals multi-step exploration. In egocentric Drag, RL increases Pearson correlation between error-normalized commands and ideal gain compensation from 0.610 to 0.803. Together, these behaviors show that models learn to explore through interaction and adapt their control to environment dynamics. The same format improves GUI grounding after RL, reaching 67.37% mean accuracy across five benchmarks versus 66.72% for conventional RL.

Figures & tables

Panel of Figure 1 — Figure 1: Measure identification; expose evidence before commitment. LatentGUIWorld spans three task pairs and evidence requirements $I_{0}$ – $I_{2}$ . The rotation example relates an executed move to its visual consequence to identify an unknown response gain and adapt control. The Move interface returns an observation for a new decision after each primitive. Native frames show distinct interaction states; overlays follow replayed movements. Gain inference and the next action are schematic illustrations of the control process.

Panel of Figure 2 — Figure 2: Task horizon and the evidence required for environment identification are distinct dimensions. The six LatentGUIWorld tasks form three matched pairs (connecting lines). Named benchmarks provide qualitative examples of selected known-interface workflows, grounding, or known-mapping manipulation [ xie2024osworld , yuan2026osworld2 , koh2024visualwebarena , rawles2024androidworld , bonatti2024windowsagentarena , li2025screenspotpro , nayak2025uivision , hu2025showuipi ] . Placement characterizes those selected tasks; bands and within-band positions are schematic.

Level	Before intervention	Minimum identifying evidence	Dependency
$\mathbf{I_{0}}$	$K$ and $D_{t}$ suffice.	Prior knowledge and observed context support control; multiple execution steps and screen updates are allowed.	$\boxed{D_{t}=(H_{t},O_{t})\rightarrow A_{t}^{*}}$
$\mathbf{I_{1}}$	$K$ and $D_{t}$ do not suffice.	The probe outcome directly reveals a semantic latent: use what the action revealed .	$\boxed{D_{t}\xrightarrow{A_{t}^{\mathrm{probe}}}O_{t+1}\rightarrow z_{\mathrm{sem}}\rightarrow A_{t+1}^{*}}$
$\mathbf{I_{2}}$	$K$ , $D_{t}$ , and a probe outcome alone do not identify the dynamics.	The self-action and induced transition jointly identify a dynamics latent: use what the action caused .	$\boxed{(O_{t},A_{t}^{\mathrm{probe}},O_{t+1})\rightarrow z_{\mathrm{dyn}}\rightarrow A_{t+1}^{*}}$

Table 1: Evidence taxonomy for environment identification.

D_{t}=(H_{t},O_{t})

is the observed pre-intervention context,

A_{t}^{\mathrm{probe}}

is the self-executed probe, and

A_{t+1}^{*}

is the subsequent task-directed action. Sufficiency is evaluated with prior domain knowledge

K

, excluding the current episode’s hidden latent. A task is assigned the lowest level sufficient for reliable success under a fixed benchmark contract.

Panel of Figure 3 — Figure 3: Shared first frames, different interaction mechanisms. Action headers apply to both variants; coordinates may differ. Each numbered primitive executes separately, with intermediate frames omitted. Middle and right panels follow the actions above them. Arrows mark motion; dashed marks indicate pre-MOVE positions (Ten-Choice: action 1; Drag: action 3) or Rotation’s fixed inner boundary. Scripted replay trajectories illustrate each control mechanism. Drag uses a matched-reset pair.

GUI agent	Drag			Ten-Choice			Rotation			$I_{2}$ Avg.	Overall
	Exocentric ( $I_{0}$ )	Egocentric ( $I_{2}$ )	\cellcolor mosiblue!7 $\Delta_{\mathrm{pair}}$	Exocentric ( $I_{1}$ )	Egocentric ( $I_{2}$ )	\cellcolor mosiblue!7 $\Delta_{\mathrm{pair}}$	Inner-ring ( $I_{2}$ )	Outer-ring ( $I_{2}$ )	\cellcolor mosiblue!7 $\Delta_{\mathrm{pair}}$

Table 2: Success rate (%) across six environments in three matched pairs.

\Delta_{\mathrm{pair}}

is the second variant minus the first in each pair, in percentage points, computed before rounding; both Rotation variants are

I_{2}

I_{2}

Avg. averages the four

I_{2}

tasks; Overall averages all six. Rotation uses strict first-release scoring. Each reported row uses one model checkpoint across environments. SFT checkpoints are selected on the separate validation set; all scores shown are test results. Bold and underline mark the best and second-best results per column, excluding pairwise differences.

GUI Agent Benchmarks

See all 21 papers

Apr 13, 2026

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present ClawGUI, an open-source framework addressing these three gaps within a single harness. ClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. ClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8% reproduction against official baselines. ClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, ClawGUI-2B achieves 17.1% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0%.

Fei Tang, Zhiqiong Lu, Boxuan Zhang +4

Zhejiang University

GUI Agents Mobile GUI Automation RL for GUI Agents AI Agent Evaluation AI Agent Benchmarks

Ask PDF

Aug 27, 2026

WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods. Code is available at https://github.com/genalyu/WM-R1 .

Yu Han, Tianwen Qian

School of Computer Science and Technology, East China Normal University

RL for GUI Agents Model-Based RL

Ask PDF

Oct 14, 2026

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

Online reinforcement learning (RL) with verifiable rewards has driven rapid progress in mathematics, coding, and tool use, but scaling this success to GUI agents remains challenging due to the high cost of interactive environments and the scarcity of realistic training tasks. We introduce MAI-Gym, an open all-in-one platform that unifies GUI agent evaluation, data collection, and online RL training across mobile, computer, and web environments. Through a shared rollout pipeline, MAI-Gym ensures consistent agent behavior across evaluation and training, supports ten established benchmarks, and scales to 20,000 parallel environment instances. Beyond scalable infrastructure, MAI-Gym enables automated synthesis of diverse GUI applications and challenging tasks, including resettable mocked apps and coherent cross-app worlds with execution-validated task verifiers. Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified. Importantly, RL in a mocked train-ticketing app transfers to its real-world counterpart, improving task success on a phys- ical phone from 28.8% to 48.1%. We release the framework, synthesis skills, mocked apps, 2,000 verified tasks, and the model checkpoint to support repro- ducible research and further advances in scalable GUI agent training.

Panrong Tong, Quyu Kong, Pengxiang Li +12

Alibaba Token Hub, Alibaba Group

RL for GUI Agents Mobile GUI Agents AI Agent Evaluation AI Agent Benchmarks Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

GUI Agent Benchmarks

Explore similar work

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments