Online reinforcement learning (RL) serves as an effective method for enhancing the capabilities of Android agents. However, guiding agents to learn through online interaction is prohibitively expensive due to the high latency of emulators and the sample inefficiency of existing RL algorithms. We identify a fundamental limitation in current approaches: the Single State Single Action paradigm, which updates the policy with one-to-one state-action pairs from online one-way rollouts without fully exploring each costly emulator state. In this paper, we propose Android Coach, a novel framework that shifts the training paradigm to Single State Multiple Actions, allowing the agent to sample and utilize multiple actions for a single online state. We enable this without additional emulator overhead by learning a critic that estimates action values. To ensure the critic serves as a reliable coach, we integrate a process reward model and introduce a group-wise advantage estimator based on the averaged critic outputs. Extensive experiments demonstrate the effectiveness and efficiency of Android Coach: it achieves 7.5% and 8.3% success rate improvements on AndroidLab and AndroidWorld over UI-TARS-1.5-7B, and attains 1.4x higher training efficiency than Single State Single Action methods PPO and GRPO at matched success rates.
Figures & tables
Figure 1: (Top): Online rollout time distribution based on the measured time on 8 parallel environments in training for 80 steps. (Bottom I): The conventional online rollout and critic training loop. The primary bottleneck is the high-latency environmental interaction, while the GUI agent action inference is relatively fast. (Bottom II): Standard agent update with Single State Single Action paradigm. Agent updates rely merely on the state-action pairs collected from the online rollout. (Bottom III): Android Coach update with Single State Multiple Actions paradigm. We fully leverage each expensive online state by generating multiple actions. The agent is then updated using this data. This approach improves training efficiency by gathering more training samples within the same online interaction cost.
Figure 2: Overview and pipeline for a training step in Android Coach . 1. Online Rollout: Policy interacts with parallel environments to collect complete trajectories. 2. Critic Update: Annotate state-action pairs with returns to train the value model. 3. Multiple sample: Resample multiple actions for each online state. 4. Actor Update: Compute action values and advantages with leave-one-out strategy, then update policy via gradient step.
Training Mode
Base Algorithm
Method
Advantage Estimation
Exploration Paradigm
Sample Efficiency
Offline
SFT
OS-Genesis Sun et al. (2025)
Reward-based
Static Data
N/A
Archer Zhou et al. (2024)
DigiQ Bai et al. (2025a)
Value-based
Static Data
N/A
Q-Learning Watkins and others (1989)
VEM Zheng et al. (2025)
Value-based
Static Data
N/A
DPO Rafailov et al. (2023)
UI-TARS-1.5 Qin et al. (2025)
Pairwise-Preference
Static Data
N/A
Hybrid
AWR Peng et al. (2019)
DigiRL Bai et al. (2025a)
Value-based
SSSA
Low
Online
GRPO Shao et al. (2024)
MobileRL Xu et al. (2025a) WebAgent-R1 Wei et al. (2025) ARPO Lu et al. (2025a)
Reward-based
SSSA
Low
Table 1: Comparison of representative GUI agent training frameworks. Android Coach is the first to achieve efficient Single State Multiple Actions (SSMA) exploration in an online setting, significantly improve sample efficiency with limited interaction costs compared to standard Single State Single Action (SSSA) approaches. ACLOO refers to Actor-Critic Leave-One-Out we proposed in our method.
Figure 3: An overview of dataset construction pipeline for process reward model training.
Models
#Params
AndroidLab SR (%)
AndroidWorld SR (%)
QD
OP
Average
Easy
Mid
Hard
Average
Proprietary Models
Gemini-Pro-1.5 (SoM) Team et al. (2024)
-
-
-
16.7
-
-
-
22.8
GPT-4o (SoM) OpenAI et al. (2024)
-
-
-
31.2
-
-
-
34.5
Claude-Sonnet-4 (SoM) Anthropic (2025)
-
-
-
40.6
-
-
-
41.0
UI-Genie-Agent Xiao et al. (2025)
72B
-
-
41.2
-
-
-
-
Table 2: Success rates of proprietary and open-source models on AndroidWorld and AndroidLab for mobile GUI interaction tasks. QD is the abbreviation for the query detect type, and OP is the abbreviation for the operation type. Standard deviations are reported in gray subscripts for all models except proprietary ones.
Figure 4: Training efficiency analysis of different methods over training time. We report the relationship between training time and (a) interaction times, (b) sampled actions number, and (c) success rate. The GRPO group size and sample number of Android Coach are both 4. Data is collected on AndroidWorld over UI-TARS-1.5-7B.
Figure 5: The effect of the ACLOO advantage estimation. (a) The training reward curves. (b) The final success rate gain on AndroidLab.
Single State N Actions
Training Time
AndroidLab SR(%)
AndroidWorld SR(%)
number of N
1
1.00x
34.8 ±0.7
36.8 ±1.3
2
1.22x
35.5 ±1.2
38.5 ±2.2
4
1.62x
37.0 ±0.7
39.1 ±0.5
8
2.18x
37.0 ±0.0
39.4 ±2.0
Table 3: Analysis of action rollout times. We report total training time and success rates on AndroidLab and AndroidWorld across different numbers of samples.
Figure 6: Critic loss with different initialization strategies during joint policy optimization.
Usage of PRM
SR
ROR
Average Steps
AndroidLab
wo/ PRM
37.0
83.2
12.7
w/ PRM
39.4 (+2.4)
89.8 (+6.6)
11.9 (-0.8)
AndroidWorld
wo/ PRM
39.1
80.0
13.0
w/ PRM
41.1 (+2.0)
85.1 (+5.1)
12.5 (-0.5)
Table 4: Analysis of PRM usage. We report the success rate (SR), reasonable operation ratio (ROR) of steps and average steps on AndroidLab for agent trained with and without process model.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Process Reward Model training data format.
Component
Hyperparameter
Value
Data
Max Prompt Length
32768
Data
Max Response Length
512
Data
Train Batch Size
8
Actor/Policy
Strategy (Parallelism)
FSDP2
Actor/Policy
PPO Micro Batch Size/GPU
1
Actor/Policy
Learning Rate (LR)
1e-6
Appendix
Table 5: Main hyperparameters in Android Coach .
Action
Definition
Click(x, y)
Clicks at coordinates (x, y).
Scroll(x1, y1, x2, y2)
Scrolls from (x1, y1) to (x2, y2).
Drag(x1, y1, x2, y2)
Drags from (x1, y1) to (x2, y2).
Type(content)
Types the specified content.
Wait()
Pauses for a brief moment.
Finished(content)
Marks the task as complete.
Appendix
Table 6: Operation action space for GUI agent in Android Coach .
Figure 8: Prompt for Android Coach . This prompt is consistent with the official prompt provided by UI-TARS-1.5-7B.
Figure 9: Prompt for outcome verifier.
Figure 10: Prompt for Process Reward Model.
Figure 11: Prompt for GPT-4o to generate reason for PRM Training.