How Complete Must an Environment Be to Train a Capable GUI Agent?
Organizations: Shanghai Jiao Tong University · OPPO Research Institute · Shanghai Innovation Institute
Abstract
Graphical user interface (GUI) agents are widely trained in executable GUI environments. To lower cost, build environments in batches, and reduce manual effort, each work trades away part of the environment through its construction choices. We describe these choices along three dimensions: reward coverage, the goal-reaching trajectories the verifier rewards; data coverage, the application data the agent is exposed to beyond what the task reads or writes; and feature coverage, the application features beyond what the task uses. Existing works set each dimension on engineering grounds, but they do not measure its effect on training. How complete must an environment be to train a capable agent, and along which dimensions? In MobileGym, we adjust each dimension separately, train Qwen3-VL-4B-Instruct, and evaluate every policy in the original full environment. A verifier that rewards only one reference path lowers success by 10.53 points, and removing unneeded records also lowers success. Pruning unneeded features instead raises success from 51.61% to 56.13%, entirely on tasks whose environments it left unchanged. The gain reverses when each episode is weighted equally and vanishes when only episodes on the pruned tasks are balanced. Without pruning, length weighting there reinforces long detour successes that erode skills on other tasks. Following these findings, we build MobileGym-Lite, which keeps reward and data coverage complete and prunes unneeded features. It outperforms Full in the full environment (55.48% versus 51.61% success) with lower construction cost.
Figures & tables
| Training | Success (%) | Successes |
|---|---|---|
| Base | 38.71 | 180 |
| Full | 51.61 | 240 |
| Path-restricted | 41.08 | 191 |
| Condition | Step8 | Step16 | Seen /396 | Unseen /69 |
|---|---|---|---|---|
| Full | 43.44 | 51.61 | 198 | 42 |
| R0 | 45.59 | 47.96 | 180 | 43 |
| R1 | 46.88 | 47.31 | 182 | 38 |
| R2 | 48.82 | 49.68 | 188 | 43 |
| Environment | Step8 | Step12 | Step16 |
|---|---|---|---|
| Full | 43.44 | 46.45 | 51.61 |
| F0 | 44.95 | 48.17 | 56.13 |
| F1 | 48.17 | 41.29 | 52.04 |
| Tail env. | Weighting | Ret. | Goal |
|---|---|---|---|
| Step-12 start | – | 81 | 83 |
| Full | Response | 62 | 65 |
| F0 | Response | 80 | 82 |
| Full | Episode | 80 | 84 |
| F0 | Episode | 70 | 72 |
| Weighting | Full | F0 | F0 Full |
|---|---|---|---|
| Response averaging | 62 | 80 | +18 |
| Episode balancing | 80 | 70 | 10 |
| Balance only | 66 | 80 | +14 |
| Balance only | 64 | 81 | +17 |
| Balance affected only | 79 | 80 | +1 |
| Balance other only | 64 | 71 | +7 |
| Environment | Split step | W /465 | N /465 | pp |
|---|---|---|---|---|
| Full | 8 | 240 | 238 | -0.43 |
| F0 | 8 | 261 | 262 | +0.22 |
| Lite | 8 | 250 | 258 | +1.72 |
| Environment | Step8 | Step12 | Step16 |
|---|---|---|---|
| Full | 43.44 | 46.45 | 51.61 |
| Lite | 51.40 | 52.04 | 55.48 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Clip | Success (%) |
|---|---|---|
| Full | W | |
| Lite | ||
| F0 |
| Setting | Value |
|---|---|
| Base model | Qwen3-VL-4B-Instruct |
| Training templates / outer steps | 192 / 16 |
| Tasks per step / rollouts per task | 12 / 8 |
| Learning rate | |
| Training temperature / token cap | 0.7 / 1024 |
| Evaluation temperature / token cap | 0 / 2048 |
| Task profile | Functions disabled by F0 | Restored in Lite |
|---|---|---|
| Clock: alarm | World clock, stopwatch, timer | World clock |
| Clock: world time | Stopwatch, timer | – |
| Notes: note operations | Todo list | – |
| Notes: todo operations | Notes settings | – |
| Calendar: fixed events/settings | Date calculator | – |
| Calendar: date tools | Calendar settings | Calendar settings |
| Environment | Clip | Branch start | Step8 | Step12 | Step16 | ID /396 | OOD /69 |
|---|---|---|---|---|---|---|---|
| Full | W | Base model | 202 | 216 | 240 | 198 | 42 |
| F0 | W | Base model | 209 | 224 | 261 | 224 | 37 |
| F0 | F0 (W): step8 | 209 s | – | 262 | – | – | |
| F1 | W | Base model | 224 | 192 | 242 | 201 | 41 |
| Full | Full: step8 | 202 s | 213 | 238 | 193 | 45 | |
| Lite | W | Base model | 239 | 233 | 250 | 206 | 44 |
| Environment | Clip | Aff. | Unmod. | OOD | All |
|---|---|---|---|---|---|
| Full | W | 97 | 101 | 42 | 240 |
| F0 | W | 96 | 128 | 37 | 261 |
| Lite | 98 | 113 | 47 | 258 |
| Quantity | Full | F0 | Lite |
|---|---|---|---|
| Training templates | 192 | 192 | 192 |
| Modified templates | 0 | 78 | 72 |
| Disablements | 0 | 127 | 87 |
| Visible nodes (9 screens) | 126 | 113 | 117 |
| Selected screen | Full | F0 | Lite |
|---|---|---|---|
| Clock alarm | 23 | 20 | 21 |
| Clock world | 10 | 8 | 8 |
| Notes note | 13 | 12 | 12 |
| Notes todo | 16 | 15 | 15 |
| Calendar event menu | 17 | 16 | 16 |
| Calendar date menu | 17 | 16 | 17 |
| Tail env. | Weighting | Ret. | Aff. | Unmod. | OOD | All |
|---|---|---|---|---|---|---|
| Step-12 start | – | 81 | 83 | 105 | 36 | 224 |
| Full | Response | 62 | 71 | 89 | 31 | 191 |
| F0 | Response | 80 | 85 | 106 | 43 | 234 |
| Full | Episode | 80 | 79 | 105 | 37 | 221 |
| F0 | Episode | 70 | 82 | 97 | 40 | 219 |
| Tail env. | Weighting | Kept /81 | New | Lost | Loops |
|---|---|---|---|---|---|
| Full | Response | 57 | 5 | 24 | 21 |
| F0 | Response | 66 | 14 | 15 | 7 |
| Full | Episode | 69 | 11 | 12 | 4 |
| F0 | Episode | 61 | 9 | 20 | 7 |
| Tail env. | Weighting | Success | Responses | Adam | Format |
|---|---|---|---|---|---|
| Full | Response | 142 | 3997 | 44 | 3 |
| F0 | Response | 141 | 4196 | 45 | 0 |
| Full | Episode | 149 | 4249 | 47 | 2 |
| F0 | Episode | 141 | 4274 | 46 | 1 |
| Environment | Weighting | Ticket price | Profile visibility |
|---|---|---|---|
| Step-12 start | – | Loop, 9 | Success, 8 |
| Full | Response | Loop, 9 | Success, 6 |
| F0 | Response | Success, 3 | Limit, 30 |
| Full | Episode | Success, 3 | Success, 7 |
| F0 | Episode | Loop, 9 | Success, 10 |
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 182 papers