GUI world models are increasingly used to improve action selection by predicting the consequences of candidate actions, but it is unclear how much of the resulting gain comes from prediction itself rather than from the evaluator that interprets those predictions. We study this attribution problem through controlled interventions on GUI action selection. With candidates and predictions held fixed, lookahead gains can change with how the prediction is presented to the evaluator, so end-to-end gains cannot be attributed to prediction alone. We then introduce Direct Action Ranking (DAR), which learns from reference actions to score candidates without generating future screens, and compare it with a prediction-augmented variant under matched supervision. Predicted successors add no detectable ranking signal once evaluation is learned directly. Without retraining, DAR improves AndroidWorld task success by 5.8--16.4 points across three proposers, outperforms world-model lookahead, and transfers zero-shot to unseen apps, where every prediction-conditioned configuration falls below proposer confidence. These results suggest that GUI world models should be evaluated by their decision value, the incremental action-selection value of their predictions beyond a strong direct evaluator, rather than by prediction fidelity or end-to-end lookahead gain alone.
Figures & tables
Figure 1: Action selection and controlled comparisons. Lookahead and DAR use alternative paths to score the same candidate actions. We vary the evaluator with predictions fixed, compare DAR with prediction-augmented DAR under matched training, and evaluate online task completion. Interface thumbnails are schematic.
Table 1: Diagnostic signals for useful overrides. Override-Benefit AUROC on each evaluator’s informative subset. Each evaluator has a selected-minus-Top-1 contrast column and a per-candidate scalar column; set- and step-level scores span both columns. The first four rows require the reference successor; the remaining signals are available before execution. Darker/lighter lavender marks the highest/second-highest scores per evaluator and availability block. Full statistics: Appendix A.3.
Table 2: Offline reference-match ranking (AUROC). Both AndroidControl columns use one Qwen2.5-VL-7B test-candidate pool; column labels identify DAR’s training-candidate source. Mind2Web uses a separate web evaluator. Darker/lighter lavender marks the highest and second-highest scores within each column.
Table 3: Candidate budget, selector supervision, and an SFT baseline. 3,000 AndroidControl decisions. (a) Candidate budget for a fixed proposer. (b) Nested DAR training sets; conditional match accuracy is computed on decisions whose candidate set contains a reference match. (c) Qwen3-VL-32B proposer SFT, evaluated on the same evaluation fold; —A— is the deduplicated candidate count. The two-decode setting has mean candidate count 3.64. Full settings: Appendices B.2–B.4.
Figure 2: Different choices on a recorded AndroidControl step. To find product prices, Top-1 and lookahead choose different clicks; DAR selects the reference scroll. Panels show the current screen and predictions for the lookahead and reference actions. The table gives candidate scores; “eval.” denotes the lookahead evaluator. Case selection: Appendix E.
Table 4: AndroidWorld application-family results (%). 116 tasks, three seeds. All: mean ± s.e.m. External/Self use Code2World. Counts appear below family names; shading ranks selectors per proposer and column. Full paired statistics: Appendix C.1.
Table 5: AndroidWorld with gWorld-8B (%). Same 116 tasks and families as Table 4 ; entries are three-seed means. Δ is the difference between the displayed three-seed mean success rates (gWorld–Code2World). Top-1/DAR do not use a world model.
Table 6: KApps transfer. Reference-Match AUROC (mean ± SD over three seeds) across 14 Korean apps. The same DAR evaluator is used without retraining across proposers; shading ranks each column.
Table 7: MiniWob task-family results (%). 98 tasks × 3 seeds. All: mean ± s.e.m.; parentheses: paired DAR–Top-1 gain ± s.e.m. Shading ranks each proposer and column.
Table 8: Evaluator and objective sensitivity. AndroidControl offline; AndroidWorld online (116 tasks, Qwen3-VL-8B, three-seed mean ± s.e.m.). Conditional match accuracy is computed on decisions whose candidate set contains a reference match; full statistics are in Appendix C.4.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: Complete diagnostic signal sweep. Directional Override-Benefit AUROC with 95% episode-clustered bootstrap intervals. Each column uses its evaluator’s own informative overrides: Sonnet 4.6 n=330 and Opus 4.8 n=366 . AUROC values with within-panel Benjamini–Hochberg adjusted q<0.05 are bold. “Contrast” denotes selected-minus-Policy Top-1; “scalar” denotes a per-candidate or step-level score.
Table 10: Main mobile DAR configuration.
Table 11: Proposer-SFT baseline. Greedy reference matching, candidate-set size, and Pass@3 on the shared evaluation fold.
Table 12: Candidate-budget sensitivity. Reference coverage is the proportion of decisions containing at least one matching candidate. Selected-action columns report Reference-Match Accuracy; gains are absolute fractions. The two-decode setting has mean candidate count 3.64.
Table 13: Training-data sensitivity. AUROC includes 95% intervals. Conditional match accuracy measures selected-action matching on observations whose candidate set contains a reference match.
Table 14: Paired AndroidWorld task-success contrasts. DAR minus each baseline, pooled over seeds 30, 31, and 32. Gains are within-seed paired differences in percentage points. Intervals use task-clustered bootstrap; p-values use exact McNemar tests. Paired differences are computed before rounding marginal means.
Table 15: Backbone sensitivity with paired online tests. AndroidControl AUROC and AndroidWorld success for the Qwen3-VL-8B proposer; online values are three-seed mean ± s.e.m. The final column reports exact paired tests against Top-1.
Figure 3: Score behavior under pointwise supervision. Reliability curves (15 equal-count bins; 8,839 candidates) and score distributions for the full and no-pointwise Qwen3-VL-8B objectives from Table 8 (b).
Table 16: Lookahead depth and representation. Reference-match change relative to Top-1, in absolute fractions, under Claude Opus 4.8. Depth results include 95% confidence intervals; representation results report point estimates.
Table 17: Three-way reference recoveries. All 12 decisions where Top-1, lookahead, and DAR choose distinct actions and DAR matches the reference. Parentheses show each selector’s score for its selected action; DAR margin is the DAR-score difference between the reference action and Top-1.
Figure 4: Neighboring controls. To snooze a reminder, Top-1 waits, lookahead selects the adjacent control, and DAR selects the reference control.
Figure 5: Equivalent actions, different predictions. Equivalent non-reference clicks receive different lookahead scores; DAR scores them identically and selects the reference control.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.
Seungjun Yu, Hojun Choi, Seojeong Park +2
Korea Advanced Institute of Science and Technology (KAIST)
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.
Yilin Cao, Yufeng Zhong, Zhixiong Zeng +6
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · Meituan · School of Artificial Intelligence, University of Chinese Academy of Sciences
World action models generate actions together with visual predictions of their consequences. These paired outputs create the potential for planning by sampling multiple actions from one state, comparing their imagined outcomes, and choosing the action with the most promising predicted outcome. However, how to use imagined futures to guide action selection remains unclear. We examine this planning potential empirically. First, we estimate an oracle upper bound on selection by choosing the sampled candidate whose realised outcome is best. In a controlled same-state analysis, this choice raises success from 68.9% under uniform random selection to 79.2%. We then test selectors based on visual quality, physical consistency, and task progression as controlled interventions. Some tested selectors yield higher observed success, but the gains are uneven and the matched selectors leave much of the measured opportunity unrecovered. To investigate this gap, we examine whether sampled actions lead to different outcomes, whether these differences are visible in the predictions, and whether a score recognises them. Counterfactual branching from the same states shows that selection opportunity is concentrated in relatively few decisions in the initial candidate sets. Action spread and outcome coverage need not increase together. In a further evaluation across trajectory phases with complete action execution, the tested scores again recover little of the available improvement despite a small gain from learned value. These findings distinguish producing consequential action choices from recognising them in generated futures, motivating the evaluation of WAM predictions through their usefulness for decisions rather than visual quality alone.
Jianhao Yuan, Yu Yuan, Benjamin Ramtoula +5
University of Oxford · Purdue University · UWE Bristol