GUI world models are increasingly used to improve action selection by predicting the consequences of candidate actions, but it is unclear how much of the resulting gain comes from prediction itself rather than from the evaluator that interprets those predictions. We study this attribution problem through controlled interventions on GUI action selection. With candidates and predictions held fixed, lookahead gains can change with how the prediction is presented to the evaluator, so end-to-end gains cannot be attributed to prediction alone. We then introduce Direct Action Ranking (DAR), which learns from reference actions to score candidates without generating future screens, and compare it with a prediction-augmented variant under matched supervision. Predicted successors add no detectable ranking signal once evaluation is learned directly. Without retraining, DAR improves AndroidWorld task success by 5.8--16.4 points across three proposers, outperforms world-model lookahead, and transfers zero-shot to unseen apps, where every prediction-conditioned configuration falls below proposer confidence. These results suggest that GUI world models should be evaluated by their decision value, the incremental action-selection value of their predictions beyond a strong direct evaluator, rather than by prediction fidelity or end-to-end lookahead gain alone.
Figures & tables
Figure 1: Action selection and controlled comparisons. Lookahead and DAR use alternative paths to score the same candidate actions. We vary the evaluator with predictions fixed, compare DAR with prediction-augmented DAR under matched training, and evaluate online task completion. Interface thumbnails are schematic.
Table 1: Diagnostic signals for useful overrides. Override-Benefit AUROC on each evaluator’s informative subset. Each evaluator has a selected-minus-Top-1 contrast column and a per-candidate scalar column; set- and step-level scores span both columns. The first four rows require the reference successor; the remaining signals are available before execution. Darker/lighter lavender marks the highest/second-highest scores per evaluator and availability block. Full statistics: Appendix A.3.
Table 2: Offline reference-match ranking (AUROC). Both AndroidControl columns use one Qwen2.5-VL-7B test-candidate pool; column labels identify DAR’s training-candidate source. Mind2Web uses a separate web evaluator. Darker/lighter lavender marks the highest and second-highest scores within each column.
Table 3: Candidate budget, selector supervision, and an SFT baseline. 3,000 AndroidControl decisions. (a) Candidate budget for a fixed proposer. (b) Nested DAR training sets; conditional match accuracy is computed on decisions whose candidate set contains a reference match. (c) Qwen3-VL-32B proposer SFT, evaluated on the same evaluation fold; —A— is the deduplicated candidate count. The two-decode setting has mean candidate count 3.64. Full settings: Appendices B.2–B.4.
Figure 2: Different choices on a recorded AndroidControl step. To find product prices, Top-1 and lookahead choose different clicks; DAR selects the reference scroll. Panels show the current screen and predictions for the lookahead and reference actions. The table gives candidate scores; “eval.” denotes the lookahead evaluator. Case selection: Appendix E.
Table 4: AndroidWorld application-family results (%). 116 tasks, three seeds. All: mean ± s.e.m. External/Self use Code2World. Counts appear below family names; shading ranks selectors per proposer and column. Full paired statistics: Appendix C.1.
Table 5: AndroidWorld with gWorld-8B (%). Same 116 tasks and families as Table 4 ; entries are three-seed means. Δ is the difference between the displayed three-seed mean success rates (gWorld–Code2World). Top-1/DAR do not use a world model.
Table 6: KApps transfer. Reference-Match AUROC (mean ± SD over three seeds) across 14 Korean apps. The same DAR evaluator is used without retraining across proposers; shading ranks each column.
Table 7: MiniWob task-family results (%). 98 tasks × 3 seeds. All: mean ± s.e.m.; parentheses: paired DAR–Top-1 gain ± s.e.m. Shading ranks each proposer and column.
Table 8: Evaluator and objective sensitivity. AndroidControl offline; AndroidWorld online (116 tasks, Qwen3-VL-8B, three-seed mean ± s.e.m.). Conditional match accuracy is computed on decisions whose candidate set contains a reference match; full statistics are in Appendix C.4.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: Complete diagnostic signal sweep. Directional Override-Benefit AUROC with 95% episode-clustered bootstrap intervals. Each column uses its evaluator’s own informative overrides: Sonnet 4.6 n=330 and Opus 4.8 n=366 . AUROC values with within-panel Benjamini–Hochberg adjusted q<0.05 are bold. “Contrast” denotes selected-minus-Policy Top-1; “scalar” denotes a per-candidate or step-level score.
Table 10: Main mobile DAR configuration.
Table 11: Proposer-SFT baseline. Greedy reference matching, candidate-set size, and Pass@3 on the shared evaluation fold.
Table 12: Candidate-budget sensitivity. Reference coverage is the proportion of decisions containing at least one matching candidate. Selected-action columns report Reference-Match Accuracy; gains are absolute fractions. The two-decode setting has mean candidate count 3.64.
Table 13: Training-data sensitivity. AUROC includes 95% intervals. Conditional match accuracy measures selected-action matching on observations whose candidate set contains a reference match.
Table 14: Paired AndroidWorld task-success contrasts. DAR minus each baseline, pooled over seeds 30, 31, and 32. Gains are within-seed paired differences in percentage points. Intervals use task-clustered bootstrap; p-values use exact McNemar tests. Paired differences are computed before rounding marginal means.
Table 15: Backbone sensitivity with paired online tests. AndroidControl AUROC and AndroidWorld success for the Qwen3-VL-8B proposer; online values are three-seed mean ± s.e.m. The final column reports exact paired tests against Top-1.
Figure 3: Score behavior under pointwise supervision. Reliability curves (15 equal-count bins; 8,839 candidates) and score distributions for the full and no-pointwise Qwen3-VL-8B objectives from Table 8 (b).
Table 16: Lookahead depth and representation. Reference-match change relative to Top-1, in absolute fractions, under Claude Opus 4.8. Depth results include 95% confidence intervals; representation results report point estimates.
Table 17: Three-way reference recoveries. All 12 decisions where Top-1, lookahead, and DAR choose distinct actions and DAR matches the reference. Parentheses show each selector’s score for its selected action; DAR margin is the DAR-score difference between the reference action and Top-1.
Figure 4: Neighboring controls. To snooze a reminder, Top-1 waits, lookahead selects the adjacent control, and DAR selects the reference control.
Figure 5: Equivalent actions, different predictions. Equivalent non-reference clicks receive different lookahead scores; DAR scores them identically and selects the reference control.
World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · Meituan · School of Artificial Intelligence, University of Chinese Academy of Sciences