Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Huawei Noah’s Ark Lab
GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.
Figures & tables
Figure 1: State aliasing and predictive-state augmentation. Observation-only conditioning can alias histories with different transition-relevant states (left). We recover predictive state from history and use it to condition a frozen GUI world model for state-consistent prediction (right).
Family
# Templates
Structured state Ztf
Semantic outcome Yt+1f
Persistent Artifact
9
mt:X→L∪{⊥}
object existence / location
Clipboard
5
ct∈V∪{⊥}
pasted payload
Ordered Collection
5
qt=(x1,…,xn)∈X∗
effective ordered sequence
Temporal Deadline
6
(ℓt,ρt)
timer fire / lifecycle outcome
Table 1: State families in StateAliasBench. ⊥ denotes absence or an empty state.
Figure 2: Overview of our framework. We construct strict pairs from real Android interactions, distill family-specific state specialists into a unified estimator Gϕ , recover Ztf from history, and condition a frozen GUI world model through a deterministic interface Γf .
Code2World-8B ( Zheng et al., 2026 )
gWorld-8B ( Koh et al., 2026 )
Condition
Pers.
Clip.
Ord.
Temp.
Pers.
Clip.
Ord.
Temp.
BranchExact (%)
Observation
29.00
0.00
0.00
0.00
42.00
0.00
0.00
15.75
Raw History
12.00
58.75
5.25
0.00
45.75
59.50
10.50
1.00
Oracle State
92.50
100.00
90.75
97.50
96.50
99.00
90.50
100.00
Specialist State
88.25
94.75
83.75
90.00
92.00
93.75
83.25
92.50
Table 2: StateAliasBench performance under different conditioning. Oracle State is a diagnostic upper bound and excluded from ranking. Best non-oracle results are in bold ; second-best results are underlined.
Code2World-8B
gWorld-8B
Metric
Obs.
Ours
Δ
Obs.
Ours
Δ
DINOv2
0.4718
0.6218
+0.1499
0.5534
0.6685
+0.1151
SigLIP
0.7234
0.8552
+0.1318
0.8021
0.8285
+0.0265
Table 3: Next-observation fidelity. Best results are in bold ; second-best results are underlined.
Agent
Base
+WM
+WM+Ours
Rel. Gain
GPT-5.4
56.03
58.62
60.34
+66.4%
Qwen3-VL-8B
41.38
44.83
47.41
+74.8%
Table 4: AndroidWorld task success (%). Both WM-assisted conditions use Code2World-8B. Best results are in bold ; second-best results are underlined.
Specialists
MTD (Ours)
Family
Strict
Cross-app
d -Stress
Interf.
Strict
Cross-app
d -Stress
Interf.
Persistent
85.5
87.0
87.0
87.0
86.0
75.5
82.5
82.5
Clipboard
89.5
85.5
89.5
89.5
85.0
88.5
85.0
88.5
Ordered
88.0
89.0
87.0
86.0
82.0
83.0
85.5
79.0
Temporal
88.5
83.5
85.0
87.5
86.5
72.0
88.0
89.5
Avg.
87.88
86.25
87.13
87.50
84.88
79.75
85.25
84.88
Table 5: State-recovery robustness. Strict reports PairCorrect on 200 A/B pairs per family; distribution shifts report Exact on 200 trajectories per family and are not directly comparable to Strict. Best results are in bold ; second-best results are underlined.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Template
State contrast and convergence
Shared probe / semantic outcome
Released support
Persistent Artifact
copy_direct
A confirms COPY while B cancels; both return to the same focused destination search.
Search the target; present vs. absent.
20 pairs; 20 targets; 5 file extensions.
move_direct
A confirms MOVE while B cancels; both return to the same focused destination search.
Search the target; present vs. absent.
20 pairs; 20 targets; 5 extensions.
delete_direct
A confirms DELETE while B cancels; both return to the same focused source search.
Search the target; absent vs. present.
20 pairs; 20 targets; 5 extensions.
copy_interference
Branches differ in target COPY while an independent decoy COPY is executed oppositely; both converge to the same neutral folder.
Open target destination; target present vs. absent.
34 pairs; one decoy operation per branch.
move_interference
Target MOVE differs while an independent decoy MOVE is inverted; both converge to the same neutral folder.
Open target destination; target present vs. absent.
33 pairs; one decoy operation per branch.
Appendix
Table 6: Causal templates in StateAliasBench. “State contrast” summarizes how the two histories establish different structured states before returning to a common prediction boundary. Parameterization reports the support observed in the released strict split.
Audit criterion
Train
Eval
Intersection
Record identifier
2,538
1,600
0
Trajectory identifier
1,658
800
0
Semantic target identity
825
611
0
Canonical content fingerprint
2,516
1,600
0
Whole-trajectory signature
1,741
854
0
Appendix
Table 7: Train–evaluation split-integrity audit for predictive-state recovery. All reported criteria test duplicate examples or complete trajectories, rather than reuse of lower-level GUI frames or action primitives.
Information
Train input
Test input
Role
Family/schema instruction qf
Yes
Yes
Task specification
Ordered GUI history
Yes
Yes
Model-visible
Historical actions
Yes
Yes
Model-visible
Current GUI observation
Yes
Yes
Model-visible
Relative duration (Temporal)
Yes
Yes
Model-visible
Wall-clock information in pixels
Possible
Possible
Model-visible
Appendix
Table 8: Information-access audit for the predictive-state estimator on StateAliasBench.
Figure 3: Persistent Artifact: move_direct . Representative strict pair move_41000 . The histories differ in whether the file move is completed, but both branches return to the same prediction-boundary GUI Ot . Audit-only state evidence shows different source/destination existence states. Executing the same search action At produces different next observations and semantic artifact-presence outcomes.
Figure 4: Clipboard: original_strict . Representative strict pair clipboard_001 . The two histories copy different payloads before converging to the same blank editor Ot . The current clipboard contents are not visible in that observation. Applying the same paste action At yields different editor contents, revealing the branch-specific clipboard state.
Figure 5: Ordered Collection: playing_queue . Representative strict pair ordered_xv2_playing_queue_001 . Branch-specific histories construct different effective queue orders while converging to the same Now Playing observation Ot . The shared queue-opening action At reveals different ordered sequences in the next observation.
Figure 6: Temporal Deadline: start_time_shift . Representative strict pair temporal_sealed_start_time_shift_01 . The histories establish different effective remaining times while eventually producing the same prediction-boundary GUI Ot . The same controlled wait-and-probe action At causes the timer to fire in one branch but not the other. Simulator-side logical-time state is shown only through audit annotations and is not supplied to the predictive-state estimator.
Family
Train
Val.
Construction
Persistent Artifact
898
110
Interaction histories centered on a single tracked artifact. Each example contains at most one state transition affecting the tracked artifact, while operations involving unrelated files or paths may appear as distractors. The target state specifies whether the artifact exists at its source and destination locations.
Clipboard
160
20
Atomic clipboard histories containing a single effective copy event. The examples isolate persistence of clipboard content across subsequent GUI navigation and provide the current clipboard payload as the structured-state target.
Ordered Collection
1,080
96
Order-sensitive interaction histories covering playlist and playing-queue construction, insertion, repositioning, reset, and interleaved collection operations. The target state is the complete effective item ordering at the final observation.
Temporal Deadline
400
48
Independently collected timer histories spanning eight atomic temporal transition types. The trajectories vary timer creation, elapsed time, pause/resume behavior, reset, deletion, and restart operations. The target state specifies whether the timer is active and its effective remaining time at the final observation.
Total
2,538
274
—
Appendix
Table 9: Family-specific training data for predictive-state recovery. The specialist teachers and unified estimators are trained from the same underlying family-specific datasets.
Family
Schema instruction
Canonical state target
Persistent Artifact
Predict the current persistent state of the specified artifact from the ordered Files interaction history while ignoring unrelated artifacts and paths.
Infer the current timer state from the ordered visible history, including interaction-visible temporal evidence.
{"timer_active": bool, "remaining_seconds": int}
Appendix
Table 10: Family-specific predictive-state schemas. The same unified estimator is conditioned by qf to emit the corresponding canonical structured state.
Family
Train
Epochs
Optim. steps
Persistent Artifact
898
4
452
Clipboard
160
4
80
Ordered Collection
1080
2
270
Temporal Deadline
400
4
200
Appendix
Table 11: Training schedules of the four family-specific specialist teachers. All models use rank-16 LoRA and an effective batch size of eight.
Method
Persistent
Clipboard
Ordered
Temporal
Avg.
Worst
Specialists
91.75 / 85.5
94.75 / 89.5
92.75 / 88.0
92.50 / 88.5
87.88
85.5
Family-balanced Joint CE
96.00 / 92.0
74.75 / 69.5
92.75 / 89.0
85.00 / 84.0
83.63
69.5
MTD (Ours)
89.75 / 86.0
89.00 / 85.0
87.25 / 82.0
88.50 / 86.5
84.88
82.0
Appendix
Table 12: Predictive-state recovery under family-specific specialists, direct family-balanced joint training, and multi-teacher distillation. Each family cell reports BranchExact / PairCorrect (%). Avg. and Worst are computed from PairCorrect across the four state families. Best results are in bold ; second-best results are underlined.
Method
Persistent
Clipboard
Ordered
Temporal
Avg.
Worst
Family-balanced Joint CE
89.50 / 79.50
74.75 / 69.50
87.25 / 79.00
82.50 / 79.00
76.75
69.50
MTD (Ours)
86.75 / 80.00
89.00 / 85.00
81.50 / 71.50
85.75 / 81.00
79.38
71.50
Appendix
Table 13: Matched end-to-end Code2World-8B evaluation of the two unified predictive-state estimators. Each family cell reports BranchExact / PairCorrect (%). Avg. and Worst are computed from final-WM PairCorrect. Both methods use the same complete state-conditioning interface and frozen world model. Best results are in bold ; second-best results are underlined.
Family
Recovered state
Action-resolved consequence Dtf
Complete model-compatible realization
Persistent Artifact
Source and destination existence of the tracked artifact.
Resolve whether the tracked artifact must be present or absent on the Files surface inspected by the candidate folder or filename-search action.
Compile the result into a Downloads-scoped search surface or the requested folder listing, project context onto that inspected surface, and impose the corresponding visible-row or empty-state constraints.
Clipboard
Current clipboard type and payload.
Resolve the payload inserted by the candidate PASTE action.
Construct a post-action editor surface containing the recovered payload as visible editor content while preserving recognizable editor chrome.
Ordered Collection
Collection type and complete ordered item sequence.
Resolve the playlist or playing-queue state exposed by the candidate action, preserving the supplied sequence without sorting or deduplication.
Construct the corresponding collection surface with the required number and order of visible primary rows and serialize the complete sequence through the WM-compatible representation.
Temporal Deadline
Timer activity and effective remaining time.
Combine timer activity and remaining time with the elapsed duration in the candidate action to resolve none , running , or fired , together with the post-action remainder when applicable.
Compile the resolved timer state into a full-screen notification-shade realization with the corresponding visible timer text, countdown, and control structure.
Appendix
Table 14: Family-specific deterministic realization of recovered predictive state. Dtf is the action-resolved consequence; the final column describes the complete model-compatible realization.
Component
Code2World-8B
gWorld-8B
Current observation
Application crop with the model-native action visual hint.
Same application crop without the Code2World visual hint.
Action representation
Lowercase model-native action names; point actions use scalar coordinate fields.
Uppercase model-native action names; point actions use coordinate arrays.
Message structure
Fixed system instruction followed by a multimodal user request.
User-only multimodal request.
Rendering convention
Fixed 1080×2400 target canvas.
360×800 CSS viewport, captured at device scale factor 3 .
Persistent Artifact
Resolved Files blueprint and projected path/entity context are inserted into the Code2World-native transition request.
The same resolved Files semantics are expressed through the gWorld-native mobile-UI request.
Clipboard
The resolved payload is represented once as authoritative visible editor content.
The same payload is inserted into the gWorld-native editor request.
Appendix
Table 15: WM-specific realization of the same deterministic predictive-state information.
Quantity
Source
Available at prediction time
Used
Current observation Ot
Current GUI observation
Yes
Yes
Candidate action At
Agent / benchmark action
Yes
Yes
Family/task specification
Fixed task protocol
Yes
Yes
Task/entity context ct
Prediction-time task specification, when applicable
Yes
Yes
Recovered state Ztf
Learned state estimator
Yes
Yes
Action-resolved consequence Dtf
Fixed deterministic state/action rules
Yes
Yes
Appendix
Table 16: Information provenance for the learned state-conditioned WM.
Family
WM
Canon.
+Cons.
+Surf.
Full
Persistent
Code2World-8B
1.0
4.0
26.0
78.5
Persistent
gWorld-8B
12.5
14.5
40.5
85.5
Temporal
Code2World-8B
0.0
0.0
0.5
84.0
Temporal
gWorld-8B
0.0
0.0
22.5
88.5
Appendix
Table 17: Progressive realization of a fixed family-specialist recovered state, measured by PairCorrect (%). Canonical → +Consequence and +Consequence → +Surface are strictly request-matched comparisons. Full uses the production WM-native interface.
Family
WM
Condition
Sem. B
Surf. B
Full B
Sem. P
Full P
Persistent
Code2World-8B
+Surface
60.5
59.0
42.8
27.5
26.0
Persistent
Code2World-8B
Full
94.0
92.8
88.2
89.5
78.5
Persistent
gWorld-8B
+Surface
70.2
92.0
64.8
41.5
40.5
Persistent
gWorld-8B
Full
93.0
98.8
92.0
87.5
85.5
Temporal
Code2World-8B
+Surface
12.5
58.0
10.5
1.0
0.5
Temporal
Code2World-8B
Full
91.0
98.2
90.0
86.0
84.0
Appendix
Table 18: Decomposition of the archived +Surface and Full results into semantic correctness and surface validity. All values are percentages. No prediction is regenerated and no scoring rule is modified.
Family
WM
Gains
Losses
Net
Semantic-only
Surface-only
Both
Persistent
Code2World-8B
200
18
+182
60
65
75
Persistent
gWorld-8B
117
8
+109
88
20
9
Temporal
Code2World-8B
319
1
+318
167
8
144
Temporal
gWorld-8B
193
2
+191
173
7
13
Appendix
Table 19: Branch-level attribution of the +Surface → Full transition. “Gains” counts branches changing from official incorrect to correct; “Losses” counts the reverse. The final three columns partition the gains. Each model–family condition contains 400 branches.
Family
State/action resolution
Required visible surface
Context/serialization rule
Persistent Artifact
For a Downloads search, resolve presence from tracked in-scope source and destination paths. For a folder probe, resolve presence at the exact inspected artifact path.
Present search: Downloads-scoped heading and one target result row. Absent search: explicit “No matches in Downloads” and no result row. Present folder: requested folder and one target row. Absent folder: requested folder with no tracked-artifact row.
Visible branches retain artifact identity and the relevant scoped path. Absent branches retain the inspected path and surface mode but omit target identity. The resulting Files blueprint is inserted through the WM-native transition representation.
Clipboard
PASTE deterministically inserts the recovered clipboard payload.
A recognizable editor containing the recovered payload as visible editor-body text.
The payload is serialized once as authoritative post-action content; no additional state is inferred from the screenshot.
Ordered Collection
Preserve the recovered collection type and ordered item sequence exactly.
A playlist-detail or playing-queue surface containing exactly the supplied number of primary rows in the supplied order.
The canonical list remains the authoritative sequence. WM-specific serialization avoids redundant list representations that could duplicate item titles.
Temporal Deadline
Inactive → none; active with Δt≥rt→ fired; active with Δt<rt→ running with rt−Δt seconds remaining.
None: notification shade without timer state. Fired: one Clock card with Time’s up and Stop . Running: one Clock/Timer card with the updated countdown and Pause .
The resolved outcome is compiled directly into the native notification request. Code2World uses its fixed-canvas format; gWorld uses a dedicated compact 360×800 notification-shade request.
Appendix
Table 20: Deterministic rules used by the Full state-conditioning interface.
Code2World-8B
gWorld-8B
Checkpoint
released Code2World-8B
released gWorld-8B
Maximum output tokens
8192
8192
Temperature
0
0
Generation seed
20260912
20260912
Repetition penalty
1.0
1.0
Top- p /Top- k override
None
None
Appendix
Table 21: World-model configuration used for the staged state-interface evaluation.
Specialists
Predictive State (Ours)
Family
Strict
Cross-app
d -Stress
Interf.
Strict
Cross-app
d -Stress
Interf.
Persistent Artifact
85.5
87.0
87.0
87.0
86.0
75.5
82.5
82.5
Clipboard
89.5
85.5
89.5
89.5
85.0
88.5
85.0
88.5
Ordered Collection
88.0
89.0
87.0
86.0
82.0
83.0
85.5
79.0
Temporal Deadline
88.5
83.5
85.0
87.5
86.5
72.0
88.0
89.5
Average
87.88
86.25
87.13
87.50
84.88
79.75
85.25
84.88
Appendix
Table 22: Predictive-state recovery under the primary strict setting and three distribution shifts. Strict reports PairCorrect (%) over 200 A/B pairs per family; Cross-app, d -Stress, and Interference report canonical-state Exact (%) over 200 independent trajectories per family. Because these settings use different evaluation units, Strict percentages should not be directly compared with the distribution-shift percentages.
WM
Metric
Observation
Ours
Δ
Code2World
DINOv2
0.4718
0.6218
+0.1499
SigLIP
0.7234
0.8552
+0.1318
gWorld
DINOv2
0.5534
0.6685
+0.1151
SigLIP
0.8021
0.8285
+0.0265
Appendix
Table 23: Next-observation feature similarity on the frozen 100-branch fidelity subset. Each generated prediction is compared with the corresponding recorded next observation.
Backbone
Agent
Agent+WM
Agent+WM+State
Rel. Gain
GPT-5.4
56.03
58.62
60.34
+66.4%
Qwen3-VL-8B
41.38
44.83
47.41
+74.8%
Appendix
Table 24: AndroidWorld task success (%) on the full 116-task benchmark. Code2World-8B is used as the world model in both WM-assisted conditions. The final column reports the additional gain from predictive-state conditioning relative to the improvement obtained by adding the WM alone. Best task-success results are in bold ; second-best results are underlined.
Figure 7: Representative paired AndroidWorld rescue on FilesDeleteFile . The Agent+WM and Agent+WM+State conditions share the same interaction prefix and current observation at the divergence point. Standard Code2World-8B prediction produces a misleading near-name file match and leads the selector to SWIPE UP . Predictive-state conditioning changes the simulated future and leads the selector to SEARCH , followed by successful localization and deletion of the target artifact. The recovered state is supplied only to the world model and is not directly exposed to the action selector.
GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI agents. This mismatch leaves a key requirement under-tested: generated states must remain contextually consistent when they are repeatedly reused for future interaction. We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments rather than isolated next-screen predictors. GUI-CC contains two complementary tracks: an offline reference-action track that rolls models along real mobile GUI trajectories, and an online agent-loop track that lets fixed probing agents interact with model-generated UIs. We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps. GUI-CC evaluates transition fidelity, transition plausibility, contextual consistency, and task progress. Experiments show that plausible single-step generation does not guarantee reliable environment simulation: current models often produce usable-looking screens while failing to preserve task-relevant context or support executable multi-step rollouts.
Lin Fu, Zheyuan Yang, Tianhui Zhang +5
Zhejiang University · Tongji University · University of California, San Diego +3
GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery. We introduce EvoGUI, a diagnostic framework that converts normalized GUI trajectories into three complementary visual question answering probes: temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination. Their labels are derived from trajectory order and logged actions, requiring no additional task-label annotation after trajectory normalization. We instantiate EvoGUI-Bench from Mind2Web and WebLINX, yielding 3,000 instances across 120 domains, and evaluate 28 vision-language model configurations zero-shot. The strongest model reaches only 60.4 EvoGain, while model scale and GUI specialization do not reliably predict performance. These results establish EvoGUI-Bench as a scalable diagnostic complement to end-to-end GUI-agent evaluation while exposing substantial headroom in state-transition understanding. The source code is publicly available at https://github.com/Yyhhh6/EvoGUI.
A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.
Jiaming Zhang, Xuan Wang, Fuyao Zhang +3
Nanyang Technological University · Institute of Science Tokyo · Sony AI