The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

Dongsheng Liu; Chao Jin; Wenkui Yang; Hejin Wang; Junwei Yang; Zeren Zhang; Ziwei Chen; Huaibo Huang

cs.LGSep 26, 2026

The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

Authors: Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, +2 more

Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Huawei Noah’s Ark Lab

Abstract

GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.

Figures & tables

Figure 1: State aliasing and predictive-state augmentation. Observation-only conditioning can alias histories with different transition-relevant states (left). We recover predictive state from history and use it to condition a frozen GUI world model for state-consistent prediction (right).

Family	# Templates	Structured state $\widetilde{Z}_{t}^{f}$	Semantic outcome $Y_{t+1}^{f}$
Persistent Artifact	9	$m_{t}:\mathcal{X}\rightarrow\mathcal{L}\cup\{\bot\}$	object existence / location
Clipboard	5	$c_{t}\in\mathcal{V}\cup\{\bot\}$	pasted payload
Ordered Collection	5	$\mathbf{q}_{t}=(x_{1},\ldots,x_{n})\in\mathcal{X}^{\ast}$	effective ordered sequence
Temporal Deadline	6	$(\ell_{t},\rho_{t})$	timer fire / lifecycle outcome

Table 1: State families in StateAliasBench.

\bot

denotes absence or an empty state.

Figure 2: Overview of our framework. We construct strict pairs from real Android interactions, distill family-specific state specialists into a unified estimator $G_{\phi}$ , recover $\widehat{Z}_{t}^{f}$ from history, and condition a frozen GUI world model through a deterministic interface $\Gamma_{f}$ .

	Code2World-8B ( Zheng et al., 2026 )				gWorld-8B ( Koh et al., 2026 )
Condition	Pers.	Clip.	Ord.	Temp.	Pers.	Clip.	Ord.	Temp.
BranchExact (%)
Observation	29.00	0.00	0.00	0.00	42.00	0.00	0.00	15.75
Raw History	12.00	58.75	5.25	0.00	45.75	59.50	10.50	1.00
Oracle State	92.50	100.00	90.75	97.50	96.50	99.00	90.50	100.00
Specialist State	88.25	94.75	83.75	90.00	92.00	93.75	83.25	92.50

Table 2: StateAliasBench performance under different conditioning. Oracle State is a diagnostic upper bound and excluded from ranking. Best non-oracle results are in bold ; second-best results are underlined.

Metric	Obs.	Ours	$\bm{\Delta}$	Obs.	Ours	$\bm{\Delta}$
	Code2World-8B			gWorld-8B
DINOv2	0.4718	0.6218	+0.1499	0.5534	0.6685	+0.1151
SigLIP	0.7234	0.8552	+0.1318	0.8021	0.8285	+0.0265

Table 3: Next-observation fidelity. Best results are in bold ; second-best results are underlined.

Agent	Base	+WM	+WM+Ours	Rel. Gain
GPT-5.4	56.03	58.62	60.34	+66.4%
Qwen3-VL-8B	41.38	44.83	47.41	+74.8%

Table 4: AndroidWorld task success (%). Both WM-assisted conditions use Code2World-8B. Best results are in bold ; second-best results are underlined.

	Specialists				MTD (Ours)
Family	Strict	Cross-app	$d$ -Stress	Interf.	Strict	Cross-app	$d$ -Stress	Interf.
Persistent	85.5	87.0	87.0	87.0	86.0	75.5	82.5	82.5
Clipboard	89.5	85.5	89.5	89.5	85.0	88.5	85.0	88.5
Ordered	88.0	89.0	87.0	86.0	82.0	83.0	85.5	79.0
Temporal	88.5	83.5	85.0	87.5	86.5	72.0	88.0	89.5
Avg.	87.88	86.25	87.13	87.50	84.88	79.75	85.25	84.88

Table 5: State-recovery robustness. Strict reports PairCorrect on 200 A/B pairs per family; distribution shifts report Exact on 200 trajectories per family and are not directly comparable to Strict. Best results are in bold ; second-best results are underlined.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Template	State contrast and convergence	Shared probe / semantic outcome	Released support
Persistent Artifact
copy_direct	A confirms COPY while B cancels; both return to the same focused destination search.	Search the target; present vs. absent.	20 pairs; 20 targets; 5 file extensions.
move_direct	A confirms MOVE while B cancels; both return to the same focused destination search.	Search the target; present vs. absent.	20 pairs; 20 targets; 5 extensions.
delete_direct	A confirms DELETE while B cancels; both return to the same focused source search.	Search the target; absent vs. present.	20 pairs; 20 targets; 5 extensions.
copy_interference	Branches differ in target COPY while an independent decoy COPY is executed oppositely; both converge to the same neutral folder.	Open target destination; target present vs. absent.	34 pairs; one decoy operation per branch.
move_interference	Target MOVE differs while an independent decoy MOVE is inverted; both converge to the same neutral folder.	Open target destination; target present vs. absent.	33 pairs; one decoy operation per branch.

Appendix

Table 6: Causal templates in StateAliasBench. “State contrast” summarizes how the two histories establish different structured states before returning to a common prediction boundary. Parameterization reports the support observed in the released strict split.

Audit criterion	Train	Eval
Record identifier	2,538	1,600
Trajectory identifier	1,658	800
Semantic target identity	825	611
Canonical content fingerprint	2,516	1,600
Whole-trajectory signature	1,741	854

Appendix

Table 7: Train–evaluation split-integrity audit for predictive-state recovery. All reported criteria test duplicate examples or complete trajectories, rather than reuse of lower-level GUI frames or action primitives.

Information	Train input	Test input	Role
Family/schema instruction $q_{f}$	Yes	Yes	Task specification
Ordered GUI history	Yes	Yes	Model-visible
Historical actions	Yes	Yes	Model-visible
Current GUI observation	Yes	Yes	Model-visible
Relative duration (Temporal)	Yes	Yes	Model-visible
Wall-clock information in pixels	Possible	Possible	Model-visible

Appendix

Table 8: Information-access audit for the predictive-state estimator on StateAliasBench.

Figure 3: Persistent Artifact: move_direct . Representative strict pair move_41000 . The histories differ in whether the file move is completed, but both branches return to the same prediction-boundary GUI $O_{t}$ . Audit-only state evidence shows different source/destination existence states. Executing the same search action $A_{t}$ produces different next observations and semantic artifact-presence outcomes.

Figure 4: Clipboard: original_strict . Representative strict pair clipboard_001 . The two histories copy different payloads before converging to the same blank editor $O_{t}$ . The current clipboard contents are not visible in that observation. Applying the same paste action $A_{t}$ yields different editor contents, revealing the branch-specific clipboard state.

Figure 5: Ordered Collection: playing_queue . Representative strict pair ordered_xv2_playing_queue_001 . Branch-specific histories construct different effective queue orders while converging to the same Now Playing observation $O_{t}$ . The shared queue-opening action $A_{t}$ reveals different ordered sequences in the next observation.

Figure 6: Temporal Deadline: start_time_shift . Representative strict pair temporal_sealed_start_time_shift_01 . The histories establish different effective remaining times while eventually producing the same prediction-boundary GUI $O_{t}$ . The same controlled wait-and-probe action $A_{t}$ causes the timer to fire in one branch but not the other. Simulator-side logical-time state is shown only through audit annotations and is not supplied to the predictive-state estimator.

Family	Train	Val.	Construction
Persistent Artifact	898	110	Interaction histories centered on a single tracked artifact. Each example contains at most one state transition affecting the tracked artifact, while operations involving unrelated files or paths may appear as distractors. The target state specifies whether the artifact exists at its source and destination locations.
Clipboard	160	20	Atomic clipboard histories containing a single effective copy event. The examples isolate persistence of clipboard content across subsequent GUI navigation and provide the current clipboard payload as the structured-state target.
Ordered Collection	1,080	96	Order-sensitive interaction histories covering playlist and playing-queue construction, insertion, repositioning, reset, and interleaved collection operations. The target state is the complete effective item ordering at the final observation.
Temporal Deadline	400	48	Independently collected timer histories spanning eight atomic temporal transition types. The trajectories vary timer creation, elapsed time, pause/resume behavior, reset, deletion, and restart operations. The target state specifies whether the timer is active and its effective remaining time at the final observation.
Total	2,538	274	—

Appendix

Table 9: Family-specific training data for predictive-state recovery. The specialist teachers and unified estimators are trained from the same underlying family-specific datasets.

Family	Schema instruction	Canonical state target
Persistent Artifact	Predict the current persistent state of the specified artifact from the ordered Files interaction history while ignoring unrelated artifacts and paths.	{"source_exists": bool, "destination_exists": bool}
Clipboard	Infer the current hidden clipboard state from the complete ordered GUI history and actions.	{"clipboard_type": "text", "clipboard_text": string}
Ordered Collection	Infer the current ordered contents of the specified playlist or playing queue while preserving the exact effective item order.	{"collection_type": string, "ordered_items": [string, ...]}
Temporal Deadline	Infer the current timer state from the ordered visible history, including interaction-visible temporal evidence.	{"timer_active": bool, "remaining_seconds": int}

Appendix

Table 10: Family-specific predictive-state schemas. The same unified estimator is conditioned by

q_{f}

to emit the corresponding canonical structured state.

Family	Train	Epochs	Optim. steps
Persistent Artifact	898	4	452
Clipboard	160	4	80
Ordered Collection	1080	2	270
Temporal Deadline	400	4	200

Appendix

Table 11: Training schedules of the four family-specific specialist teachers. All models use rank-16 LoRA and an effective batch size of eight.

Method	Persistent	Clipboard	Ordered	Temporal	Avg.	Worst
Specialists	91.75 / 85.5	94.75 / 89.5	92.75 / 88.0	92.50 / 88.5	87.88	85.5
Family-balanced Joint CE	96.00 / 92.0	74.75 / 69.5	92.75 / 89.0	85.00 / 84.0	83.63	69.5
MTD (Ours)	89.75 / 86.0	89.00 / 85.0	87.25 / 82.0	88.50 / 86.5	84.88	82.0

Appendix

Table 12: Predictive-state recovery under family-specific specialists, direct family-balanced joint training, and multi-teacher distillation. Each family cell reports BranchExact / PairCorrect (%). Avg. and Worst are computed from PairCorrect across the four state families. Best results are in bold ; second-best results are underlined.

Method	Persistent	Clipboard	Ordered	Temporal	Avg.	Worst
Family-balanced Joint CE	89.50 / 79.50	74.75 / 69.50	87.25 / 79.00	82.50 / 79.00	76.75	69.50
MTD (Ours)	86.75 / 80.00	89.00 / 85.00	81.50 / 71.50	85.75 / 81.00	79.38	71.50

Appendix

Table 13: Matched end-to-end Code2World-8B evaluation of the two unified predictive-state estimators. Each family cell reports BranchExact / PairCorrect (%). Avg. and Worst are computed from final-WM PairCorrect. Both methods use the same complete state-conditioning interface and frozen world model. Best results are in bold ; second-best results are underlined.

Family	Recovered state	Action-resolved consequence $D_{t}^{f}$	Complete model-compatible realization
Persistent Artifact	Source and destination existence of the tracked artifact.	Resolve whether the tracked artifact must be present or absent on the Files surface inspected by the candidate folder or filename-search action.	Compile the result into a Downloads-scoped search surface or the requested folder listing, project context onto that inspected surface, and impose the corresponding visible-row or empty-state constraints.
Clipboard	Current clipboard type and payload.	Resolve the payload inserted by the candidate PASTE action.	Construct a post-action editor surface containing the recovered payload as visible editor content while preserving recognizable editor chrome.
Ordered Collection	Collection type and complete ordered item sequence.	Resolve the playlist or playing-queue state exposed by the candidate action, preserving the supplied sequence without sorting or deduplication.	Construct the corresponding collection surface with the required number and order of visible primary rows and serialize the complete sequence through the WM-compatible representation.
Temporal Deadline	Timer activity and effective remaining time.	Combine timer activity and remaining time with the elapsed duration in the candidate action to resolve none , running , or fired , together with the post-action remainder when applicable.	Compile the resolved timer state into a full-screen notification-shade realization with the corresponding visible timer text, countdown, and control structure.

Appendix

Table 14: Family-specific deterministic realization of recovered predictive state.

D_{t}^{f}

is the action-resolved consequence; the final column describes the complete model-compatible realization.

Component	Code2World-8B	gWorld-8B
Current observation	Application crop with the model-native action visual hint.	Same application crop without the Code2World visual hint.
Action representation	Lowercase model-native action names; point actions use scalar coordinate fields.	Uppercase model-native action names; point actions use coordinate arrays.
Message structure	Fixed system instruction followed by a multimodal user request.	User-only multimodal request.
Rendering convention	Fixed $1080\times 2400$ target canvas.	$360\times 800$ CSS viewport, captured at device scale factor $3$ .
Persistent Artifact	Resolved Files blueprint and projected path/entity context are inserted into the Code2World-native transition request.	The same resolved Files semantics are expressed through the gWorld-native mobile-UI request.
Clipboard	The resolved payload is represented once as authoritative visible editor content.	The same payload is inserted into the gWorld-native editor request.

Appendix

Table 15: WM-specific realization of the same deterministic predictive-state information.

Quantity	Source	Available at prediction time	Used
Current observation $O_{t}$	Current GUI observation	Yes	Yes
Candidate action $A_{t}$	Agent / benchmark action	Yes	Yes
Family/task specification	Fixed task protocol	Yes	Yes
Task/entity context $c_{t}$	Prediction-time task specification, when applicable	Yes	Yes
Recovered state $\widehat{Z}_{t}^{f}$	Learned state estimator	Yes	Yes
Action-resolved consequence $D_{t}^{f}$	Fixed deterministic state/action rules	Yes	Yes

Appendix

Table 16: Information provenance for the learned state-conditioned WM.

Family	WM	Canon.	+Cons.	+Surf.	Full
Persistent	Code2World-8B	1.0	4.0	26.0	78.5
Persistent	gWorld-8B	12.5	14.5	40.5	85.5
Temporal	Code2World-8B	0.0	0.0	0.5	84.0
Temporal	gWorld-8B	0.0	0.0	22.5	88.5

Appendix

Table 17: Progressive realization of a fixed family-specialist recovered state, measured by PairCorrect (%). Canonical

\rightarrow

+Consequence and +Consequence

\rightarrow

+Surface are strictly request-matched comparisons. Full uses the production WM-native interface.

Family	WM	Condition	Sem. B	Surf. B	Full B	Sem. P	Full P
Persistent	Code2World-8B	+Surface	60.5	59.0	42.8	27.5	26.0
Persistent	Code2World-8B	Full	94.0	92.8	88.2	89.5	78.5
Persistent	gWorld-8B	+Surface	70.2	92.0	64.8	41.5	40.5
Persistent	gWorld-8B	Full	93.0	98.8	92.0	87.5	85.5
Temporal	Code2World-8B	+Surface	12.5	58.0	10.5	1.0	0.5
Temporal	Code2World-8B	Full	91.0	98.2	90.0	86.0	84.0

Appendix

Table 18: Decomposition of the archived +Surface and Full results into semantic correctness and surface validity. All values are percentages. No prediction is regenerated and no scoring rule is modified.

Family	WM	Gains	Losses	Net	Semantic-only	Surface-only	Both
Persistent	Code2World-8B	200	18	+182	60	65	75
Persistent	gWorld-8B	117	8	+109	88	20	9
Temporal	Code2World-8B	319	1	+318	167	8	144
Temporal	gWorld-8B	193	2	+191	173	7	13

Appendix

Table 19: Branch-level attribution of the +Surface

\rightarrow

Full transition. “Gains” counts branches changing from official incorrect to correct; “Losses” counts the reverse. The final three columns partition the gains. Each model–family condition contains 400 branches.

Family	State/action resolution	Required visible surface	Context/serialization rule
Persistent Artifact	For a Downloads search, resolve presence from tracked in-scope source and destination paths. For a folder probe, resolve presence at the exact inspected artifact path.	Present search: Downloads-scoped heading and one target result row. Absent search: explicit “No matches in Downloads” and no result row. Present folder: requested folder and one target row. Absent folder: requested folder with no tracked-artifact row.	Visible branches retain artifact identity and the relevant scoped path. Absent branches retain the inspected path and surface mode but omit target identity. The resulting Files blueprint is inserted through the WM-native transition representation.
Clipboard	PASTE deterministically inserts the recovered clipboard payload.	A recognizable editor containing the recovered payload as visible editor-body text.	The payload is serialized once as authoritative post-action content; no additional state is inferred from the screenshot.
Ordered Collection	Preserve the recovered collection type and ordered item sequence exactly.	A playlist-detail or playing-queue surface containing exactly the supplied number of primary rows in the supplied order.	The canonical list remains the authoritative sequence. WM-specific serialization avoids redundant list representations that could duplicate item titles.
Temporal Deadline	Inactive $\rightarrow$ none; active with $\Delta_{t}\geq r_{t}$ $\rightarrow$ fired; active with $\Delta_{t}<r_{t}$ $\rightarrow$ running with $r_{t}-\Delta_{t}$ seconds remaining.	None: notification shade without timer state. Fired: one Clock card with Time’s up and Stop . Running: one Clock/Timer card with the updated countdown and Pause .	The resolved outcome is compiled directly into the native notification request. Code2World uses its fixed-canvas format; gWorld uses a dedicated compact $360\times 800$ notification-shade request.

Appendix

Table 20: Deterministic rules used by the Full state-conditioning interface.

	Code2World-8B	gWorld-8B
Checkpoint	released Code2World-8B	released gWorld-8B
Maximum output tokens	8192	8192
Temperature	0	0
Generation seed	20260912	20260912
Repetition penalty	1.0	1.0
Top- $p$ /Top- $k$ override	None	None

Appendix

Table 21: World-model configuration used for the staged state-interface evaluation.

	Specialists				Predictive State (Ours)
Family	Strict	Cross-app	$d$ -Stress	Interf.	Strict	Cross-app	$d$ -Stress	Interf.
Persistent Artifact	85.5	87.0	87.0	87.0	86.0	75.5	82.5	82.5
Clipboard	89.5	85.5	89.5	89.5	85.0	88.5	85.0	88.5
Ordered Collection	88.0	89.0	87.0	86.0	82.0	83.0	85.5	79.0
Temporal Deadline	88.5	83.5	85.0	87.5	86.5	72.0	88.0	89.5
Average	87.88	86.25	87.13	87.50	84.88	79.75	85.25	84.88

Appendix

Table 22: Predictive-state recovery under the primary strict setting and three distribution shifts. Strict reports PairCorrect (%) over 200 A/B pairs per family; Cross-app,

d

-Stress, and Interference report canonical-state Exact (%) over 200 independent trajectories per family. Because these settings use different evaluation units, Strict percentages should not be directly compared with the distribution-shift percentages.

WM	Metric	Observation	Ours	$\Delta$
Code2World	DINOv2	0.4718	0.6218	+0.1499
	SigLIP	0.7234	0.8552	+0.1318
gWorld	DINOv2	0.5534	0.6685	+0.1151
	SigLIP	0.8021	0.8285	+0.0265

Appendix

Table 23: Next-observation feature similarity on the frozen 100-branch fidelity subset. Each generated prediction is compared with the corresponding recorded next observation.

Backbone	Agent	Agent+WM	Agent+WM+State	Rel. Gain
GPT-5.4	56.03	58.62	60.34	+66.4%
Qwen3-VL-8B	41.38	44.83	47.41	+74.8%

Appendix

Table 24: AndroidWorld task success (%) on the full 116-task benchmark. Code2World-8B is used as the world model in both WM-assisted conditions. The final column reports the additional gain from predictive-state conditioning relative to the improvement obtained by adding the WM alone. Best task-success results are in bold ; second-best results are underlined.

Figure 7: Representative paired AndroidWorld rescue on FilesDeleteFile . The Agent+WM and Agent+WM+State conditions share the same interaction prefix and current observation at the divergence point. Standard Code2World-8B prediction produces a misleading near-name file match and leads the selector to SWIPE UP . Predictive-state conditioning changes the simulated future and leads the selector to SEARCH , followed by successful localization and deletion of the target artifact. The recovered state is supplied only to the world model and is not directly exposed to the action selector.

Explore similar work

Latent World Action Model for GUI Agents
Oct 9, 2026Seungjun Yu, Hojun Choi, Seojeong Park +2World Models Action-Conditioned World Models
Korea Advanced Institute of Science and Technology (KAIST)
GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
Aug 30, 2026Lin Fu, Zheyuan Yang, Tianhui Zhang +5World Model Evaluation World Models
Zhejiang University · Tongji University · University of California, San Diego +3
EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Jul 19, 2026Yaohan Yang, Minglei Shi, Borui Zhang +2Computer-Use Agent Benchmarks GUI Agents
Department of Automation, Tsinghua University

Oct 9, 2026J/K move · Enter open · S save

Latent World Action Model for GUI Agents

Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin+1

Korea Advanced Institute of Science and Technology (KAIST)

World Models Action-Conditioned World Models Latent World Models Efficient Inference for World Action Models

Open paper AskPDF arXiv

Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.

Abstract

Explore similar work

Latent World Action Model for GUI Agents

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding