WorldMind: Decoupled Game World Model for State-Aware NPC Behavior

Zhiyang Deng; Boran Zhang; Danze Chen; Yeying Jin

cs.CVAug 18, 2026

WorldMind: Decoupled Game World Model for State-Aware NPC Behavior

Authors: Zhiyang Deng, Boran Zhang, Danze Chen, Yeying Jin

Organizations: Tencent · National University of Singapore

Abstract

Game world models have recently demonstrated promising capabilities in generating visually coherent and action-controllable gameplay videos. However, non-player character (NPC) behavior in existing models is either implicitly entangled with video generation or explicitly prescribed through external control signals. Consequently, a game world model has to jointly understand the state, plan the NPC's response and render its visual outcome, limiting its ability to produce responsive and state-aware NPC behavior. The challenge lies in the lack of an explicit interface for state-grounded decision-making. To this end, we introduce WorldMind, to our knowledge the first decoupled framework for state-aware NPC behavior in game world models. WorldMind separates interactive world modeling into four layers: an Understanding Layer that constructs a compact state from generated frames; a Decision Layer that reasons over the compact state to plan the NPC's next action; a Control Layer that translates the actions into temporally aligned conditions; and a Generation Layer that synthesizes their visual outcomes. By reconnecting layers in a closed interaction loop, WorldMind grounds NPC behavior in the evolving game state. We further introduce BOSS-140K, a dataset of gameplay videos paired with rich internal game states, together with an agent that automates the collection at scale. Experiments on BOSS-140K demonstrate reliable compact state reconstruction and mechanics-grounded planning, with WorldMind preferred over the baselines in approximately 70% of pairwise comparisons for its more tactically appropriate and coherent NPC behavior. Project page: https://teawhite.cn/worldmind_projectpage/

Figures & tables

Figure 1: Comparison with baseline NPC-control schemes. Using the same Wan base generator, the two baselines either leave the NPC action implicit in video generation (top left) or require it to be externally specified (top right). WorldMind instead constructs a compact state and reasons over it to plan the NPC’s next action within the interaction loop, producing responsive, state-aware NPC behavior (bottom). Green and red bounding boxes denote the player and NPC, respectively.

Figure 2: Automated construction of BOSS-140K. A state-conditioned gameplay agent combines engine-internal state with game-specific strategies to collect diverse interactions without manual operation. The pipeline records gameplay video, internal state, and keyboard/mouse input on a shared timeline to produce frame-aligned annotations (see above).

Figure 3: Overview of the WorldMind framework. WorldMind connects four decoupled layers in a closed interaction loop. From the generated frames and boss action history, L1 constructs a compact state and passes it to L2, which reasons over the compact state and skill mechanics to plan the boss’s next actions. L3 pairs the resulting boss action with player behavior obtained through either Direct or Director Control and converts both into a temporally aligned compositional natural-language conditioning sequence. Guided by this sequence, L4 synthesizes the next gameplay segment, whose frames are fed back to L1 to begin the next cycle of understanding, reasoning, and generation.

Figure 4: Qualitative examples of boss behavior grounded in the compact state. The top, middle, and bottom rows show Game A , The Binding of Isaac , and Hollow Knight , respectively. Each row presents the compact state, the resulting boss action and optional rationale, the compositional prompt, and the generated outcome. Together, the examples show how the framework operates in a closed loop with visual outcomes. Green and red bounding boxes denote the player and boss, respectively.

Game	Dist. $\downarrow$	Rel. $\downarrow$	Ang. $\downarrow$	D-bin $\uparrow$	A-bin $\uparrow$
Game A	0.312	2.9%	12.66	0.807	0.702
Hollow Knight	0.444	6.7%	2.96	0.772	0.996
Isaac	8.187	8.4%	29.38	0.904	0.839

Table 1: Compact-state reconstruction. Distance is measured on game-specific scales, while Rel. normalizes Dist. by each game’s median distance for cross-game comparison.

Game	$N$	No-desc $\Delta$	Swap $\Delta$	Follow	Stay
Game A	200	52.0	83.5	69.0	16.5
Hollow Knight	200	90.0	87.0	73.5	13.0
Isaac	121	55.4	80.2	79.3	19.8

Table 2: Mechanics-grounded skill selection. No-desc / Swap

\Delta

report first-skill change rates after removing/swapping descriptions. Under a swap, Follow moves to the option carrying the original choice’s mechanics.

		GPT-5.5			Gemini-3.1-pro
Method	NPC Behavior	Ours Pref. (%) $\uparrow$	Action Valid. (%) $\uparrow$	Sequence Fit (0–5) $\uparrow$	Ours Pref. (%) $\uparrow$	Action Valid. (%) $\uparrow$	Sequence Fit (0–5) $\uparrow$
Wan w/o NPC Control	Implicit	71.5	63.6	3.28	69.8	70.6	3.20
Wan w/ NPC Control	Explicit	70.9	63.6	3.22	70.3	70.9	3.17
Ours	State-Aware	N/A	74.0	3.85	N/A	77.6	4.00

Table 3: Closed-loop NPC behavior evaluation. One-minute Game A rollouts evaluated by two LLM judges. All systems use the same Wan 2.2 backbone and training corpus. Wan w/o NPC Control uses the player-only generator, whereas Wan w/ NPC Control and WorldMind use the boss-conditioned generator. All results are averaged over three independent judging passes to reduce LLM-judge variability. Bold values indicate the best result in each column.

Game	Method	Dist. $\downarrow$	Ang. $\downarrow$	D-bin $\uparrow$	A-bin $\uparrow$
Game A	Ours	0.312	12.66	0.807	0.702
	DINOv2-S	0.722	19.16	0.601	0.606
	VideoMAE-B	0.750	16.81	0.611	0.597
Hollow	Ours	0.444	2.96	0.772	0.996
Knight	DINOv2-S	0.707	11.72	0.610	0.934
	VideoMAE-B	0.830	18.60	0.549	0.836

Table 4: Visual encoder ablation. Boss–player geometry reconstruction using task-trained and frozen encoders under matched three-frame inputs.

Game	Text	Text + Image	$\Delta$
Game A	69.0	69.5	$+0.5$
Hollow Knight	73.5	67.0	$-6.5$
Isaac	79.3	71.1	$-8.2$

Table 5: Image-modality ablation. Mechanics-following rates (%) using text alone or augmented with the frame.

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 122 papers

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Jul 29, 2026Zijun Lin, Zeqing Wang, Cheston Tan +2World Models Action-Conditioned World Models
Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1
ReactiveGWM: Flexible Control and NPC Reactivity in Game World Models
May 14, 2026Zeqing Wang, Danze Chen, Zhaohu Xing +4World Models Multi-Agent Control
Tencent · National University of Singapore · The Hong Kong University of Science and Technology (Guangzhou) +3
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab

Jul 29, 2026J/K move · Enter open · S save

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Zijun Lin, Zeqing Wang, Cheston Tan, Bihan Wen+1

Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR+1

World Models Action-Conditioned World Models Video World Models

Open paper AskPDF arXiv

Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.

Abstract

Interactive World Models

Explore similar work

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

ReactiveGWM: Flexible Control and NPC Reactivity in Game World Models

Programmable World Model