GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models

Zijun Lin; Zhiyang Deng; Yuzhe Wu; Bihan Wen; Yeying Jin

cs.CVSep 22, 2026

GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models

Authors: Zijun Lin, Zhiyang Deng, Yuzhe Wu, Bihan Wen, Yeying Jin

Organizations: Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR · National University of Singapore

Abstract

Recent game world models support realistic visual simulation and interactive gameplay based on player inputs. However, they typically learn environment dynamics from pixel-level supervision, jointly modeling perception, memory, state transitions, and rendering within a single end-to-end framework. While this design enables open-ended, action-controllable generation, it still falls short of delivering a complete gameplay experience. Games are governed by explicit mechanics, such as health deduction, skill activation, combat rules, and termination conditions. These mechanics depend on precise and consistent state transitions that generative models alone cannot reliably enforce. In contrast, game engines can guarantee such mechanics through hard-coded rules, but provide limited flexibility for player-driven creation. To bridge these paradigms, we introduce GameDirector, the first agentic framework that decouples rule-based gameplay logic from visual rendering. Given player-defined configurations, the framework acts as an intelligent director that interprets visual observations, updates game states, tactically controls NPCs, and enforces gameplay rules. It then translates these decisions into text prompts that guide the video world model to render the resulting gameplay. This separation allows players to configure characters, states, and rules much like a game developer while preserving coherent game mechanics. Experiments on three games, using data collected by our automated gameplay agent, show that GameDirector achieves accurate state tracking, reliable rule following, and improves boss action quality by more than 39.9% over various end-to-end game world model settings. Overall, by externalizing player-controllable game logic, GameDirector establishes a middle ground between hard-coded simulation and generative modeling, enabling more flexible and closed-loop gameplay experiences.

Figures & tables

Figure 1: Overview of GameDirector . Our framework bridges hard-coded rules and generative modeling to enable player-configurable, state-aware, and mechanics-consistent gameplay.

Figure 2: Data Collection Pipeline. An automated gameplay agent collects keyboard and mouse inputs, gameplay recordings, and temporally aligned state information across three games, followed by filtering, synchronization, and annotation to construct the final dataset.

Figure 3: Framework of GameDirector . Player-defined configurations are processed by an agentic control layer that performs visual understanding, state tracking, action planning, and rule enforcement, producing structured prompts for mechanics-consistent world-model generation.

	State Modeling		Boss Control		State Align. (%) $\uparrow$	Mechanics Fidelity (%) $\uparrow$		Decision Quality (%) $\uparrow$
Setting	None	Predictive	Implicit	Explicit	State Align. (%) $\uparrow$	GPT-5.5	Gemini	GPT-5.5	Gemini
(1) ( He et al., 2025 )	$\checkmark$		$\checkmark$		–	$25.1_{\pm 0.7}$	$25.0_{\pm 0.3}$	$28.1_{\pm 0.9}$	$26.4_{\pm 0.2}$
(2) ( Wang et al., 2026 )	$\checkmark$			$\checkmark$	–	$21.4_{\pm 0.2}$	$21.4_{\pm 0.1}$	$48.5_{\pm 0.9}$	$47.2_{\pm 0.3}$
(3) ( Li et al., 2026a )		$\checkmark$	$\checkmark$		$70.38$	$34.3_{\pm 0.5}$	$34.1_{\pm 0.4}$	$27.1_{\pm 0.4}$	$26.6_{\pm 0.3}$
(4) ( Lin et al., 2026 )		$\checkmark$		$\checkmark$	$72.01$	$28.1_{\pm 0.3}$	$28.2_{\pm 0.1}$	$44.3_{\pm 1.0}$	$44.6_{\pm 0.1}$
GameDirector (Ours)	Agentic		Agentic		$\mathbf{100.0}$	$\mathbf{98.5_{\pm 0.2}}$	$\mathbf{99.6_{\pm 0.2}}$	$\mathbf{88.4_{\pm 0.7}}$	$\mathbf{93.0_{\pm 0.6}}$

Table 1: Comparison of different state modeling and boss control strategies. Each setting adopts a representative method with minimal adaptation, and all methods are trained using the same dataset. We use

N=3

VLM evaluations and “–” denotes metrics that are not applicable to a given method.

	AttackNet		SituationNet
Game	Macro-F1 $\uparrow$	Accuracy (%) $\uparrow$	Dist. MAE $\downarrow$	Angle MAE ( ∘ ) $\downarrow$	Front-Cone (%) $\uparrow$	Decision-Zone (%) $\uparrow$	Tactical-Sector (%) $\uparrow$
Game N	$0.8476$	$90.53$	$0.3556$	$10.91$	$89.29$	$87.06$	$86.95$
Game V	$0.9135$	$92.48$	$0.1876$	$2.22$	$96.72$	$96.31$	$96.31$
Game H	$0.8871$	$96.11$	$0.2637$	$6.39$	$96.49$	$96.49$	$96.49$
Average	$\mathbf{0.8906}$	$\mathbf{92.94}$	$\mathbf{0.2690}$	$\mathbf{6.51}$	$\mathbf{94.17}$	$\mathbf{93.29}$	$\mathbf{93.25}$

Table 2: Performance of AttackNet and SituationNet in Visual Understanding across three games.

Figure 4: Comparison of gameplay trajectories under different player-defined state initializations. The UI overlays are added only for visualization and dynamically reflect the explicit states tracked by the agentic control layer. Please zoom in for better visibility.

Figure 5: Cross-character skill transfer with GameDirector. Skills originally associated with one boss can be assigned to different bosses through player-configurable initialization.

Figure 6: Adaptive boss action planning. GameDirector selects boss skills based on the spatial relation between the player and boss, adapting actions across different distances.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Game	Train	Eval.	Total	Share	Hours
Game N	22,605	100	22,705	43.8%	31.53
Game V	18,557	100	18,657	36.0%	25.91
Game H	10,324	100	10,424	20.1%	14.48
Total	51,486	300	51,786	100.0%	71.93

Appendix

Table 5: Dataset statistics. Percentages are computed over all 51,786 clips, and hours are based on five-second clips.

Role	Game	Entity	Clips	Hours	Role	Game	Entity	Clips	Hours
Player	N	Green Scout	5,835	8.10	Boss	N	Golden Bird	10,642	14.78
Player	N	Grey Monk	11,284	15.67	Boss	N	Mutant Soldier	7,148	9.93
Player	N	Red Knight	5,586	7.76	Boss	N	Plague Leader	4,915	6.83
Player	V	Vampire Lord	18,657	25.91	Boss	V	Harpy Queen	18,657	25.91
Player	H	the Knight	10,424	14.48	Boss	H	Hornet	3,465	4.81
					Boss	H	Broken Vessel	3,308	4.59

Appendix

Table 6: Per-character and per-boss coverage in the dataset.

Game	Boss	Special Skill	Role	Duration / cooldown (s)	Best range
N	Golden Bird	Hammer Smash	area punish	7.00 / 14.50	1.1–5.5
N	Golden Bird	Jump Slam	short engage	4.50 / 18.25	0.0–3.0
N	Golden Bird	Spin Attack	area denial	9.25 / 20.00	2.5–7.0
N	Mutant Soldier	Back Spin	rear punish	3.75 / 5.50	1.0–4.0
N	Mutant Soldier	Front Spin	frontal sweep	4.25 / 4.75	2.0–7.0
N	Mutant Soldier	Running Slash	far engage	1.50 / 5.25	5.5–9.0

Appendix

Table 7: Special-skill cards used for action planning.

Panel of Figure 7 — Figure 7: VLM prompt for rollout Mechanics Fidelity evaluation.

Panel of Figure 8 — Figure 8: Condensed text-only LLM prompt for Decision Quality.

		Decision Quality
		GPT-5.5			Gemini-3.1-Pro
Setting	Boss	Action Valid. (%) $\uparrow$	Seq. Fit $\uparrow$	Ours Pref. (%) $\uparrow$	Action Valid. (%) $\uparrow$	Seq. Fit $\uparrow$	Ours Pref. (%) $\uparrow$
(1) Implicit No State	Golden Bird	$54.1_{\pm 1.6}$	$2.51_{\pm 0.01}$	$100.0_{\pm 0.0}$	$49.1_{\pm 1.1}$	$2.62_{\pm 0.05}$	$99.0_{\pm 1.7}$
	Mutant Soldier	$14.3_{\pm 0.7}$	$1.70_{\pm 0.02}$	$93.9_{\pm 0.0}$	$13.5_{\pm 0.3}$	$1.61_{\pm 0.16}$	$93.9_{\pm 0.0}$
	Plague Leader	$13.4_{\pm 1.2}$	$1.68_{\pm 0.04}$	$91.7_{\pm 1.8}$	$14.4_{\pm 0.6}$	$1.70_{\pm 0.13}$	$90.6_{\pm 0.0}$
	Overall	$28.1_{\pm 0.9}$	$1.98_{\pm 0.02}$	$\mathbf{95.3_{\pm 0.6}}$	$26.4_{\pm 0.2}$	$2.00_{\pm 0.11}$	$\mathbf{94.6_{\pm 0.6}}$

Appendix

Table 8: Boss-level breakdown of decision quality across the five rollout settings in Game N. Action Validity is reported as a percentage, Sequence Fit is scored from 0 to 5, and Ours Pref. denotes the percentage of matched rollouts in which GameDirector outperforms the baseline setting. Results are mean

\pm

standard deviation over three evaluations.

Component	Device or Deployment	Mean Latency / Speed	Calls
DiT world model	GPU 0	225.060 ms/cell	220
AttackNet	GPU 1	15.108 ms/call	221
SituationNet	GPU 2	17.749 ms/call	22
Gemma-4-E2B-it	GPU 3	2230.264 ms/request	11
Agentic rollout	Four GPUs	17.449 FPS	10
Agentic rollout	Two GPUs	17.277 FPS	10

Appendix

Table 9: Deployment configuration and steady-state inference speed. Component latencies are reported for the four-GPU deployment. Throughput is averaged over ten measured rollouts after one excluded warm-up rollout.

Figure 9: Additional visualization of player-configurable initialization. Varying boss HP, boss attack strength, and player attack strength results in distinct gameplay trajectories and outcomes, consistently reflecting the configured game states over long-horizon rollouts.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 206 papers

Oct 29, 2026

Wizard: Manipulate Any World in Real Time via Agentic Control

Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.

Yuzhe Wu, Zijun Lin, Yeying Jin

Tencent · NUS · NTU

Interactive World Models Real-Time Interactive Video Generation Agentic Video Generation Video World Models Action-Conditioned Video Generation

Ask PDF

Jul 29, 2026

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.

Zijun Lin, Zeqing Wang, Cheston Tan +2

Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1

World Models Action-Conditioned World Models Video World Models

Ask PDF

Sep 9, 2026

Programmable World Model

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8

Alaya Lab

World Models Controllable Video Generation World Model Evaluation Interactive World Models Video World Models

Ask PDF

Abstract

Game-Playing Agents

Explore similar work

Wizard: Manipulate Any World in Real Time via Agentic Control

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Programmable World Model