GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models

Zijun Lin; Zhiyang Deng; Yuzhe Wu; Bihan Wen; Yeying Jin

cs.CVSep 22, 2026

GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models

Authors: Zijun Lin, Zhiyang Deng, Yuzhe Wu, Bihan Wen, Yeying Jin

Organizations: Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR · National University of Singapore

Abstract

Recent game world models support realistic visual simulation and interactive gameplay based on player inputs. However, they typically learn environment dynamics from pixel-level supervision, jointly modeling perception, memory, state transitions, and rendering within a single end-to-end framework. While this design enables open-ended, action-controllable generation, it still falls short of delivering a complete gameplay experience. Games are governed by explicit mechanics, such as health deduction, skill activation, combat rules, and termination conditions. These mechanics depend on precise and consistent state transitions that generative models alone cannot reliably enforce. In contrast, game engines can guarantee such mechanics through hard-coded rules, but provide limited flexibility for player-driven creation. To bridge these paradigms, we introduce GameDirector, the first agentic framework that decouples rule-based gameplay logic from visual rendering. Given player-defined configurations, the framework acts as an intelligent director that interprets visual observations, updates game states, tactically controls NPCs, and enforces gameplay rules. It then translates these decisions into text prompts that guide the video world model to render the resulting gameplay. This separation allows players to configure characters, states, and rules much like a game developer while preserving coherent game mechanics. Experiments on three games, using data collected by our automated gameplay agent, show that GameDirector achieves accurate state tracking, reliable rule following, and improves boss action quality by more than 39.9% over various end-to-end game world model settings. Overall, by externalizing player-controllable game logic, GameDirector establishes a middle ground between hard-coded simulation and generative modeling, enabling more flexible and closed-loop gameplay experiences.

Figures & tables

Figure 1: Overview of GameDirector . Our framework bridges hard-coded rules and generative modeling to enable player-configurable, state-aware, and mechanics-consistent gameplay.

Figure 2: Data Collection Pipeline. An automated gameplay agent collects keyboard and mouse inputs, gameplay recordings, and temporally aligned state information across three games, followed by filtering, synchronization, and annotation to construct the final dataset.

Figure 3: Framework of GameDirector . Player-defined configurations are processed by an agentic control layer that performs visual understanding, state tracking, action planning, and rule enforcement, producing structured prompts for mechanics-consistent world-model generation.

	State Modeling		Boss Control		State Align. (%) $\uparrow$	Mechanics Fidelity (%) $\uparrow$		Decision Quality (%) $\uparrow$
Setting	None	Predictive	Implicit	Explicit	State Align. (%) $\uparrow$	GPT-5.5	Gemini	GPT-5.5	Gemini
(1) ( He et al., 2025 )	$\checkmark$		$\checkmark$		–	$25.1_{\pm 0.7}$	$25.0_{\pm 0.3}$	$28.1_{\pm 0.9}$	$26.4_{\pm 0.2}$
(2) ( Wang et al., 2026 )	$\checkmark$			$\checkmark$	–	$21.4_{\pm 0.2}$	$21.4_{\pm 0.1}$	$48.5_{\pm 0.9}$	$47.2_{\pm 0.3}$
(3) ( Li et al., 2026a )		$\checkmark$	$\checkmark$		$70.38$	$34.3_{\pm 0.5}$	$34.1_{\pm 0.4}$	$27.1_{\pm 0.4}$	$26.6_{\pm 0.3}$
(4) ( Lin et al., 2026 )		$\checkmark$		$\checkmark$	$72.01$	$28.1_{\pm 0.3}$	$28.2_{\pm 0.1}$	$44.3_{\pm 1.0}$	$44.6_{\pm 0.1}$
GameDirector (Ours)	Agentic		Agentic		$\mathbf{100.0}$	$\mathbf{98.5_{\pm 0.2}}$	$\mathbf{99.6_{\pm 0.2}}$	$\mathbf{88.4_{\pm 0.7}}$	$\mathbf{93.0_{\pm 0.6}}$

Table 1: Comparison of different state modeling and boss control strategies. Each setting adopts a representative method with minimal adaptation, and all methods are trained using the same dataset. We use

N=3

VLM evaluations and “–” denotes metrics that are not applicable to a given method.

	AttackNet		SituationNet
Game	Macro-F1 $\uparrow$	Accuracy (%) $\uparrow$	Dist. MAE $\downarrow$	Angle MAE ( ∘ ) $\downarrow$	Front-Cone (%) $\uparrow$	Decision-Zone (%) $\uparrow$	Tactical-Sector (%) $\uparrow$
Game N	$0.8476$	$90.53$	$0.3556$	$10.91$	$89.29$	$87.06$	$86.95$
Game V	$0.9135$	$92.48$	$0.1876$	$2.22$	$96.72$	$96.31$	$96.31$
Game H	$0.8871$	$96.11$	$0.2637$	$6.39$	$96.49$	$96.49$	$96.49$
Average	$\mathbf{0.8906}$	$\mathbf{92.94}$	$\mathbf{0.2690}$	$\mathbf{6.51}$	$\mathbf{94.17}$	$\mathbf{93.29}$	$\mathbf{93.25}$

Table 2: Performance of AttackNet and SituationNet in Visual Understanding across three games.

Figure 4: Comparison of gameplay trajectories under different player-defined state initializations. The UI overlays are added only for visualization and dynamically reflect the explicit states tracked by the agentic control layer. Please zoom in for better visibility.

Figure 5: Cross-character skill transfer with GameDirector. Skills originally associated with one boss can be assigned to different bosses through player-configurable initialization.

Figure 6: Adaptive boss action planning. GameDirector selects boss skills based on the spatial relation between the player and boss, adapting actions across different distances.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Game	Train	Eval.	Total	Share	Hours
Game N	22,605	100	22,705	43.8%	31.53
Game V	18,557	100	18,657	36.0%	25.91
Game H	10,324	100	10,424	20.1%	14.48
Total	51,486	300	51,786	100.0%	71.93

Appendix

Table 5: Dataset statistics. Percentages are computed over all 51,786 clips, and hours are based on five-second clips.

Role	Game	Entity	Clips	Hours	Role	Game	Entity	Clips	Hours
Player	N	Green Scout	5,835	8.10	Boss	N	Golden Bird	10,642	14.78
Player	N	Grey Monk	11,284	15.67	Boss	N	Mutant Soldier	7,148	9.93
Player	N	Red Knight	5,586	7.76	Boss	N	Plague Leader	4,915	6.83
Player	V	Vampire Lord	18,657	25.91	Boss	V	Harpy Queen	18,657	25.91
Player	H	the Knight	10,424	14.48	Boss	H	Hornet	3,465	4.81
					Boss	H	Broken Vessel	3,308	4.59

Appendix

Table 6: Per-character and per-boss coverage in the dataset.

Game	Boss	Special Skill	Role	Duration / cooldown (s)	Best range
N	Golden Bird	Hammer Smash	area punish	7.00 / 14.50	1.1–5.5
N	Golden Bird	Jump Slam	short engage	4.50 / 18.25	0.0–3.0
N	Golden Bird	Spin Attack	area denial	9.25 / 20.00	2.5–7.0
N	Mutant Soldier	Back Spin	rear punish	3.75 / 5.50	1.0–4.0
N	Mutant Soldier	Front Spin	frontal sweep	4.25 / 4.75	2.0–7.0
N	Mutant Soldier	Running Slash	far engage	1.50 / 5.25	5.5–9.0

Appendix

Table 7: Special-skill cards used for action planning.

Panel of Figure 7 — Figure 7: VLM prompt for rollout Mechanics Fidelity evaluation.

Panel of Figure 8 — Figure 8: Condensed text-only LLM prompt for Decision Quality.

		Decision Quality
		GPT-5.5			Gemini-3.1-Pro
Setting	Boss	Action Valid. (%) $\uparrow$	Seq. Fit $\uparrow$	Ours Pref. (%) $\uparrow$	Action Valid. (%) $\uparrow$	Seq. Fit $\uparrow$	Ours Pref. (%) $\uparrow$
(1) Implicit No State	Golden Bird	$54.1_{\pm 1.6}$	$2.51_{\pm 0.01}$	$100.0_{\pm 0.0}$	$49.1_{\pm 1.1}$	$2.62_{\pm 0.05}$	$99.0_{\pm 1.7}$
	Mutant Soldier	$14.3_{\pm 0.7}$	$1.70_{\pm 0.02}$	$93.9_{\pm 0.0}$	$13.5_{\pm 0.3}$	$1.61_{\pm 0.16}$	$93.9_{\pm 0.0}$
	Plague Leader	$13.4_{\pm 1.2}$	$1.68_{\pm 0.04}$	$91.7_{\pm 1.8}$	$14.4_{\pm 0.6}$	$1.70_{\pm 0.13}$	$90.6_{\pm 0.0}$
	Overall	$28.1_{\pm 0.9}$	$1.98_{\pm 0.02}$	$\mathbf{95.3_{\pm 0.6}}$	$26.4_{\pm 0.2}$	$2.00_{\pm 0.11}$	$\mathbf{94.6_{\pm 0.6}}$

Appendix

Table 8: Boss-level breakdown of decision quality across the five rollout settings in Game N. Action Validity is reported as a percentage, Sequence Fit is scored from 0 to 5, and Ours Pref. denotes the percentage of matched rollouts in which GameDirector outperforms the baseline setting. Results are mean

\pm

standard deviation over three evaluations.

Component	Device or Deployment	Mean Latency / Speed	Calls
DiT world model	GPU 0	225.060 ms/cell	220
AttackNet	GPU 1	15.108 ms/call	221
SituationNet	GPU 2	17.749 ms/call	22
Gemma-4-E2B-it	GPU 3	2230.264 ms/request	11
Agentic rollout	Four GPUs	17.449 FPS	10
Agentic rollout	Two GPUs	17.277 FPS	10

Appendix

Table 9: Deployment configuration and steady-state inference speed. Component latencies are reported for the four-GPU deployment. Throughput is averaged over ten measured rollouts after one excluded warm-up rollout.

Figure 9: Additional visualization of player-configurable initialization. Varying boss HP, boss attack strength, and player attack strength results in distinct gameplay trajectories and outcomes, consistently reflecting the configured game states over long-horizon rollouts.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 206 papers

Wizard: Manipulate Any World in Real Time via Agentic Control
Oct 29, 2026Yuzhe Wu, Zijun Lin, Yeying JinInteractive World Models Real-Time Interactive Video Generation
Tencent · NUS · NTU
StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Jul 29, 2026Zijun Lin, Zeqing Wang, Cheston Tan +2World Models Action-Conditioned World Models
Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab

Oct 29, 2026J/K move · Enter open · S save

Wizard: Manipulate Any World in Real Time via Agentic Control

Yuzhe Wu, Zijun Lin, Yeying Jin

Tencent · NUS · NTU

Interactive World Models Real-Time Interactive Video Generation Agentic Video Generation Video World Models Action-Conditioned Video Generation

Open paper AskPDF arXiv

Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.

Abstract

Game-Playing Agents

Explore similar work

Wizard: Manipulate Any World in Real Time via Agentic Control

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation

Programmable World Model