Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models

Pengfei Zhou; Hexin Wang; Zhengfeiyang Zhang; Yixing Ma; Zhenglin Wan; Kaipeng Zhang; Wangbo Zhao; Yang You

cs.AIAug 26, 2026

Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models

Authors: Pengfei Zhou, Hexin Wang, Zhengfeiyang Zhang, Yixing Ma, Zhenglin Wan, Kaipeng Zhang, Wangbo Zhao, Yang You

Organizations: National University of Singapore, HPC-AI Lab · InfRec, Cardinal AI Lab · University of California, Berkeley · Independent Researcher ·

Abstract

A common strategy for scaling world models is to train on more crawled video with more compute. We argue that this strategy is inefficient: scaling world models also requires a recursive data engine that offers grounded reward signals. The success of code agents illustrates why this matters. As code is executable, compilers and runtimes can provide high-quality rewards for Reinforcement Learning (RL) post-training of LLMs. By contrast, spatial generation still relies largely on fuzzy proxies such as CLIP scores. These signals are fuzzy and biased, making them hard to support RL post-training. Compared with these, game development provides a missing reward environment for spatial world models. A scene encoded by a game engine is an executable world specification: the engine can efficiently check collision, physics, navigability and bounded playability, while the developer provides the global verification signal by judging whether the scene should be accepted. Game development also provides real-world long-horizon trajectory data for RL post-training. We therefore propose Reinforcement Learning with Human-Engine Verification (RLHEV), a post-training paradigm that combines dense engine signals with implicit human acceptance feedback from the development process.

Figures & tables

Figure 1: From subjective reward proxies to a human-engine verification loop. Conventional spatial-data pipelines convert sparse human or model annotations into noisy final-output rewards. A game-development loop can instead pair human intent and feedback with engine execution, development trajectories, rendered evidence, and localized verifier failures, allowing future builders to improve from both grounded checks and developer judgment based on traces rather than final artifacts alone.

Figure 2: Fuzzy reward in unverifiable spatial generation versus human-engine feedback in game development. The top pipeline illustrates how model-as-a-judge and human aesthetic variance can produce ambiguous rewards and performance bottleneck. The bottom pipeline turns assets into executable worlds: game engine and internal test suites expose local failures, rendered evidence supports review, and developer supplies the final signal for acceptance.

Figure 3: A shared executable scene-program representation for understanding and generation. Game creation projects human world understanding, design intent, constraints, and playable experience into the model’s understanding and generation capacity. Generation maps text intent, design goals, or task specifications to the scene program; understanding reads the same object in reverse. The engine checks it through rendering, physics rollouts, and temporal consistency, while humans judge whether the resulting world preserves intent and workflow value.

Figure 4: Unity asset classification evaluation on UnitySceneBench . Bars represent the primary score, and markers denote accuracy and AUC. This figure reports the best performance for each method across eight random seeds. Full RLHEV obtains the highest best-of-eight primary score (

0.681

), with

0.665

accuracy,

0.665

balanced accuracy,

0.733

F1, and

0.690

AUC.

Figure 5: Scaling with different numbers of training samples and tested on UnitySceneBench . (a) reports the primary score for Fuzzy Proxies Baseline, SFT Baseline, Offline RLHF, Engine-based RLVR, and Full RLHEV on the test set, as mean

\pm

std over eight seeds. (b) Unity assets generation uses the same training budgets and reports mean

\pm

std over eight seeds per method and budget.

Figure 6: Generalization and embodied diagnostic evaluations. (a) Generalization under distribution and engine shifts. Bars report the MLLM-as-a-judge score in

[0,1]

for training from scratch and target-adapted from a pretrained checkpoint. Pretrained on source data and adapted on the target distribution achieve significant gain; the cross-engine settings show smaller but measurable positive signal. (b) Embodied diagnostic evaluation on R2R, Gymnasium MuJoCo, and D4RL Gym-MuJoCo. Values are relative improvements over the original baseline after normalizing scores (positive is better for all benchmarks). Naive aug. denotes standard data augmentation, whereas AWoMo-augmented denotes the policy performance trained with AWoMo-generated data.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Example value	Functionality
$b$	Arrange a playable storage room with a clear path to the exit.	Conditions generation on design intent.
$o_{t}$	crate_07 ; relation near(chair_03) .	Localizes checks and repairs to objects or regions.
$s_{t}$	Transform, collider bounds, material tags, navmesh status, and visibility flags.	Grounds the model in executable scene state.
$a_{t}$	Move crate_07 to $(2.4,0.0,2.6)$ and regenerate its collider.	Supervises next-edit and repair prediction.
$g_{t}$	Collision fail before repair; navmesh pass after repair.	Supplies verifier gates and dense rewards.
$v_{t}$	Rendered preview, object mask, and camera metadata.	Provides reviewer-visible evidence.

Appendix

Table 1: Example fields in one UWDP instance. The values illustrate how stable object IDs connect intent, scene state, engine checks, repairs, rendered evidence, and review.

Figure 7: Protocol traces preserve the process data that is absent from final snapshots. A final scene records only the terminal state, whereas a trace records intent, intermediate scene states, failed checks, repair edits, validation outcomes, and engine rewards. This object-linked sequence provides a training objective for asset diagnosis and repair.

Figure 8: Source-side trace transfer under an 8-label target budget. Values are held-out Spearman correlations over 8 seeds and two target engines. Protocol-trace features provide substantially stronger target-engine ranking signal than coarse final-snapshot features; increasing the number of Unity source instances gives a smaller additional gain.

Name	Definition
Zero-shot CLIP	A visual-proxy baseline. It classifies the acceptance or rejection of candidates using a threshold on CLIP similarity, without a learned task model or RL.
Fuzzy Proxies Baseline	The baseline model trained with CLIP similarity rewards. It tests whether visual-proxy rewards help the supervised baseline.
SFT Baseline	The baseline model trained only with supervised GT accept/reject labels from the training split. The GT is the binary accepted/rejected target label. It is excluded from the model input, and no RL reward is used.
Offline RLHF	Offline RL using only the human-review reward, without engine reward.
Engine-based RLVR	Offline RL using only engine-verification reward, without the human-review reward.
Full RLHEV	The full model trained with fused human-review and engine rewards, using weights $0.65$ and $0.35$ .

Appendix

Table 2: Method definitions for the UnitySceneBench comparison.

Method	40	80	160	320	640	720
Fuzzy Proxies Baseline	0.7478	0.7246	0.7154	0.7316	0.7127	0.7174
SFT Baseline	0.7651	0.7423	0.7270	0.7362	0.7486	0.7441
Offline RLHF	0.7735	0.7558	0.7449	0.7515	0.7589	0.7652
Engine-based RLVR	0.7430	0.7165	0.7137	0.7550	0.7904	0.7934
Full RLHEV	0.8002	0.7833	0.7821	0.7936	0.8106	0.8197

Appendix

Table 3: Unity assets generation quality for Figure 5 . Values are mean quality over eight seeds per method for each training budget.

Category	Transfer	Condition	Assert	Proxy	Engine acc.	Loss
Distribution	Unity $\rightarrow$ held-out Unity	zero-shot	$0.25$	$0.451$	$0.007$	$11.685$
Distribution	Unity $\rightarrow$ held-out Unity	target	$\mathbf{0.75}$	$0.449$	$0.008$	$11.590$
Distribution	Unity $\rightarrow$ held-out Unity	scratch	$0.25$	$0.421$	$0.005$	$11.909$
Cross-engine	Unity $\rightarrow$ Unreal	target	$0.35$	$0.474$	$0.008$	$11.596$
Cross-engine	Unity $\rightarrow$ Unreal	scratch	$0.25$	$0.437$	$0.009$	$11.825$
Cross-engine	Unity $\rightarrow$ Godot	target	$0.35$	$0.478$	$0.007$	$11.575$

Appendix

Table 4: Generalization aggregate. Target means target-adapted transfer; scratch means target-only scratch training.

Transfer	Condition	MLLM judge	Proxy	Loss
Unity $\rightarrow$ Unreal	zero-shot	$0.25$	$0.396$	$12.377$
Unity $\rightarrow$ Unreal	Unity-source only	$0.15$	$0.005$	$16.693$
Unity $\rightarrow$ Unreal	target-adapted transfer	$0.35$	$0.474$	$11.596$
Unity $\rightarrow$ Unreal	scratch	$0.25$	$0.437$	$11.825$
Unity $\rightarrow$ Godot	zero-shot	$0.25$	$0.396$	$12.375$
Unity $\rightarrow$ Godot	Unity-source only	$0.15$	$0.005$	$16.695$

Appendix

Table 5: Cross-engine target-adapted transfer results. Target-adapted transfer gives measurable positive signal over scratch for both target engines, with higher proxy scores and lower losses.

Benchmark	Metric	Original	Naive aug.	AWoMo-aug.	$\Delta$	Best
R2R	Success rate $\uparrow$	$76.006{\pm}0.133$	$76.176{\pm}0.175$	$76.607{\pm}0.094$	$+0.79\%$	AWoMo
Gymnasium MuJoCo	Rollout return $\uparrow$	$1568.44{\pm}1756.69$	$1648.03{\pm}1689.15$	$1724.73{\pm}1642.60$	$+9.96\%$	AWoMo
D4RL Gym-MuJoCo	Normalized score $\uparrow$	$18.295{\pm}14.663$	$25.555{\pm}12.448$	$27.155{\pm}9.491$	$+48.43\%$	AWoMo

Appendix

Table 6: Final embodied generalization results. Values are mean

\pm

std where available.

\Delta

is the direction-normalized relative improvement of AWoMo-augmented training over the original baseline.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 419 papers

Scaling Automatic Research Agents via World Models
Aug 12, 2026Xiyuan Yang, Sheikh Sarwar, Jingru Cheng +8World Model Learning RL Post-Training
University of Illinois Urbana-Champaign · Amazon
RSIGame: Autonomous Agentic Game Development with Recursive Self-improvement
Sep 30, 2026Wenyi Wu, Minghao Fu, Jieyu You +10Agentic Code Generation Self-Improving Agents
University of California San Diego · ByteDance Inc. · Carnegie Mellon University
Multiplayer Interactive World Models with Representation Autoencoders
Jul 6, 2026Anthony Hu, Václav Volhejn, Adrien Ramanana Rahary +24Action-Conditioned World Models Latent World Models
1General Intuition · 2Kyutai · ‡École nationale des ponts et chaussées +1

Aug 12, 2026J/K move · Enter open · S save

Scaling Automatic Research Agents via World Models

Xiyuan Yang, Sheikh Sarwar, Jingru Cheng, Zhan Shi+7

University of Illinois Urbana-Champaign · Amazon

World Model Learning RL Post-Training RL for LLM Agents Autonomous Research Agents Model-Based RL

Open paper AskPDF arXiv

Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, which demonstrates the generalizability of our method.

Abstract

World Model Learning

Explore similar work

Scaling Automatic Research Agents via World Models

RSIGame: Autonomous Agentic Game Development with Recursive Self-improvement

Multiplayer Interactive World Models with Representation Autoencoders