Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models
Authors: Pengfei Zhou, Hexin Wang, Zhengfeiyang Zhang, Yixing Ma, Zhenglin Wan, Kaipeng Zhang, Wangbo Zhao, Yang You
Organizations: National University of Singapore, HPC-AI Lab · InfRec, Cardinal AI Lab · University of California, Berkeley · Independent Researcher · Hong Kong University of Science and Technology
A common strategy for scaling world models is to train on more crawled video with more compute. We argue that this strategy is inefficient: scaling world models also requires a recursive data engine that offers grounded reward signals. The success of code agents illustrates why this matters. As code is executable, compilers and runtimes can provide high-quality rewards for Reinforcement Learning (RL) post-training of LLMs. By contrast, spatial generation still relies largely on fuzzy proxies such as CLIP scores. These signals are fuzzy and biased, making them hard to support RL post-training. Compared with these, game development provides a missing reward environment for spatial world models. A scene encoded by a game engine is an executable world specification: the engine can efficiently check collision, physics, navigability and bounded playability, while the developer provides the global verification signal by judging whether the scene should be accepted. Game development also provides real-world long-horizon trajectory data for RL post-training. We therefore propose Reinforcement Learning with Human-Engine Verification (RLHEV), a post-training paradigm that combines dense engine signals with implicit human acceptance feedback from the development process.
Figures & tables
Figure 1: From subjective reward proxies to a human-engine verification loop. Conventional spatial-data pipelines convert sparse human or model annotations into noisy final-output rewards. A game-development loop can instead pair human intent and feedback with engine execution, development trajectories, rendered evidence, and localized verifier failures, allowing future builders to improve from both grounded checks and developer judgment based on traces rather than final artifacts alone.
Figure 2: Fuzzy reward in unverifiable spatial generation versus human-engine feedback in game development. The top pipeline illustrates how model-as-a-judge and human aesthetic variance can produce ambiguous rewards and performance bottleneck. The bottom pipeline turns assets into executable worlds: game engine and internal test suites expose local failures, rendered evidence supports review, and developer supplies the final signal for acceptance.
Figure 3: A shared executable scene-program representation for understanding and generation. Game creation projects human world understanding, design intent, constraints, and playable experience into the model’s understanding and generation capacity. Generation maps text intent, design goals, or task specifications to the scene program; understanding reads the same object in reverse. The engine checks it through rendering, physics rollouts, and temporal consistency, while humans judge whether the resulting world preserves intent and workflow value.
Figure 4: Unity asset classification evaluation on UnitySceneBench . Bars represent the primary score, and markers denote accuracy and AUC. This figure reports the best performance for each method across eight random seeds. Full RLHEV obtains the highest best-of-eight primary score ( 0.681 ), with 0.665 accuracy, 0.665 balanced accuracy, 0.733 F1, and 0.690 AUC.
Figure 5: Scaling with different numbers of training samples and tested on UnitySceneBench . (a) reports the primary score for Fuzzy Proxies Baseline, SFT Baseline, Offline RLHF, Engine-based RLVR, and Full RLHEV on the test set, as mean ± std over eight seeds. (b) Unity assets generation uses the same training budgets and reports mean ± std over eight seeds per method and budget.
Figure 6: Generalization and embodied diagnostic evaluations. (a) Generalization under distribution and engine shifts. Bars report the MLLM-as-a-judge score in [0,1] for training from scratch and target-adapted from a pretrained checkpoint. Pretrained on source data and adapted on the target distribution achieve significant gain; the cross-engine settings show smaller but measurable positive signal. (b) Embodied diagnostic evaluation on R2R, Gymnasium MuJoCo, and D4RL Gym-MuJoCo. Values are relative improvements over the original baseline after normalizing scores (positive is better for all benchmarks). Naive aug. denotes standard data augmentation, whereas AWoMo-augmented denotes the policy performance trained with AWoMo-generated data.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Example value
Functionality
b
Arrange a playable storage room with a clear path to the exit.
Conditions generation on design intent.
ot
crate_07 ; relation near(chair_03) .
Localizes checks and repairs to objects or regions.
st
Transform, collider bounds, material tags, navmesh status, and visibility flags.
Grounds the model in executable scene state.
at
Move crate_07 to (2.4,0.0,2.6) and regenerate its collider.
Supervises next-edit and repair prediction.
gt
Collision fail before repair; navmesh pass after repair.
Supplies verifier gates and dense rewards.
vt
Rendered preview, object mask, and camera metadata.
Provides reviewer-visible evidence.
Appendix
Table 1: Example fields in one UWDP instance. The values illustrate how stable object IDs connect intent, scene state, engine checks, repairs, rendered evidence, and review.
Figure 7: Protocol traces preserve the process data that is absent from final snapshots. A final scene records only the terminal state, whereas a trace records intent, intermediate scene states, failed checks, repair edits, validation outcomes, and engine rewards. This object-linked sequence provides a training objective for asset diagnosis and repair.
Figure 8: Source-side trace transfer under an 8-label target budget. Values are held-out Spearman correlations over 8 seeds and two target engines. Protocol-trace features provide substantially stronger target-engine ranking signal than coarse final-snapshot features; increasing the number of Unity source instances gives a smaller additional gain.
Name
Definition
Zero-shot CLIP
A visual-proxy baseline. It classifies the acceptance or rejection of candidates using a threshold on CLIP similarity, without a learned task model or RL.
Fuzzy Proxies Baseline
The baseline model trained with CLIP similarity rewards. It tests whether visual-proxy rewards help the supervised baseline.
SFT Baseline
The baseline model trained only with supervised GT accept/reject labels from the training split. The GT is the binary accepted/rejected target label. It is excluded from the model input, and no RL reward is used.
Offline RLHF
Offline RL using only the human-review reward, without engine reward.
Engine-based RLVR
Offline RL using only engine-verification reward, without the human-review reward.
Full RLHEV
The full model trained with fused human-review and engine rewards, using weights 0.65 and 0.35 .
Appendix
Table 2: Method definitions for the UnitySceneBench comparison.
Method
40
80
160
320
640
720
Fuzzy Proxies Baseline
0.7478
0.7246
0.7154
0.7316
0.7127
0.7174
SFT Baseline
0.7651
0.7423
0.7270
0.7362
0.7486
0.7441
Offline RLHF
0.7735
0.7558
0.7449
0.7515
0.7589
0.7652
Engine-based RLVR
0.7430
0.7165
0.7137
0.7550
0.7904
0.7934
Full RLHEV
0.8002
0.7833
0.7821
0.7936
0.8106
0.8197
Appendix
Table 3: Unity assets generation quality for Figure 5 . Values are mean quality over eight seeds per method for each training budget.
Category
Transfer
Condition
Assert
Proxy
Engine acc.
Loss
Distribution
Unity → held-out Unity
zero-shot
0.25
0.451
0.007
11.685
Distribution
Unity → held-out Unity
target
0.75
0.449
0.008
11.590
Distribution
Unity → held-out Unity
scratch
0.25
0.421
0.005
11.909
Cross-engine
Unity → Unreal
target
0.35
0.474
0.008
11.596
Cross-engine
Unity → Unreal
scratch
0.25
0.437
0.009
11.825
Cross-engine
Unity → Godot
target
0.35
0.478
0.007
11.575
Appendix
Table 4: Generalization aggregate. Target means target-adapted transfer; scratch means target-only scratch training.
Transfer
Condition
MLLM judge
Proxy
Loss
Unity → Unreal
zero-shot
0.25
0.396
12.377
Unity → Unreal
Unity-source only
0.15
0.005
16.693
Unity → Unreal
target-adapted transfer
0.35
0.474
11.596
Unity → Unreal
scratch
0.25
0.437
11.825
Unity → Godot
zero-shot
0.25
0.396
12.375
Unity → Godot
Unity-source only
0.15
0.005
16.695
Appendix
Table 5: Cross-engine target-adapted transfer results. Target-adapted transfer gives measurable positive signal over scratch for both target engines, with higher proxy scores and lower losses.
Benchmark
Metric
Original
Naive aug.
AWoMo-aug.
Δ
Best
R2R
Success rate ↑
76.006±0.133
76.176±0.175
76.607±0.094
+0.79%
AWoMo
Gymnasium MuJoCo
Rollout return ↑
1568.44±1756.69
1648.03±1689.15
1724.73±1642.60
+9.96%
AWoMo
D4RL Gym-MuJoCo
Normalized score ↑
18.295±14.663
25.555±12.448
27.155±9.491
+48.43%
AWoMo
Appendix
Table 6: Final embodied generalization results. Values are mean ± std where available. Δ is the direction-normalized relative improvement of AWoMo-augmented training over the original baseline.
Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee. Empirically, WMRL accelerates training by 3-4x on various tasks at different agent scales, while exceeding the performance of standard RL baselines. Moreover, our post-trained 4B and 9B agents outperform much larger open-weight agents of 48B and 120B on held-out benchmarks. Beyond AutoResearch, WMRL also transfers to post-training embodied VLA policies, which demonstrates the generalizability of our method.
Recent advances in large language models have made automatic game generation increasingly feasible, yet reliably improving generated games beyond a playable version remains challenging. Naive iterative refinement can easily overfit a small set of test cases, producing fragile games with unresolved bugs, missing behaviors, and poor generalization to broader player interactions. We introduce RSIGame, an autonomous agentic game development framework with recursive self-improvement. RSIGame organizes development into complementary local and global loops. Concretely, a local explore-diagnose-improve loop broadly explores the executable game, diagnoses and prioritizes discovered issues, and performs evidence-grounded revision, where an evolving checklist continually accumulates new testing and improvement guidance. A global loop tracks overall quality, preserves the best checkpoint, and detects saturation or regression over long-horizon development. Beyond test-time improvement, RSIGame further internalizes successful development experience into the generator through training. Across 140 GameCraft-Bench tasks, two game engines, and five generators, RSIGame consistently improves game quality under matched development budgets. Notably, experience internalization enables Qwen3.8-27B to reach 61.38 on Godot and 58.53 on Phaser, exceeding GPT-5.5 one-shot scores while reducing Qwen's generation tokens by 11 times.
Wenyi Wu, Minghao Fu, Jieyu You +10
University of California San Diego · ByteDance Inc. · Carnegie Mellon University
We introduce the first multiplayer world model for highly dynamic environments governed by complex physical interactions. Whereas single-player world models treat the other agents as part of the environment, ours conditions on the action streams of multiple agents, learning to attribute changes in the scene to the correct player and to stay coherent under arbitrary combinations of their actions. We study this problem in the game of Rocket League, where players compete and cooperate under fast, tightly coupled dynamics. Trained on 10,000 hours of gameplay collected with publicly available bots, our 5-billion-parameter latent diffusion model generates four-player matches in real time, producing 20 frames per second on a single Nvidia B200 GPU. Although trained only on short clips, its rollouts stay stable far beyond the training horizon: distributional quality holds steady out to five minutes, the longest horizon we measure, and in practice we observe rollouts continuing for hours with no sign of collapse. We systematically investigate the central design choices: the video codec, the generative objective, and the multiplayer conditioning scheme. In addition, we characterize how behavior changes with model and data scale, including the capabilities that emerge and the failure modes that persist. We further develop targeted evaluations that probe the model's physical understanding rather than visual appearance alone. To support continued research on multiplayer world models, we release our dataset, our full training and inference codebase, and a live demo.
Anthony Hu, Václav Volhejn, Adrien Ramanana Rahary +24
1General Intuition · 2Kyutai · ‡École nationale des ponts et chaussées +1