From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
Organizations: Roblox · The Pennsylvania State University
Abstract
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
Figures & tables
| Method | Training data | Total data | MC data | Embodied | GUI | Combat | Avg |
| Commercial VLMs (low reasoning effort) | |||||||
| Claude Opus 5.5 | – | – | – | 72.9 | 80.6 | 53.1 | 68.9 |
| GPT-6.1-Sol | – | – | – | 68.7 | 84.9 | 69.4 | 74.3 |
| Gemini 4 Argon | – | – | – | 80.7 | 96.0 | 71.4 | 82.7 |
| Trained Game Policies | |||||||
| VPT [ 3 ] | Human play | 70,000 h | 70,000 h | 6.0 | 0.8 | 3.6 | 3.5 |
| Agent | Training data | Data scale | Obstacle Run3D | Last Stand | Monster Shoot | Scene Escape | Cue Chase | Mean |
| Commercial VLMs | ||||||||
| Claude Opus 4.7 [ 18 ] | – | – | 0.094 | 0.308 | 0.400 | 0.460 | 0.380 | 0.328 |
| Claude Opus 4.6 [ 18 ] | – | – | 0.172 | 0.147 | 0.362 | 0.540 | 0.840 | 0.412 |
| Claude Sonnet 4.6 [ 18 ] | – | – | 0.200 | 0.144 | 0.166 | 0.440 | 0.440 | 0.278 |
| GPT-5.5 [ 18 ] | – | – | 0.133 | 0.416 | 0.464 | 0.720 | 0.580 | 0.463 |
| GPT-5.4 [ 18 ] | – | – | 0.089 | 0.148 | 0.326 | 0.680 | 0.300 | 0.309 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Field | Examples |
| Skill | a gameplay category such as those of Fig. 3 , e.g . traversal and parkour |
| Camera | first person, third-person follow, top-down, side view |
| Dynamics | ranges for speeds, tolerances, hold durations, gravity, gap widths |
| Controls | WASD or arrows, Space, E/F, left button, mouse aim, with a plain-language description |
| Contract | instructed variable (which target, pad, or gate); violations ( wrong_target , fell , over_budget ) |
| Check | Purpose |
| Static validity | Check that the code compiles and that scene, task, and wording use separate random streams. Counterfactual arms must share the scene while requesting different targets. |
| Solvability | Run the teacher in the offline mirror under nominal timing and added latency to check that the task can be completed reliably. |
| Non-triviality | Test idle, hold-forward, and random-key policies, and a teacher pursuing the wrong target, to detect shortcuts and instruction-agnostic success. |
| Engine parity | Compare teacher behavior and success in the engine with the mirror, checking whether input timing or engine execution changes the outcome. |
| Visual integrity | Inspect decoded recordings for blank frames, missing scene changes, and failures of appearance variation. |
| Layer | Calls |
| Episode | params samples template parameters; choose selects the arm’s target; say generates the instruction and paraphrases; contract specifies the target, effect, and constraints; part places geometry. Scene, appearance, task, and text use separate random streams: paired arms share a scene, and appearance changes preserve its layout. |
| World | world surrounds the arena with one of seven themes (terrain, Toolbox props, sky, lighting), assigns Roblox avatars to the player and NPCs, and samples camera settings |
| Game | fixedStep , pressed , key , mouseLook (input); part , model , figure , chaseCamera (scene); publish (the state the teacher reads); complete , violate (the verifier) |
| Checks | Luau analyzer (G1); mirror tests (G2–G3); teacher rollouts in the headless engine, recorded at 20 fps (G4); export to 200 ms windows; recorded-frame checks (G5) |
| Field | Laser Maintenance Bridge |
| Skill | Traversal and parkour: timed jumps and waiting out moving hazards; local movement secondary |
| Camera | First person on a 3-stud-wide bridge high over a drop, in a theme world; the view starts within of the bridge |
| Dynamics | Gaps 2.4–3.0 studs (a running jump covers 5); sweeping lasers move to and fro across the bridge at 5–8 studs/s over a half-width of 5–7; a low laser at 2.9–3.4 studs (4.8 standing, 2.4 crouched); hazards 3–5 studs apart; walking 7 studs/s, crouching 3.5 |
| Controls | “The mouse turns your view and W, A, S and D walk. Space jumps. Hold C to crouch (and walk slower).” |
| Contract | Instructed variable: the first or the second marker (red and blue, colors shuffled); effect: reach it and stand still (0.3 s), then 0.8 s without a violation; violations: fell , laser , too_far ; 14 s budget |
| Templates | gap_jump : cross one gap to the first marker or two to the second; laser_wait : pass one sweeping laser or two; route : pass a sweeping laser and a gap to the first marker, then crouch under a low laser to reach the second. Each instruction has two paraphrases. |