cs.CVNov 8, 2026

From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models

Authors: Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang

Organizations: Roblox · The Pennsylvania State University

Abstract

Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games

    Sep 8, 2026Ryan Truong, Lance Ying, Samuel J. Gershman +1Reinforcement LearningGame-Playing Agents

  2. GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

    Oct 2, 2026Haoyue Yang, Jingyao Li, Zhengfan Wu +3Agentic Code GenerationSynthetic Data Generation for Language Models

  3. Distilling Game Code World Model Generation into Lightweight Large Language Models

    May 23, 2026Tyrone Serapio, Arjun Prakash, Haoyang Xu +2LLM-Based Program Synthesis