AI for Games in the Foundation Model Era
Organizations: National University of Singapore · Nanyang Technological University
Abstract
Foundation models, alongside advances in learned game-world models, are reshaping AI across the game lifecycle. Beyond playing games, recent systems model players and game dynamics, support design and development, adapt player-facing experiences at runtime, and evaluate resulting artifacts. Yet these directions have evolved largely separately, obscuring which capabilities transfer across settings and which remain tied to particular games, engines, interfaces, or player populations. We organize the literature into six roles according to the immediate use of AI output: playing and acting; modeling players and games; designing games; building and maintaining games; generating and adapting at runtime; and testing and evaluating games. For each role, we examine what structure is supplied by the game or workflow, what AI learns or produces, which capabilities and artifacts transfer across settings and roles, and what evidence supports the claims. We identify cross-role connections: trajectories train world models, learned environments provide experience for agents, design specifications drive executable implementations, and play or testing feedback guides revision. However, control schemes, rules, engine interfaces, state representations, and player contexts often remain setting-specific, so downstream claims require validation in the target setting. Evaluation is most standardized for bounded game playing and selected learned environments, while persistent state in learned worlds, repeated software revision, validated player modeling, sustained runtime adaptation, and representative automated testing remain less established. The central challenge is to reuse or transfer outputs and capabilities across roles while re-establishing evidence for effectiveness in the game-specific contexts where they are used.
Figures & tables
| Role | Output | Applications | Claim |
| Play and Act | actions; plans; messages | players; teammates; NPCs | action quality |
| Model Players and Games | states; transitions; player forecasts | planning; simulation; player models | predictive quality ; policy transfer |
| Design | levels; rules; story structures | PCG; automated design; co-creation | validity; control |
| Build and Maintain | code; scenes; project edits | engine agents; debugging; repair | working software |
| Generate and Adapt at Runtime | dialogue; quests; live content | characters; adaptive narrative | consistent content ; player response |
| Test and Evaluate | traces; verdicts; diagnoses | playtesting; verification; judging | state coverage ; verdict accuracy |
| Setting | Reused | Game-specific work | Examples |
| Per-game training | algorithm; settings | policy training; dynamics learning | DQN; AlphaZero; DreamerV3 [ 222 , 290 , 124 ] |
| Formal specification | solver; formal interface | rules; forward model | GGP; GVGAI [ 107 , 249 ] |
| Parameter sharing | policy weights | held-out testing; fine-tuning | Gato; Multi-Game DT; SIMA 2; NitroGen [ 263 , 171 , 292 , 206 ] |
| Test-time adaptation | base model; retrieval | demonstrations; rules; exploration | REGENT; Code World Models; Twin [ 300 , 172 , 295 ] |
| System | Generated object | Feedback / checks | Supplied structure |
| MarioGPT [ 302 ] | tile sequence | A*; prompt features | tiles; movement rules |
| DreamCraft [ 88 ] | block-aligned 3D scene | text alignment; adjacency losses | Minecraft block vocabulary |
| CubePart [ 416 ] | part-based 3D asset | part control; geometry | schemas; external behavior |
| GAVEL [ 326 ] | Ludii rule program | compile; simulated play; diversity search | game-description language |
| ScriptDoctor [ 89 ] | rules; tile levels | compiler; BFS; iterative repair | DSL; engine; solver budget |
| KNUDGE [ 352 ] | branching dialogue | lore and quest consistency | lore; quest goals |
| Role | Targets and example measures | Interpretation boundary |
| Play and Act | Completion; progress; action efficiency | Protocol-specific; transfer across rules, controls, or timing remains untested |
| Model Players and Games | World-state fidelity; reference-game policy return; player prediction or trait recovery | Short-horizon fidelity does not establish persistence; synthetic traits do not establish real-player validity |
| Design | Validity; diversity; constraint satisfaction; designer control | Does not establish authoring efficiency or intended player experience |
| Build and Maintain | Launchability; requirement satisfaction; repair; regression | Does not establish long-term maintenance or developer handoff |
| Generate and Adapt at Runtime | Latency; consistency; controllability; player outcomes | Single-session evidence; adaptation effects require suitable controls |
| Test and Evaluate | Coverage; validated defects; verdict accuracy; reproduction cost | Coverage is not correctness; sampled behavior is not population validity |
| Model / player | Progress (%) | Success (%) |
| Computer use: keyboard and mouse commands | ||
| Seed-1.8 | 39.8 | 20.0 |
| Claude Sonnet 4.6 | 38.3 | 19.4 |
| Gemini 2.5 Computer Use | 36.1 | 16.5 |
| OpenAI Computer Use | 35.8 | 16.5 |
| Qwen3-VL-Plus | 33.6 | 15.9 |
| (a) Standard harness | ||
| Model | Reasoning effort | Score (%) |
| GPT-6 Astra | Max | 62.71 |
| Claude Opus 5 | High | 30.16 |
| GPT-5.6 Sol | Max | 7.78 |
| Grok 4.6 | XHigh | 2.11 |
| GPT-5.6 Terra | Max | 0.80 |
| Target | What to check | Suitable evidence |
| Perceptual quality | Appearance; motion continuity; visual diversity | FVD; perceptual scores; human ratings |
| Mechanics correctness | Legal transitions; health/resource changes; contact and terminal events | Reference-state error; rule assertions; controlled replay |
| Persistent state | Identity and consequences after occlusion, revisit, or delayed events | Revisit/mutation probes; save–restore; synchronized views |
| Model | GC | IF | IE | OE | Overall | Validity (%) |
| Genie 3 | 2.74 | 2.40 | 1.51 | 1.81 | 2.12 | 87.1 |
| HappyOyster | 2.54 | 2.15 | 1.47 | 1.54 | 1.92 | 79.6 |
| LingBot-World2 | 2.04 | 2.13 | 1.95 | 1.16 | 1.82 | 78.8 |
| LingBot-World | 2.11 | 2.23 | 1.33 | 1.43 | 1.78 | 72.7 |
| HY-World2 | 2.14 | 2.06 | 1.13 | 1.09 | 1.61 | 51.7 |
| SANA-WM | 1.72 | 1.89 | 1.13 | 1.16 | 1.48 | 80.4 |
| Model (effort, where specified) | Harness | Pass@1 (%) |
| Claude Fable 5 (xhigh) | Claude Code | |
| GPT-5.6 Sol (xhigh) | Codex | |
| GPT-5.6 Sol (high) | Codex | |
| Muse Spark 1.2 (high) | Muse Code | |
| GPT-5.6 Sol (medium) | Codex | |
| Kimi K3 | Kimi Code |
| Model (effort) | Harness | Core mechanics | Content depth | Functional visuals | Art and presentation | Overall |
| Claude Opus 5 (xhigh) | Claude Code | 76.75 | 63.31 | 68.04 | 70.11 | 68.44 |
| Claude Fable 5 (high) | Claude Code | 76.52 | 58.61 | 66.93 | 67.68 | 65.72 |
| GPT-5.6 Sol (high) | Codex | 74.50 | 56.10 | 64.80 | 57.00 | 60.50 |
| Kimi K3 | Kimi Code | 73.03 | 53.38 | 56.90 | 53.59 | 56.96 |
| Seele02-pro | SeeleAgent | 68.42 | 48.76 | 52.74 | 44.17 | 50.70 |
| Claude Opus 4.7 (high) | Claude Code | 55.34 | 39.48 | 42.78 | 36.86 | 41.46 |
| Held out | What changes | Claim supported by testing | Example / test |
| Map / level | Layout, placement, visual setting | Robustness within specified rules | Procgen: unseen levels [ 66 ] |
| Mode | Objectives, opponents, difficulty, or rules | Adaptation to named rule/task variants | Atari mode transfer [ 97 ] |
| Game | Title; potentially rules, assets, controls | Transfer of the evaluated behavior or model function | SIMA: task success; SCOPE: action response [ 291 , 328 ] |
| Benchmark / setting | Task and measure | Evidence | Comparison limits |
|---|---|---|---|
| Play and Act | |||
| TextWorld / Jericho / TextArena [ 74 , 126 , 115 ] | Generated quests, authored adventures, and single-/multi-player text games; task-specific outcomes | Play; executable checks | Command vocabulary, rule access, partners, and scoring conventions differ |
| NLE / Crafter [ 164 , 121 ] | Procedural exploration and survival; game scores and achievement coverage | Play; executable checks | Progress within a game, not arbitrary-rule transfer |
| BALROG [ 239 ] | Six game environments; text/visual inputs; normalized progress | Play; executable checks | Abstracted actions; native motor timing excluded |
| VideoGameBench [ 394 ] | Full: 10 real-time test games; Lite: three development and three test games, with the game paused during model inference; checkpoint progress | Gameplay traces | Different game sets; progress is not completion |
| GameWorld [ 238 ] | 170 tasks / 34 browser games; state-verifiable success and partial progress | Interaction; state checks | Native vs. semantic actions; game paused during inference |
| Flow | Artifact | Example system | Outcome to validate |
| Play Model | Action-linked traces | GameNGen [ 333 ] | Prediction fidelity; interactive consistency |
| Model Play | Imagined trajectories | Dreamer 4 [ 125 ] | Reference-game policy performance |
| Design Build | Editable plan | DreamGarden [ 90 ] | Executable output satisfying requirements |
| Build Model | Executable state; rendering controls | Programmable World Model [ 143 ] | Generated views agreeing with engine events |
| Test Design | Compiler/search feedback | ScriptDoctor [ 89 ] | Compilation; solvability within search budget |
| Design Runtime | Narrative structure | NarrativeGenie [ 163 ] | Narrative continuity; intended player experience |
| Role | Reported evidence | Evaluation conditions | Research gap |
| Play | Cross-game skills [ 292 , 206 ] ; rule inference [ 295 ] | Selected games; specified adaptation budgets | Joint transfer across rules, controls, and timing |
| Model | State prediction [ 193 ] ; NPC behavior [ 79 ] ; player prediction [ 319 ] | Domain-specific state and behavior data | Persistent world state; reliable player-model updates and transfer |
| Design | Solvable levels [ 302 ] ; rule search [ 326 ] ; designer steering [ 90 ] | Fixed representations; solver/creator feedback | Intended player experience; sustained creative control |
| Build | Complete projects [ 202 ] ; scoped edits [ 165 ] ; short revision chains [ 56 ] | Specified engines, requests, tests | Evolving requirements; developer handoff |
| Runtime | Constrained rule execution [ 134 ] ; player-outcome studies [ 133 , 120 ] | Bounded encounters or field deployments | Cross-session consistency, experience, and shared-state fairness |
| Test | Mechanic verification [ 149 ] ; human-reference defect testing [ 19 ] ; fairness diagnostics [ 285 ] | Specified access, oracles, sampled behaviors | Independent verification; broader behavioral coverage |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| System / source | Role | Environment and mechanism | Data and evaluation |
|---|---|---|---|
| Shared policies and learned controllers | |||
| VPT [ 24 ] | P | Env: Minecraft; pixels; keyboard/mouse Mech: Inverse-dynamics labels; BC, then RL | Data: About 70K h of video; smaller labeled set Eval: Native control; Minecraft-specific pretraining |
| Gato [ 263 ] | P | Env: Atari plus language/robotics tasks Mech: One autoregressive, multimodal policy | Data: Mixed offline demonstrations Eval: Shared weights across tasks; adaptation evaluated separately |
| Multi-Game DT [ 171 ] | P | Env: Atari; pixels and discrete actions Mech: Offline return-conditioned Transformer | Data: 41 training / 5 held-out games Eval: Shared training-game policy; new titles require fine-tuning |
| DreamerV3 [ 124 ] | P ( M ) | Env: Visual or vector observations Mech: Latent world model; imagination RL | Data: 150+ tasks; one configuration Eval: Algorithm reuse with per-task learning |
| STEVE-1 [ 189 ] | P | Env: Minecraft; language goals; native actions Mech: VPT policy; latent goals; hindsight | Data: Relabeled Minecraft behavior Eval: Text-to-behavior transfer within one game |
| System / source | Role | Representation and control | Data and evidence |
|---|---|---|---|
| Planning and policy-learning models | |||
| World Models [ 119 ] | M ( P ) | Repr: VAE + recurrent latent dynamics Control: Continuous control; visual observations | Data: CarRacing and VizDoom Evidence: Controller return; transfer from imagined to real rollouts |
| MuZero [ 275 ] | P ( M ) | Repr: Value-, reward-, and policy-trained latent state Control: Discrete actions; tree search | Data: Atari, Go, chess, shogi Evidence: Decision quality; no image reconstruction requirement |
| Dreamer / DreamerV3 [ 123 , 124 ] | P ( M ) | Repr: Recurrent latent dynamics; imagined trajectories Control: Actions and rewards | Data: Visual and vector control tasks Evidence: Policy return in reference environments |
| I2A [ 260 ] | P ( M ) | Repr: Encoded rollouts from an imperfect model Control: Discrete actions | Data: Sokoban; MiniPacman Evidence: Policy use of prediction errors |
| Action-conditional prediction [ 235 ] | M | Repr: Feedforward / recurrent image prediction Control: Recorded Atari actions | Data: Five Atari games Evidence: Prediction error; selected control tests |
| System / source | Role | Input and construct | Validation and boundary |
|---|---|---|---|
| Behavior, skill, and individual action prediction | |||
| player2vec [ 343 ] | M | Input: Mobile-game event sequences Model: Masked-token Longformer | Data: Masked-event recovery; embedding structure Boundary: No independent preference labels |
| Behavior Structformer [ 296 ] | M | Input: Structured tracking events Model: Transformer over typed event fields | Data: Constructed session-count targets Boundary: Activity proxy rather than experience |
| TrueSkill [ 129 ] | M | Input: Match outcomes Model: Bayesian skill and uncertainty | Data: Predictive performance of ratings Boundary: Historical skill model; not preference inference |
| Maia / individual Maia [ 214 , 215 ] | M | Input: Chess positions and player histories Model: Skill-group / individual behavior models | Data: Held-out human moves Boundary: Behavior imitation, not strongest play |
| Maia-2 [ 318 ] | M | Input: Position and skill level Model: Shared model; skill-aware attention | Data: Human moves across rating groups Boundary: Population conditioning |
| System / source | Role | Artifact and mechanism | Feedback and evaluation |
|---|---|---|---|
| Content, rules, and mixed-initiative design | |||
| Tanagra / Sentient Sketchbook [ 299 , 187 ] | D | Artifact: Platformer levels / map sketches Mech: Constraint solving / novelty search | Feedback: Playability checks; designer edits Eval: Historical mixed-initiative authoring |
| ANGELINA / Mechanic Miner [ 73 , 72 ] | D | Artifact: Games and mechanics Mech: Automated design; reflective mechanism search | Feedback: Simulated play and design criteria Eval: Historical automated-game-design systems |
| CubePart [ 416 ] | D | Artifact: Part-controllable 3D assets Mech: Semantic schemas; separate meshes | Feedback: Part identity and geometry Eval: Part control; external behavior scripts |
| WorldSculpt [ 231 ] | D | Artifact: Compositional mesh scenes Mech: Posed views; object mesh prior | Feedback: Mesh/scene fidelity Eval: Unreal reconstruction tests |
| AutoBG [ 186 ] | D ( T ) | Artifact: Board-game ideas and rulebooks Mech: Rule iteration; persona feedback | Feedback: Designer revision; model critiques Eval: 207 held-out games; 30-creator study |
| System / source | Role | Generated element and execution | Human evidence and finding |
|---|---|---|---|
| Narrative and content systems | |||
| CALYPSO [ 411 ] | R ( D ) | Element: Encounter prose; summaries; ideas Execution: Human-mediated D&D; current scenario | Exposure: 71 players and DMs; field deployment Finding: 51 chat threads; direct use plus DM revision |
| Concordia [ 335 ] | R ( P ) | Element: Character actions and narrated consequences Execution: Actor / game-master components; memory | Exposure: Framework demonstrations Finding: Configurable social simulation; no common player-outcome trial |
| KG-conditioned quests [ 21 ] | R ( D ) | Element: Personalized quests and dialogue Execution: Player input; graph paths; grammar; GPT-2 | Exposure: Text-rating study; generated-quest analysis Finding: Intent grounding; unsupported entities remain possible |
| NarrativeGenie [ 163 ] | D ( R ) | Element: Story graphs; hints; summaries Execution: Graph/state dialogue manager | Exposure: 19 players; one mystery episode Finding: Ratings plus 30 simulated-history checks |
| Drama Llama [ 307 ] | R ( D ) | Element: Responsive text narrative Execution: Storylet triggers checked by an LLM | Exposure: Six authors; authoring study Finding: Authorable responses; not a player-outcome trial |
| System / source | Role | Target and tester | Oracle and evidence |
|---|---|---|---|
| Exploration, coverage, and regression | |||
| Procedural Personas [ 130 ] | T | Target: Executable game states Tester: MCTS; evolved goal heuristics | Oracle: Game-specific outcomes Evidence: Deliberate style diversity; not population sampling |
| Wuji [ 407 ] | T | Target: Instrumented combat games Tester: Multiobjective evolutionary RL | Oracle: Bug predicates; developer confirmation Evidence: Two commercial games; three newly confirmed bugs |
| Synthetic / humanlike agents [ 19 ] | T | Target: Three GVG-AI games Tester: Agents with different behavior objectives | Oracle: Separate execution oracle; human trajectories Evidence: 45 seeded bugs; 427 human traces |
| EA navigation AutoPlayers [ 108 ] | T | Target: Battlefield 2042 / Dead Space; structured state Tester: RL navigation within scripted test bots | Oracle: Existing production QA infrastructure Evidence: Navigation deployment, not end-to-end learned QA |
| CA2 [ 3 ] | T | Target: State/images and call stacks Tester: RL guided toward code targets | Oracle: Function-entry instrumentation Evidence: Improved target reachability; diagnosis is separate |
| Game / system | Family | Role | AI contribution | Resource / status |
|---|---|---|---|---|
| Players, opponents, and teammates | ||||
| Minecraft / Voyager [ 341 ] | Sandbox | P | Code-skill acquisition | Research: code and videos |
| GT7 / Sophy † [ 356 , 257 ] | Racing | P | Learned rival driving | PS5 custom-race feature |
| PUBG / Ally [ 161 , 294 ] | Battle royale | P , R | Voice-guided cooperation | Limited beta: June 2026 |
| StarCraft II / AlphaStar † [ 337 ] | RT strategy | P | Strategy and unit control | Research match recordings |
| Dota 2 / OpenAI Five † [ 33 ] | Team MOBA | P | Team coordination | Historical match videos |