GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games
Authors: Dhananjay Ashok, Adam Shen, Aslan Huo Feng, Chinmay Khanna, Jun Rui Huang, Raghav Sarmukaddam, Surendira Balaji Natarajan, Xiaotong Cui, Xincan Zhang, Thomson Yen, Hongseok Namkoong, Jonathan May, Jesse Thomason
Organizations: Information Sciences Institute, University of Southern California · Columbia University · University of Chicago · Purdue University · Georgia Institute of Technology
Powered by expert guidance, agents can operate in interactive environments; however, it is unclear whether they can learn autonomously from their own experience. To evaluate such self-improvement methods, we introduce GameBoyWorlds, a testbed for agentic self-improvement in video games. GameBoyWorlds-Execution evaluates task execution on a collection of 5 distinct game series. Agents are allowed access to dedicated training games but are provided no demonstrations, documentation, or rewards. Agents must ground themselves in the environment through self-directed exploration and by inferring actionable knowledge from their own experience. At test time, agents must complete short-horizon tasks that evaluate their ability to navigate, interact, and engage with game-specific mechanics in unseen games. Out-of-the-box frontier models complete fewer than 50% of the 500 tasks due to failures in multimodal grounding, establishing that self-improvement methods have room to push performance. We demonstrate that contemporary approaches to self-improvement are lacking, with world modelling and autonomous skill discovery failing, and a novel strategy that uses curiosity-based exploration to write guides achieving only partial success. GameBoyWorlds-Playthrough tests end-to-end game completion in two fan-made Pokémon games. We show that while frontier models have been pre-exposed to official releases such as Pokémon Red, they lack essential information on the games in our testbed. Instead of relying on their parametric knowledge to succeed, agents must learn from their own experience and autonomously improve over the course of the playthrough. We show that a sophisticated agentic pipeline with multimodal memory and hierarchical subgoals fails to reach even the first major milestone in both games, establishing GameBoyWorlds as an ambitious target for self-improving agents.
Figures & tables
Figure 1: GameBoyWorlds, a testbed for embodied self-improvement based on a collection of diverse games from the GameBoy universe (left). The execution leg (middle) allows agents to conduct self-driven exploration in game environments and tests their short-horizon task execution capabilities. The playthrough leg (right) tests deployment-time self-improvement in unseen games.
Figure 2: Model performance when deployed on GameBoyWorlds-Execution. Left: success rate (%) per game. Right: overall success rate. Models fail to exceed 50% average task completion.
Figure 3: Success rate by common task categories (left) and failure modes (right). While models are often more proficient in menu usage, they are often poor at basic navigation and interaction.
Figure 4: Insights drawn from ‘interesting’ trajectories surfaced during curiosity based exploration.
Figure 5: World modeling (WM) and autonomous skill acquisition (PAE) fail on GameboyWorlds-Execution. Our curiosity pipeline improves performance on average (+2%), but lacks robustness.
Figure 6: Contamination checks show that frontier models nearly perfectly recover the progression steps in the classic Pokémon games but recover less than 30% of the steps in the fan-made versions.
Figure 7: Substantial contamination in pretraining enables the agent to clear several milestones in Pokémon Red, while the same system remains stuck in the starter town in Pokémon Brown.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Capture of a screen region that can be used to compare against for the completion of the task ‘Read the trainers tips sign’.
Game
Wins
Ties
Losses
Deja Vu 2
6
2
1
Harvest Moon 3
6
0
3
Oracle of Seasons
7
0
2
Total
19
2
6
Appendix
Table 1: Paired comparison of visual history against no history with Gemma-4-31B. Each game contributes 9 comparisons, one for every combination of supervisor and action selection. A win indicates that visual history achieves a strictly higher task completion rate.
vs. No Supervisor
vs. Error Correction
Game
Wins
Ties
Losses
Wins
Ties
Losses
Deja Vu 2
2
2
5
1
6
2
Harvest Moon 3
8
1
0
4
2
3
Oracle of Seasons
9
0
0
5
0
4
Total
19
3
5
10
8
9
Appendix
Table 2: Paired comparison of the Subgoal Planning supervisor against each alternative supervisor with Gemma-4-31B. Each game contributes 9 comparisons, one for every combination of action selection and history. A win indicates that Subgoal Planning achieves a strictly higher task completion rate.
Figure 9: Example trajectory of agent successfully picking up a sickle in Harvest Moon 1. The episode terminates after the final frame. For brevity, the ‘Visual Difference’ step, where the VLM summarizes what changed between frames, has been omitted.
Figure 10: Random sample of high-reward trajectories from curiosity based exploration. Bomberman agents uncover bomb placement, Pokemon agents uncover specific attacks, Survival Kids agents reveal specific dialogue responses when attempting interaction, etc.
Setting
Value
Random Agent
Timesteps per training state
10,000
Curiosity Agent
Text weight α
0 (observation novelty only)
Rounds
2
Agents per round
1
Appendix
Figure 11: Configuration of experience collection for world model training.
Component
Specification
Architecture
Patch size
8×8 (360 patches per frame)
Patch normalisation
BatchNorm (non-affine)
Encoder
Conv 1×1 ( 1→2 ), ReLU
Conv 2×2 ( 2→4 ), ReLU
Conv 2×2 ( 4→4 ), ReLU
Appendix
Figure 12: Architecture and training hyperparameters of the observation embedder.
Component
Specification
Architecture
Input
Latents of 2 stacked frames ( 2×1440 ) and action index
Observation encoder
Linear ( 2880→512 )
Action encoder
Embedding ( ∣A∣→512 )
Predictor
Linear ( 1024→512 ), ReLU, LayerNorm
Linear ( 512→1440 )
Appendix
Figure 13: Architecture and training hyperparameters of the world model.
Game (State)
Proposed Tasks
Pokémon Red (Viridian City)
move right and enter the building using the a button. move down through the gap in the hedge to reach the lower area. move down and interact with the signpost using the a button. open the game menu using the start button.
Legend of Zelda: Link’s Awakening (Near Monster)
defeat the chepom enemy using the b button. move link to the right to reach the staircase. descend/ascend the staircase to transition to the next area. move link to the left wall to check for hidden secrets or boundaries.
Deja Vu 1 (In Cellar)
interact with the square button on the control panel. interact with the diamond button on the control panel. move toward the wooden door to attempt to leave the cellar. investigate the area directly beneath the hanging light bulb.
Sword of Hope 1 (Battle)
use the fight command to attack moth1. select the magic menu to view available spells. attempt to escape from the battle. defeat both moth1 and moth2 to end the encounter.
Bomberman Quest (Next to Guide)
move forward (up) to reach the entrance of the building. enter the building by pressing an action button (a or b) once positioned at the doorway. move to the left (left) to investigate the wooded area. navigate to the rightmost edge of the visible path.
Harvest Moon 1 (Chicken Coop)
move to and collect the egg lying on the floor to the right of the player. interact with the chicken standing to the left of the player. move to and examine the large barrel on the left side of the room. navigate to the upper right area of the room to exit the structure.
Appendix
Table 3: Example tasks proposed by Gemma-4-31B from a single training state of each training game (a subset of the full proposal list for that state).
Game
States
Proposed Tasks
Successful Tasks
Success Rate (%)
Pokémon Red
40
267
164
61.4
Legend of Zelda: Link’s Awakening
44
297
147
49.5
Deja Vu 1
42
335
233
69.6
Sword of Hope 1
18
127
102
80.3
Bomberman Quest
35
222
78
35.1
Harvest Moon 1
28
199
86
43.2
Appendix
Table 4: Task proposal and attempt statistics for the initial step of autonomous skill discovery with Gemma-4-31B. States is the number of training states from which tasks were proposed. Successful is the number of proposed tasks judged successful within 5 attempts.
Hyperparameter
Value
Base model
Gemma-4-31B
Adapters
One per training game
Objective
Supervised fine-tuning
LoRA rank
64
LoRA α
128
Learning rate
2×10−4
Appendix
Figure 14: Hyperparameters for VLA fine-tuning.
Task Section
Image Section
Game
Inferred Tasks
Pairs
Extracted Insights
Entries
Insights
Entries
Insights
Pokémon Red
445
288
565
66
247
72
249
Legend of Zelda: Link’s Awakening
275
100
200
52
151
57
157
Deja Vu 1
193
114
241
53
180
55
169
Sword of Hope 1
120
95
212
43
149
44
153
Bomberman Quest
228
167
322
66
208
66
206
Appendix
Table 5: Insight document statistics for each training game. Pairs is the number of (task, trajectory) pairs from which at least one insight was extracted.
Figure 15: Comparison of the region maps in Pokémon Red and Pokémon Brown.
Figure 16: A player-controlled Pokémon clearing a blocked passage in Pokémon Prism by eating the dirt obstruction.
Game
Question
Reference answer
Pokémon Red
Which Badge does Brock award after his defeat?
Boulder Badge
Which HM does the player receive from the captain of the S.S. Anne?
HM01, Cut
Which item allows the player to identify the ghosts in Pokémon Tower?
Silph Scope
Pokémon Crystal
Which Badge does Falkner award after his defeat?
Zephyr Badge
Which item is used to move the Sudowoodo blocking the path?
SquirtBottle
Which field move allows the player to cross bodies of water?
Surf
Appendix
Table 6: Representative knowledge-check questions. Each question requests a single verifiable fact concerning the corresponding game.
Model
Red
Crystal
Brown
Prism
GPT-5.6-sol
100%
100%
29%
29%
Claude Opus 5
97%
93%
8%
16%
Gemini 3.5 Flash-Lite
92%
94%
13%
15%
Gemma 4 31B
60%
61%
8%
16%
Appendix
Table 7: Strict human-annotated accuracy by game. Each cell contains 100 questions.
Model
Official
ROM hacks
Gap
Fisher p
GPT-5.6-sol
100.0%
29.0%
71.0 pp
6.0×10−61
Claude Opus 5
95.0%
12.0%
83.0 pp
7.5×10−72
Gemini 3.5 Flash-Lite
93.0%
14.0%
79.0 pp
7.5×10−64
Gemma 4 31B
60.5%
12.0%
48.5 pp
6.5×10−25
Appendix
Table 8: Strict human accuracy on the pooled official and ROM-hack subsets. Each subset contains 200 questions per model.
Model
LLM judge
Judge partial
BLEU-2
BERTScore-F1
GPT-5.6-sol
53.2%
57.5%
0.291
0.557
Claude Opus 5
49.2%
53.0%
0.157
0.330
Gemini 3.5 Flash-Lite
37.0%
42.9%
0.173
0.455
Gemma 4 31B
25.5%
30.0%
0.140
0.440
Appendix
Table 9: Automatic evaluation results over all 400 questions. The LLM-judge columns report classification-based accuracy, whereas BLEU-2 and BERTScore-F1 are mean similarity scores.
Model
Game
LLM judge
Judge partial
BLEU-2
BERTScore-F1
GPT-5.6-sol
Pokémon Red
91.0%
94.0%
0.458
0.699
Pokémon Crystal
90.0%
94.5%
0.401
0.732
Pokémon Brown
13.0%
19.0%
0.164
0.409
Pokémon Prism
19.0%
22.5%
0.140
0.387
Claude Opus 5
Pokémon Red
91.0%
93.5%
0.244
0.421
Pokémon Crystal
90.0%
91.5%
0.268
0.460
Appendix
Table 10: Automatic evaluation results by game and model. LLM-judge accuracy counts only answers classified as Right . Judge partial credit assigns 1 point to Right , 0.5 points to Partially Right , and 0 points to Wrong . BLEU-2 and BERTScore-F1 are mean similarity scores over the 100 questions for each game.
Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce S\textsuperscript{3}Gym, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. S3Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S3Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.
Self-evolving agents are expected to improve through interaction without external supervision, but this remains difficult in partially observable environments where agents must explore actively, learn from limited feedback, and decide when to trust prior experience. Existing LLM-agent methods often rely on memory or planning modules, yet they rarely close the loop between them to continually refine an internal understanding of environment dynamics. We introduce ProPlay, a procedural world model that supports procedure-level preplay, where agents can rehearse future procedural paths using the learned world knowledge. Rather than representing experience as isolated rules or low-level action constraints, ProPlay abstracts successful trajectories into procedures and organizes them in a procedure graph that captures causal transitions among task stages. Each transition is associated with a reliability record embedding to estimate its task-specific contribution from past outcomes. Before each episode, ProPlay simulates future procedural trajectories over known graph structures as structured soft guidance; after execution, it refines the graph using environment feedback. Experiments on public benchmarks show that ProPlay consistently improves environment understanding and self-evolution capability over strong baselines. Our code has been released in https://github.com/antman9914/proplay.
Yijun Ma, Zehong Wang, Yiyang Li +5
University of Notre Dame · University of Connecticut
Recent advances in large language models have made automatic game generation increasingly feasible, yet reliably improving generated games beyond a playable version remains challenging. Naive iterative refinement can easily overfit a small set of test cases, producing fragile games with unresolved bugs, missing behaviors, and poor generalization to broader player interactions. We introduce RSIGame, an autonomous agentic game development framework with recursive self-improvement. RSIGame organizes development into complementary local and global loops. Concretely, a local explore-diagnose-improve loop broadly explores the executable game, diagnoses and prioritizes discovered issues, and performs evidence-grounded revision, where an evolving checklist continually accumulates new testing and improvement guidance. A global loop tracks overall quality, preserves the best checkpoint, and detects saturation or regression over long-horizon development. Beyond test-time improvement, RSIGame further internalizes successful development experience into the generator through training. Across 140 GameCraft-Bench tasks, two game engines, and five generators, RSIGame consistently improves game quality under matched development budgets. Notably, experience internalization enables Qwen3.8-27B to reach 61.38 on Godot and 58.53 on Phaser, exceeding GPT-5.5 one-shot scores while reducing Qwen's generation tokens by 11 times.
Wenyi Wu, Minghao Fu, Jieyu You +10
University of California San Diego · ByteDance Inc. · Carnegie Mellon University