Games are a rich testbed for visual agents, demanding perception, long-horizon planning, and precise control. However, collecting human gameplay is costly, and learning from gameplay videos requires inferring missing actions. Existing datasets also cover a limited range of games, often without explicit tasks or verifiable outcomes. To address these gaps, we introduce Game2Player, a framework that co-develops game environments and players to collect diverse and verifiable gameplay experience at scale. In the Game2Player loop, a Builder agent adapts existing web games, exposes their internal state, and defines tasks with state-based progress measures and success checks. A Coding Player and a VLM Player then attempt the tasks and report problems to the Builder for repair, so that play serves as both environment validation and training data collection. With Game2Player, we build 568 verified web games with 3,436 tasks, collect 15.7K trajectories (307 hours of gameplay), and hold out 50 games as the benchmark PlayerBench. To evaluate the collected experience as training data, we train VLM agents with progress-weighted SFT, assigning greater weight to steps that advance task progress. Game2Player-9B improves progress from 14.3% to 31.2% on PlayerBench and from 30.0% to 53.1% on the external GameWorld benchmark. More broadly, our work suggests a path for visual agents to scale their gameplay experience by building, verifying, and learning from existing game worlds.
Figures & tables
Resource
# Games
Actions
Goal
State
Prog.
Autom.
VideoGameBench [ 77 ]
10
–
–
✗
✓
✗
LMGame-Bench [ 20 ]
6
–
–
✓
∘
✗
GameWorld [ 30 ]
34
–
–
✓
✓
✗
PlaySuite [ 50 ]
5,734
–
–
✗
✓
∘
VPT [ 2 ]
1
Human
✗
✗
✗
–
NitroGen [ 28 ]
> 1K
Human
✗
✗
✗
–
Table 1: Comparison with game resources. Goal: demonstrations carry task instructions. State: scoring reads internal game state. Prog.: intermediate progress is scored. Autom.: environments or evaluators are built automatically. ✓ yes, ∘ partial, ✗ no, – not applicable.
Figure 2: Overview of Game 2 Player . The Scout selects games from web portals ( top ). In the co-development loop ( middle ), the Builder turns each game into an environment with state-verified tasks, the VLM Player and the Coding Player play every task and report faults of the game, and the Builder checks each report against the recording before repairing it. Both players then record gameplay in the accepted environments to train Game 2 Player with progress-weighted SFT ( bottom ).
Figure 3: Two players on the same task and one training format. The code policy reads the player observation ψ and sends keys at every frame without calling a model ( top ). The VLM Player sees only the screenshot and pauses the game to choose each action ( bottom ). The code policy’s keys are cut into steps in the VLM Player’s format, so both players yield the same kind of training example ( middle ).
Figure 4: The Game 2 Player collection and its training data. The 568 games cover diverse genres, of which the ten most common are shown ( left ). Share of each player in the training data, and the best progress of its training trajectories ( right ). Figure 19 compares all genres and engines with Player Bench .
Figure 5: Progress-weighted sampling. Actions receive weights from progress and repetition during play (1), followed by terminal outcomes (2). Paired bars compare the action budgets before and after game balancing (3), in units of the median game size M . Stochastic rounding converts the weights into integer numbers of context–target pairs (x,y) (4). Solid cards are guaranteed copies, and dashed cards are drawn with the displayed probabilities. The examples are schematic and use the factors in Tab. 11 .
Figure 6
Game
Report
Change in the full loop
Correct
Solved
There Is No Game
Coding Player: progress stays at 0.9 after the letter O falls
Hooks each game variable when it appears, so later story events are recorded
0/9
0/9
Dungeons & Degenerate Gamblers
VLM Player: the table stays empty after the Gambler is chosen; Coding Player: cards and buttons cannot be read
Cancels pending room animations before freeing the cards; adds cards, buttons and battle state
6/7
0/0
Dome Romantik
VLM Player: the upgrade list runs below the screen
Keeps five upgrades on screen and scrolls the list with the selection
5/6
4/0
Where the Water Flows
Coding Player: terrain and objects cannot be read
Adds terrain, platforms, mechanisms and movable objects
7/7
0/4
Strike Force Heroes
Coding Player: the player, soldiers and flags cannot be read
Adds the player, visible units, flags and pickups
5/5
0/4
Table 2: Selected reports and repairs with the GPT-6.1 Terra Builder. Correct and Solved give the number of tasks without reports / with the full loop. Table 16 lists all reports.
Player Bench
GameWorld
Model
Progress
Success
Progress
Success
Human players
69.4
51.8
82.6 †
77.1 †
Closed-source models
GPT-6.1 Sol
62.7
39.0
64.3
44.1
medium effort
62.1
40.1
–
–
GPT-6.1 Terra
60.4
36.2
69.2
49.4
Table 3: Benchmark results on Player Bench and GameWorld. Progress and success in %. Bold marks the best of each group, and our models are shaded. All models play with thinking on, and API models at low reasoning effort unless marked as medium; Fig. 8 compares thinking on and off. Averages in the text use the low-effort rows of the models evaluated on both benchmarks. Unless marked, entries use our harness with 100 steps and one run per task, and game-specific models their own harnesses adapted to our games. † Expert player reported by GameWorld [ 30 ] . ‡ Published results with the GameWorld harness [ 30 , 17 ] ; CU denotes its computer-use mode, the others its generalist mode. § Published results under the PLAYER-1 protocol with 400 steps and three trials per task [ 31 ] .
Figure 8: Base and Game 2 Player models with thinking on and off. Progress ( top ) and success ( bottom ) in % of each Qwen3.5 base model (light) and its Game 2 Player counterpart (dark) on both benchmarks.
Figure 9: Human play by difficulty tier on Player Bench . Left: progress and success in %, averaged over the tasks of each tier, with the number of tasks under each tier. Right: median length of all attempts in seconds.
Figure 10: Training data and training steps. Progress and success on Player Bench and GameWorld with 0, 100K, 400K, 800K and all 1.56M training examples ( left ) and after 0, 3K, 6K and all 9.2K training steps ( right ). Hollow points at zero denote the untrained Qwen3.5 base models.
Player Bench
GameWorld
Setting
Progress
Success
Progress
Success
Data source
4B
Base
12.7
4.9
27.3
9.4
Coding Player only
17.3
6.3
36.8
14.7
VLM Player only
25.1
11.5
35.5
13.5
Both
23.4
10.5
34.9
13.5
Table 4: Data-source and sampling ablations. Progress and success in %. Uniform sampling draws as many examples without weighting or balancing.
Figure 11: Qwen3.5-4B and Game2Player-4B on four held-out Player Bench tasks , with thinking on and at most 100 steps. Each panel shows the final frame of each model with its final progress, and the best progress reached after each step.
Figure 12: An engine skill. The Builder receives this skill once it has identified a Godot game. The linked workflow and tools are part of the skill.
Figure 13: Rules quoted from the agent prompts. The Scout selects games, the Builder freezes the tasks, measures progress toward the objective, plays from the player’s view and checks player reports against recordings, and the Coding Player reads only its probe and asks only for observable state. Grey brackets mark omitted text.
Source
Allowed content
VLM Player, defect
End of an episode with a report, based on the screenshots, that the game is broken, merged across episodes of a game.
Coding Player, defect
Crash, start routine that misses its situation, key with no effect, progress that never moves, success that fires too early.
Coding Player, observation
A field the code policy needs, its meaning and where it is visible on screen.
Not allowed
Hidden information, shortcuts to success, easier tasks.
Table 5: Reports that return to the Builder. A defect report describes something that does not work, and an observation request asks for a field of the player observation.
Coding Player
Games
GPT-6.1 Sol
21
Claude Opus 5.5
11
Claude Sonnet 5.5
5
GPT-6 Luna
3
DeepSeek-V4.1-Flash
3
No accepted code policy
7
Table 6: Coding agents that wrote the code policies of the 50 Player Bench games.
Figure 14: Code policies against VLM Players, game by game. Each dot is one of the 50 Player Bench games. Dots above the diagonal are games on which the Coding Player reaches higher progress than all four compared VLM Players, Qwen3.8-Flash, DeepSeek-V4.1-Flash, GLM-5.3-Flash and Gemini-3.8-Flash.
Coding Player
Best VLM Player
Game
Prog.
Succ.
Prog.
Succ.
MOBS, INC
100.0
100.0
90.5
80.0
Gun Knight
62.7
40.0
17.3
0.0
Kingdom Rush
98.3
83.3
92.8
66.7
Zombotron Re-Boot
0.0
0.0
12.8
11.1
Tiny Kingdom
76.5
50.0
22.3
0.0
Table 7: Per-game Player Bench results of the Coding Player and the best of four VLM Players. Progress and success in %. For each game, the VLM column uses the model with the highest mean progress among Qwen3.8-Flash, DeepSeek-V4.1-Flash, GLM-5.3-Flash and Gemini-3.8-Flash.
Figure 15: Progress of repeated code policy runs. Top, progress of the ten runs of the same code policy on each of twelve tasks drawn at random from the tasks with ten runs, with the progress distance Dprog of each task. Bottom, the density of Dprog over tasks for the code policies and for repeated runs of MiniMax-M3.1-Flash.
Figure 16: Key presses of repeated code policy runs. Top, the key presses in the first ten seconds of the ten runs of the same code policy on each of six tasks drawn at random from the tasks whose runs press different keys, one lane per run and one colour per key, with the key distance Dkey of each task. Bottom, the density of Dkey over tasks for the code policies and for repeated runs of MiniMax-M3.1-Flash.
Source
Trajectories
Hours
Eligible steps
Training examples
Note only
With thinking
Neither
Count
%
Count
%
Coding Player
12.9K
81.8
217.69
1,835.0K
1,372.1K
87.7
0
0
1,372.1K
VLM Player
2.9K
18.2
88.89
219.1K
191.5K
12.3
169.1K
22.5K
0
Claude Haiku 5.5
2.2K
14.3
72.53
180.0K
157.1K
10.0
157.1K
0
0
DeepSeek-V4.1-Flash
0.3K
1.8
5.09
15.5K
14.4K
0.9
< 0.1K
14.3K
0
GPT-6 Luna
0.2K
1.3
7.00
14.3K
11.8K
0.8
11.8K
0
0
Table 8: Recorded gameplay and training examples by source. Percentages are shares of the full corpus. The four VLM sources appear below their subtotal. Gameplay hours exclude inference pauses, and training examples include resampled copies. The last three columns partition the training examples by the note and thinking at the target step.
Figure 17: Characteristics of the training data. (a) Cumulative fraction of trajectories by the number of eligible steps after segmentation and filtering, before resampling. The VLM Player has a 100-step budget. (b) Action composition of the final training examples. Individual keyboard actions, key sequences, mouse actions and waits are counted separately. (c) Distribution of samples per game before and after resampling. Each point counts the games in a sample-count interval, with neighboring interval counts connected by lines. Intervals use a logarithmic scale.
Source
Progress
Success
Coding Player
97.9
96.5
VLM Player
39.0
21.8
Table 9: Outcomes of retained training trajectories. Mean final progress and task success in %, averaged over recordings. Code policy recordings come from tasks completed by a policy during collection.
Stage
Actions
Waits
Total
Segmented candidates
1,426.6K
661.8K
2,088.5K
Invalid actions
− 13.6K
0
− 13.6K
Zero-weight steps
− 1.2K
− 19.6K
− 20.8K
Not sampled
− 379.1K
− 243.5K
− 622.6K
Repeated uses
+ 132.2K
0
+ 132.2K
Training examples
1,164.9K
398.8K
1,563.6K
Table 10: How preprocessing changes the sample count. Actions and waits are counted separately. Negative rows subtract steps, and the positive row adds repeated uses. Waits with positive weights below one may be omitted during sampling, in addition to waits assigned zero weight.
Stage
Weight
Application
Progress credit
≥3,2,1.5
Latest three actions ending within [e−5,e+0.15] s of a progress event at e .
Success credit
≥3,2,1.5
Final three eligible actions of a successful episode.
Failure suffix
×0
Up to three actions in the final 5 s without later progress.
Timeout suffix
×0.25
Up to five actions in the final 10 s without later progress.
Flat repetition
×0.35
From the third repeated action, if progress does not improve within 5 s.
Game balancing
Bj
Allocate by action type, then in proportion to weight, at most 3 uses per row.
Table 11: Sampling weights. Weights become resampling multiplicities through Eq. 7 .
Figure 18: Prompt of a VLM Player at one step, with thinking on. Grey text in angle brackets stands for the game rules, controls and task instruction written by the Builder and for the content of earlier steps. The user message holds up to 50 previous steps for open-weight models and all previous steps for API models, the last six with their screenshots, and the current screenshot.
Figure 19: The collection and the held-out split in full. Every genre and engine of the 568 games and the difficulty tiers of their 3,436 tasks, as shares of the whole collection (light) and of Player Bench (dark). The single-game genres are dress-up, interactive fiction, narrative, party, puzzle horror, runner, tower defense, typing and visual novel, and unclassified games fit none of the listed genres.
Table 29Table 30Table 31
Figure 21: Task outcomes of all three settings in both experiments. A task is solved when its environment is correct and the final code policy completes it.
Figure 22: A VLM Player and a code policy on the same task. In each game, the upper row shows four steps of an episode of Claude Haiku 5.5 with the screenshot it saw, its action and its note, and the lower row four moments of a code policy episode that completes the task, each at a key press or click, with the frame, the keys held and fields of the policy’s observation. Pointers mark clicks. The bar and p give the progress the VLM Player reaches.
Figure 23: Base and trained models on held-out Player Bench tasks , with thinking on and at most 100 steps. Each row shows the task, the initial frame and the final frames of Qwen3.5-4B and Game2Player-4B. The curves show the best progress reached after each step.
Figure 24: Participants of the human study. (a) Age group and gender. (b) Attempts of each participant, split into successful and unsuccessful ones.
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
Pre-trained models can read screens and reason about games, yet on an unfamiliar game, an agent still has to learn to play. We ask how an agent can self-improve on such a game from a few of its own attempts, in real-time visual games where the agent sees only the screen and the game keeps running while the agent thinks. We study two kinds of policy that can carry this improvement. In model as policy, a model selects actions during play; we adapt real-time chunking so that committed actions keep executing while the model reasons, and train a 2B controller for this interface. In code as policy, a model writes, tests, and revises a program between attempts, and the program acts during play. Across 170 GameWorld tasks, with at most eight one-minute executions per run, code as policy reaches 47.80% mean progress, against 30.90% for our trained controller, and leads in all five game categories. Improving the controller's weights on a single game, by contrast, yields only modest gains. These results favor letting a model think offline and storing what it learns in a program that acts in real time. Project repository: https://github.com/TabibitoQZP/SIGame
Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.
Mingyu Ouyang, Yiran Wang, Siyuan Hu +9
National University of Singapore · ARC Lab, Tencent IEG · Nanyang Technological University