Player-1: Learning to Play from Synthetic Experience

Mingyu Ouyang; Yiran Wang; Siyuan Hu; Xingyilang Yin; Huiqiang Sun; Guangzhi Wang; Junfu Pu; Cheng Cheng

cs.CVOct 14, 2026

Player-1: Learning to Play from Synthetic Experience

Authors: Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng,

Organizations: National University of Singapore · ARC Lab, Tencent IEG · Nanyang Technological University

Abstract

Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.

Figures & tables

Figure 1: Overview of Player-1 : learning game control beyond demonstrations. A shared game-control action space connects supervised initialization from gameplay demonstrations with multi-environment RL in Player-1-Gym . Learning from synthetic experience with verifiable rewards advances control capabilities beyond SFT, with gains that generalize beyond the RL training suite.

Method	Learning focus	Supervision	RL environments	Evaluation	Availability
Game-TARS	Large-scale pretraining	Reasoning + action	—	Held-out games	✗
SIMA 2	Embodied assistance	Reasoning + action	Games + world model	Held-out games	✗
NitroGen	System-1 generalist	Action	—	Games + adaptation	✓
Odysseus	Long-horizon RL	Reasoning + action	5 Mario levels	Mario games	✓
Player-1	Beyond demonstrations	Reasoning + action	16 synthetic envs	RL held-out games	✓

Table 1: Comparison of representative game-agent learning pipelines, including supervision, RL environments, evaluation scope, and pipeline and model availability.

Figure 2: Curated gameplay demonstrations for supervised initialization. (a) Recorded controls summarized by control type, number of inputs per control step, and pointer-motion mode in our demonstration trajectories. Two or more concurrent inputs form a composite action. (b) A demonstration sample pairing recorded actions with an annotated task instruction and visual reasoning.

Figure 3: Player-1-Gym : synthetic experience with verifiable rewards. (a) Shared control demands across five task families. (b) Visualization of sixteen 2D and 3D environments. (c) Examples of appearance, camera, and scene randomization applied to the same environment.

	Success rate (%) $\uparrow$				Coverage by successes ( $n/300$ ) $\uparrow$
Method	Combat	Harvest	Overall	[95% CI]	$\geq$ 1 of 8	$\geq$ 2 of 8	$\geq$ 4 of 8
JARVIS-VLA	7.56	5.00	6.71	[5.38, 8.08]	95/300	41/300	5/300
UI-TARS-1.5-7B	0.56	0.00	0.38	[0.13, 0.67]	8/300	1/300	0/300
NitroGen	0.00	0.00	0.00	[-]	0/300	0/300	0/300
Open-P2P-1.2B	0.44	0.00	0.29	[0.08, 0.50]	7/300	0/300	0/300
GameWAM	14.00	9.88	12.63	[10.79, 14.54]	155/300	79/300	16/300

Table 2: Task success and coverage on MineDojo-300. Results use 8 trials per task. Coverage counts tasks meeting the success thresholds; brackets show 95% task-bootstrap confidence intervals.

Table 6

Figure 4: Player-1-9B in the held-out commercial game Watch Dogs 2 : on-foot navigation, car approach, vehicle entry, and city driving. The sequence illustrates coordinated movement and camera control during navigation, and specified inputs for vehicle entry and driving. Reasoning and action text has been lightly shortened and reformatted for readability.

Figure 5: Training dynamics and the effect of supervised initialization. (a,b) Mean episode returns and game scores across 16 environments for 9B and 35B-A3B over 200 RL steps; faint traces show raw values and bold curves show EMA. (c) Base

\rightarrow

RL and SFT

\rightarrow

RL for 9B over 0–100 RL steps.

Figure 6: Per-environment episode returns for 9B and 35B-A3B over 200 RL steps. The curves show differences in learning progress, return scales, and relative model performance across diverse synthetic environments.

Figure 7: Environment scaling at a fixed budget of 25,600 episodes over 100 RL steps. Zero denotes the SFT reference.

Figure 8: Evolution of action composition during Player-1-Gym validation. Composite-action use rises from 64.7% to 93.2% over 100 RL steps. Bars show each checkpoint’s most frequent actions as shares of all turns. K-H: key hold; M-M: mouse movement; M-LC: mouse left click.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Semantics
Key hold / tap	Hold a set of keys throughout the step, or briefly press and release them.
Mouse movement	Move to an absolute screen position or apply a relative displacement.
Mouse hold / click	Hold one button throughout the step, or click one button. Different buttons may execute concurrently.
Mouse scroll	Apply one upward or downward scroll step.
Wait	No operation. Advance one step without keyboard or mouse input.

Appendix

Table 5: Game-control semantics. Inputs in a composite action execute concurrently within one step; bindings are restricted to those supported by the environment.

Figure 9: Sources and game distribution of the 6,044 gameplay demonstrations used for SFT. (a) Sources and scale, with sampled control hours. (b) Game distribution.

Source	Mapping
In-house	Recording-specific calibration coefficients.
D2E	Source-level $k_{x}=k_{y}=0.66$ for relative input; absolute endpoints for UI operations.
Pixel2Play	Hierarchical selection: session GUI fit $\rightarrow$ metadata profile $\rightarrow$ display scale.
VPT	$k_{x}=k_{y}=2$ in logical $1280\times 720$ coordinates for relative input; absolute endpoints for UI operations.
Gaming 500h	Recorded absolute positions projected using capture geometry.

Appendix

Table 6: Source-specific mouse-action mapping. Relative inputs use calibration coefficients bound to recording geometry; UI operations use absolute endpoints.

Environment	Task and control	Score components	State fields (API)
2D aiming	Track and click moving targets	Hits $+0.1\times$ accumulated precision	targets_collected , precision_total
2D shooting	Move, aim, and shoot	Enemies defeated	kills
2D collect dodge	Collect while avoiding hazards	Coins collected	coins_collected
2D lane driving	Time lane changes	Coins collected	coins_collected
2D circuit driving	Steer through a route	Checkpoints reached	checkpoints_passed
2D platforming	Run, jump, and dash	Coins, cleared gaps/spikes, and flags	coins_collected , gaps_cleared , spikes_cleared , flags_reached

Appendix

Table 7: 2D environments, controls, and verifiable score components. State fields are read from gameGymAPI.getState().metrics ; primary_score gives the resulting cumulative score.

Environment	Task and control	Score components	State fields (API)
3D aiming	Track and hit targets	Target hits	hits
3D combat arena	Coordinate movement, aim, and fire	Enemies defeated	kills
3D road driving	Follow bends and checkpoints	Checkpoints reached	checkpoints_passed
3D lane driving	Collect and avoid with lane changes	Coins collected	coins_collected
3D spear duel	Track, charge, release, and dodge	Spear hits $-0.5\times$ damage received	spear_hits , damage_taken
3D sword duel	Approach, aim, and strike	Attack hits + defeats $-0.5\times$ damage received	attack_hits , kills , damage_taken

Appendix

Table 8: 3D environments, controls, and verifiable score components. State fields are read from gameGymAPI.getState().metrics ; primary_score gives the resulting cumulative score.

(a) Appearance
Visual attribute	Native	Perturbed distribution / choice	Applies to
Entity hue shift	$0^{\circ}$	Uniform over $\{0,60,120,180,240,300\}^{\circ}$	All games
Entity saturation	1	$U(0.95,1.25)$	All games
Entity lightness	1	$U(0.96,1.15)$	All games
Background hue shift	$0^{\circ}$	$\operatorname{Tri}(-10,0,10)^{\circ}$	All games
Background saturation	1	$\operatorname{Tri}(0.55,1,1.05)$	All games

Appendix

Table 9: Appearance, camera, scene, and control-response randomization configurations. Multipliers are relative to each game’s native setting.

U

denotes uniform sampling and

\operatorname{Tri}

is parameterized by lower bound, mode, and upper bound. Rig parameters affect only camera components used by the game. Scene variation is task-specific and may already be present in native settings.

Panel of Figure 10 — Figure 10: Visual examples of Player-1-Gym randomization: 16 environments $\times$ seven views. Original uses default profiles; Appearance, Camera, and Scene change one domain; Joint 1–3 combine these three dimensions. Control-response variation is omitted because its effects require action sequences rather than static screenshots.

Setting	Player-1-9B	Player-1-35B-A3B
Actor / critic	Dense 9B / dense 9B	MoE 35B-A3B / dense 9B
SFT
Training trajectories	6,044
History (visual–action + action-only)	$2+2$
Sampling	Pool weights: nine action pools 1 each; Other 0.1
Epoch budget / global batch	2 / 512 turns

Appendix

Table 10: Post-training configuration. Both model sizes share the settings below except for actor architecture. Each RL step includes rollout collection and PPO updates; critic-only warmup is excluded.

Envs.	Environment subset	Episodes / env. / step
2	2D shooting; 3D navigation	128
4	2-env. subset + 2D circuit driving; 3D aiming	64
8	4-env. subset + 2D UI crafting; 2D platforming; 3D sword duel; 3D road driving	32
16	Full Player-1-Gym suite	16

Appendix

Table 11: Environment subsets for 9B scaling. All runs use 100 RL steps and 25,600 episodes (256 per step), allocated equally across environments.

Panel	Tasks / maps	Trials	Budget	Metrics
Player-1-Gym	16 games	128 / game	50 decisions	Mean episode return
MineDojo-300	300 tasks	8 / task	1,000 decisions	Success; task coverage counts
ViZDoom-10	10 maps	50 / map	150–2,100 decisions	Scaled return
GameWorld	170 tasks; 34 games	3 / task	400 decisions	Success; progress

Appendix

Table 12: Evaluation panels, budgets, and reporting units. Decision limits are per episode (per role in GameWorld). The three external benchmarks use environments outside Player-1-Gym .

Panel of Figure 11 — Figure 11: Example tasks and scenes from MineDojo-300.

Category	Map	Raw reward rule	Scale $c_{m}$
Precision shooting	Basic	$+106$ for eliminating the target; $-5$ per shot; $-1$ per tic.	100
	Defend the Center	$+1$ per kill; $-1$ on death.	10
	Defend the Line	$+1$ per kill; $-1$ on death.	10
Goal navigation	Deadly Corridor	Forward position along the corridor; $+1000$ for collecting armor; $-100$ on death.	1000
	My Way Home	$+1$ for reaching the goal; $-0.0001$ per tic.	1
Mobile combat	Battle 1	$+1$ per kill.	10

Appendix

Table 13: ViZDoom-10 map categories, raw reward rules, and fixed scales. Rewards accumulate over an episode; a tic is one simulator tick. Scaled Return divides the raw episode return by

c_{m}

Panel of Figure 12 — Figure 12: Example tasks and scenes from ViZDoom-10.

Panel of Figure 13 — Figure 13: Example tasks and scenes from GameWorld.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 201 papers

Sep 30, 2026

Code to Control: Synthesizing Parameterized Reactive Controllers

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.

Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1

Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

RL Control Language Model-Based Control LLM-Based Program Synthesis Derivative-Free Optimization

Ask PDF

Oct 2, 2026

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in web front-end execution, with browser-based game generation emerging as a particularly prominent frontier. While previous efforts frequently rely on complex multi-turn workflows or focus on static game evaluation benchmarks, this work targets direct end-to-end real-world game synthesis driven by coding agents. However, generating complex games directly from sparse user queries often forces coding agents to make underspecified assumptions, yielding incomplete mechanics, disconnected gameplay flows, and limited visual aesthetics. To resolve this issue, this paper presents GameGo, a scalable framework that systematically transforms brief game seeds into comprehensive Product Requirements Documents grounded in industry game-development practices. To retain core gameplay constraints without restricting design exploration, GameGo uses task-specific dynamic compression to maximize information density while preserving instruction following. Based on this pipeline, GameGoData is constructed with 55,060 development trajectories across 2D, 2.5D, and 3D games, alongside GameGoBench, a benchmark comprising 124 diverse game queries. Training GameGoCoder on GameGoData yields a model that outperforms matched baselines and is comparable to frontier models across gamedev benchmarks. All code, datasets, and models will be made publicly available.

Haoyue Yang, Jingyao Li, Zhengfan Wu +3

Institute of Automation, Chinese Academy of Sciences · Baidu Inc., Beijing, China

Agentic Code Generation Synthetic Data Generation for Language Models LLM Agent Trajectory Synthesis AI Agent Benchmarks

Ask PDF

Sep 8, 2026

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games

While many video-game environments (VGEs) have played crucial roles in advancing reinforcement learning (RL), developing novel VGEs or modifying existing ones to support new features, has been a laborious process requiring extensive hand-coding. Here we present PlayTrain, an RL framework that combines the abilities of large language models (LLMs) to robustly generate JavaScript (JS) games from a minimal human prompt, and an efficient pipeline that can run any JS game in a standard 'gym' environment. Not only are recent LLMs particularly good at writing JS code, but the JS format also allows users to easily play generated VGEs, while PlayTrain enables us to train RL agents on the exact same games. We demonstrate multiple use cases of PlayTrain, including cloning well-known Atari and ProcGen games in simple JS, where PlayTrain trains pixel-based agents end-to-end at over 1M agent-decisions per second on a single GPU node; and creating modified versions thereof (e.g., that support novel test sets, procedural generation logics, or game dynamics). Through PlayTrain, we reimagine RL VGE development: all we need is a single JS file, generated and modified through an LLM. We discuss promising future RL research directions that PlayTrain unlocks.

Ryan Truong, Lance Ying, Samuel J. Gershman +1

Harvard University, Cambridge, MA, USA · MIT, Department of Brain and Cognitive Sciences, Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Cambridge, MA, USA +1

Reinforcement Learning Game-Playing Agents RL for Game AI

Ask PDF

Abstract

Game-Playing Agents

Explore similar work

Code to Control: Synthesizing Parameterized Reactive Controllers

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games