Player-1: Learning to Play from Synthetic Experience

Mingyu Ouyang; Yiran Wang; Siyuan Hu; Xingyilang Yin; Huiqiang Sun; Guangzhi Wang; Junfu Pu; Cheng Cheng

cs.CVOct 14, 2026

Player-1: Learning to Play from Synthetic Experience

Authors: Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng,

Organizations: National University of Singapore · ARC Lab, Tencent IEG · Nanyang Technological University

Abstract

Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.

Figures & tables

Figure 1: Overview of Player-1 : learning game control beyond demonstrations. A shared game-control action space connects supervised initialization from gameplay demonstrations with multi-environment RL in Player-1-Gym . Learning from synthetic experience with verifiable rewards advances control capabilities beyond SFT, with gains that generalize beyond the RL training suite.

Method	Learning focus	Supervision	RL environments	Evaluation	Availability
Game-TARS	Large-scale pretraining	Reasoning + action	—	Held-out games	✗
SIMA 2	Embodied assistance	Reasoning + action	Games + world model	Held-out games	✗
NitroGen	System-1 generalist	Action	—	Games + adaptation	✓
Odysseus	Long-horizon RL	Reasoning + action	5 Mario levels	Mario games	✓
Player-1	Beyond demonstrations	Reasoning + action	16 synthetic envs	RL held-out games	✓

Table 1: Comparison of representative game-agent learning pipelines, including supervision, RL environments, evaluation scope, and pipeline and model availability.

Figure 2: Curated gameplay demonstrations for supervised initialization. (a) Recorded controls summarized by control type, number of inputs per control step, and pointer-motion mode in our demonstration trajectories. Two or more concurrent inputs form a composite action. (b) A demonstration sample pairing recorded actions with an annotated task instruction and visual reasoning.

Figure 3: Player-1-Gym : synthetic experience with verifiable rewards. (a) Shared control demands across five task families. (b) Visualization of sixteen 2D and 3D environments. (c) Examples of appearance, camera, and scene randomization applied to the same environment.

	Success rate (%) $\uparrow$				Coverage by successes ( $n/300$ ) $\uparrow$
Method	Combat	Harvest	Overall	[95% CI]	$\geq$ 1 of 8	$\geq$ 2 of 8	$\geq$ 4 of 8
JARVIS-VLA	7.56	5.00	6.71	[5.38, 8.08]	95/300	41/300	5/300
UI-TARS-1.5-7B	0.56	0.00	0.38	[0.13, 0.67]	8/300	1/300	0/300
NitroGen	0.00	0.00	0.00	[-]	0/300	0/300	0/300
Open-P2P-1.2B	0.44	0.00	0.29	[0.08, 0.50]	7/300	0/300	0/300
GameWAM	14.00	9.88	12.63	[10.79, 14.54]	155/300	79/300	16/300

Table 2: Task success and coverage on MineDojo-300. Results use 8 trials per task. Coverage counts tasks meeting the success thresholds; brackets show 95% task-bootstrap confidence intervals.

Table 6

Figure 4: Player-1-9B in the held-out commercial game Watch Dogs 2 : on-foot navigation, car approach, vehicle entry, and city driving. The sequence illustrates coordinated movement and camera control during navigation, and specified inputs for vehicle entry and driving. Reasoning and action text has been lightly shortened and reformatted for readability.

Figure 5: Training dynamics and the effect of supervised initialization. (a,b) Mean episode returns and game scores across 16 environments for 9B and 35B-A3B over 200 RL steps; faint traces show raw values and bold curves show EMA. (c) Base

\rightarrow

RL and SFT

\rightarrow

RL for 9B over 0–100 RL steps.

Figure 6: Per-environment episode returns for 9B and 35B-A3B over 200 RL steps. The curves show differences in learning progress, return scales, and relative model performance across diverse synthetic environments.

Figure 7: Environment scaling at a fixed budget of 25,600 episodes over 100 RL steps. Zero denotes the SFT reference.

Figure 8: Evolution of action composition during Player-1-Gym validation. Composite-action use rises from 64.7% to 93.2% over 100 RL steps. Bars show each checkpoint’s most frequent actions as shares of all turns. K-H: key hold; M-M: mouse movement; M-LC: mouse left click.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Semantics
Key hold / tap	Hold a set of keys throughout the step, or briefly press and release them.
Mouse movement	Move to an absolute screen position or apply a relative displacement.
Mouse hold / click	Hold one button throughout the step, or click one button. Different buttons may execute concurrently.
Mouse scroll	Apply one upward or downward scroll step.
Wait	No operation. Advance one step without keyboard or mouse input.

Appendix

Table 5: Game-control semantics. Inputs in a composite action execute concurrently within one step; bindings are restricted to those supported by the environment.

Figure 9: Sources and game distribution of the 6,044 gameplay demonstrations used for SFT. (a) Sources and scale, with sampled control hours. (b) Game distribution.

Source	Mapping
In-house	Recording-specific calibration coefficients.
D2E	Source-level $k_{x}=k_{y}=0.66$ for relative input; absolute endpoints for UI operations.
Pixel2Play	Hierarchical selection: session GUI fit $\rightarrow$ metadata profile $\rightarrow$ display scale.
VPT	$k_{x}=k_{y}=2$ in logical $1280\times 720$ coordinates for relative input; absolute endpoints for UI operations.
Gaming 500h	Recorded absolute positions projected using capture geometry.

Appendix

Table 6: Source-specific mouse-action mapping. Relative inputs use calibration coefficients bound to recording geometry; UI operations use absolute endpoints.

Environment	Task and control	Score components	State fields (API)
2D aiming	Track and click moving targets	Hits $+0.1\times$ accumulated precision	targets_collected , precision_total
2D shooting	Move, aim, and shoot	Enemies defeated	kills
2D collect dodge	Collect while avoiding hazards	Coins collected	coins_collected
2D lane driving	Time lane changes	Coins collected	coins_collected
2D circuit driving	Steer through a route	Checkpoints reached	checkpoints_passed
2D platforming	Run, jump, and dash	Coins, cleared gaps/spikes, and flags	coins_collected , gaps_cleared , spikes_cleared , flags_reached

Appendix

Table 7: 2D environments, controls, and verifiable score components. State fields are read from gameGymAPI.getState().metrics ; primary_score gives the resulting cumulative score.

Environment	Task and control	Score components	State fields (API)
3D aiming	Track and hit targets	Target hits	hits
3D combat arena	Coordinate movement, aim, and fire	Enemies defeated	kills
3D road driving	Follow bends and checkpoints	Checkpoints reached	checkpoints_passed
3D lane driving	Collect and avoid with lane changes	Coins collected	coins_collected
3D spear duel	Track, charge, release, and dodge	Spear hits $-0.5\times$ damage received	spear_hits , damage_taken
3D sword duel	Approach, aim, and strike	Attack hits + defeats $-0.5\times$ damage received	attack_hits , kills , damage_taken

Appendix

Table 8: 3D environments, controls, and verifiable score components. State fields are read from gameGymAPI.getState().metrics ; primary_score gives the resulting cumulative score.

(a) Appearance
Visual attribute	Native	Perturbed distribution / choice	Applies to
Entity hue shift	$0^{\circ}$	Uniform over $\{0,60,120,180,240,300\}^{\circ}$	All games
Entity saturation	1	$U(0.95,1.25)$	All games
Entity lightness	1	$U(0.96,1.15)$	All games
Background hue shift	$0^{\circ}$	$\operatorname{Tri}(-10,0,10)^{\circ}$	All games
Background saturation	1	$\operatorname{Tri}(0.55,1,1.05)$	All games

Appendix

Table 9: Appearance, camera, scene, and control-response randomization configurations. Multipliers are relative to each game’s native setting.

U

denotes uniform sampling and

\operatorname{Tri}

is parameterized by lower bound, mode, and upper bound. Rig parameters affect only camera components used by the game. Scene variation is task-specific and may already be present in native settings.

Panel of Figure 10 — Figure 10: Visual examples of Player-1-Gym randomization: 16 environments $\times$ seven views. Original uses default profiles; Appearance, Camera, and Scene change one domain; Joint 1–3 combine these three dimensions. Control-response variation is omitted because its effects require action sequences rather than static screenshots.

Setting	Player-1-9B	Player-1-35B-A3B
Actor / critic	Dense 9B / dense 9B	MoE 35B-A3B / dense 9B
SFT
Training trajectories	6,044
History (visual–action + action-only)	$2+2$
Sampling	Pool weights: nine action pools 1 each; Other 0.1
Epoch budget / global batch	2 / 512 turns

Appendix

Table 10: Post-training configuration. Both model sizes share the settings below except for actor architecture. Each RL step includes rollout collection and PPO updates; critic-only warmup is excluded.

Envs.	Environment subset	Episodes / env. / step
2	2D shooting; 3D navigation	128
4	2-env. subset + 2D circuit driving; 3D aiming	64
8	4-env. subset + 2D UI crafting; 2D platforming; 3D sword duel; 3D road driving	32
16	Full Player-1-Gym suite	16

Appendix

Table 11: Environment subsets for 9B scaling. All runs use 100 RL steps and 25,600 episodes (256 per step), allocated equally across environments.

Panel	Tasks / maps	Trials	Budget	Metrics
Player-1-Gym	16 games	128 / game	50 decisions	Mean episode return
MineDojo-300	300 tasks	8 / task	1,000 decisions	Success; task coverage counts
ViZDoom-10	10 maps	50 / map	150–2,100 decisions	Scaled return
GameWorld	170 tasks; 34 games	3 / task	400 decisions	Success; progress

Appendix

Table 12: Evaluation panels, budgets, and reporting units. Decision limits are per episode (per role in GameWorld). The three external benchmarks use environments outside Player-1-Gym .

Panel of Figure 11 — Figure 11: Example tasks and scenes from MineDojo-300.

Category	Map	Raw reward rule	Scale $c_{m}$
Precision shooting	Basic	$+106$ for eliminating the target; $-5$ per shot; $-1$ per tic.	100
	Defend the Center	$+1$ per kill; $-1$ on death.	10
	Defend the Line	$+1$ per kill; $-1$ on death.	10
Goal navigation	Deadly Corridor	Forward position along the corridor; $+1000$ for collecting armor; $-100$ on death.	1000
	My Way Home	$+1$ for reaching the goal; $-0.0001$ per tic.	1
Mobile combat	Battle 1	$+1$ per kill.	10

Appendix

Table 13: ViZDoom-10 map categories, raw reward rules, and fixed scales. Rewards accumulate over an episode; a tic is one simulator tick. Scaled Return divides the raw episode return by

c_{m}

Panel of Figure 12 — Figure 12: Example tasks and scenes from ViZDoom-10.

Panel of Figure 13 — Figure 13: Example tasks and scenes from GameWorld.

Game-Playing Agents

Game-playing agents use search, learning, and reasoning to choose actions in games with strategic, uncertain, or changing environments.

See all 201 papers

Code to Control: Synthesizing Parameterized Reactive Controllers
Sep 30, 2026Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1RL Control Language Model-Based Control
Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition
GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets
Oct 2, 2026Haoyue Yang, Jingyao Li, Zhengfan Wu +3Agentic Code Generation Synthetic Data Generation for Language Models
Institute of Automation, Chinese Academy of Sciences · Baidu Inc., Beijing, China
PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games
Sep 8, 2026Ryan Truong, Lance Ying, Samuel J. Gershman +1Reinforcement Learning Game-Playing Agents
Harvard University, Cambridge, MA, USA · MIT, Department of Brain and Cognitive Sciences, Cambridge, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Cambridge, MA, USA +1

Sep 30, 2026J/K move · Enter open · S save

Code to Control: Synthesizing Parameterized Reactive Controllers

Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates, Samuel J. Gershman

Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

RL Control Language Model-Based Control LLM-Based Program Synthesis Derivative-Free Optimization

Open paper AskPDF arXiv

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.

Abstract

Game-Playing Agents

Explore similar work

Code to Control: Synthesizing Parameterized Reactive Controllers

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games