We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.
Figures & tables
Figure 1: Overview of Gamer. A generalist game agent framework operates across a broad range of visually and mechanically diverse games. At each step, the agent uses the current screen, task context, recent interaction history, long-term memory, and reasoning to generate native keyboard and mouse actions in real time. The agent interacts continuously through this loop. Cross-game training, multi-level supervision, and persistent memory support transferable control and sustained task execution.
Method
Initialization
Training Recipe
Training Data
Real-Time
Open-Source
VPT
From scratch
BC + RL
Minecraft
✓
✓
Open-P2P
From scratch
BC
Diverse
✓
✓
NitroGen
From scratch
BC
Diverse
✗
✓
Game-TARS
Seed-VL-1.5
CPT + SFT
Diverse
✗
✗
Lumine
Qwen2-VL
CPT + SFT
Genshin Impact
✓
✗
SIMA 2
Gemini 2.5 Pro + Flash-Lite
SFT + RL
Diverse
✗
✗
Table 1: Comparison between Gamer and other representative game agents in terms of model initialization, training recipe, training data, real-time inference, and open-source availability. Here, BC, SFT, CPT, and RL denote behavior cloning, supervised fine-tuning, continued pre-training, and reinforcement learning, respectively. We open-source our inference engine and checkpoints for CPT scaling behaviors.
Figure 2: CPT for unified keyboard-mouse control. Synchronized gameplay observations and device events are mapped to a shared action representation. Random key remapping reduces reliance on key identity, while repeat-aware loss decay reweights temporally persistent actions. The resulting control prior is used by the subsequent task-conditioned training stages.
Figure 3: Serialized device-level action. For each observation interval, the policy predicts accumulated mouse movement and six sequential sets of active keys and buttons.
Figure 4: Sparse reasoning and cross-context memory in supervised fine-tuning. The shared autoregressive policy integrates visual observations, task memory, optional reasoning, and executable keyboard-mouse actions. Context memory compaction preserves task progress across long interactions, while adaptive reasoning allows the policy to act directly at routine steps and generate explicit reasoning when needed.
Figure 5: Memory-linked long-horizon optimization with TA-GRPO . A long interaction may exceed the model’s context budget (top). Memory compaction divides online rollouts into finite segments whose task states remain connected by generated summaries (bottom). Segments at corresponding stages are compared using environment-verifiable outcomes, and temporal credit assignment distributes their group-relative advantage across decisions. The figure is schematic, and model execution uses the real-time serving backend in Sec. 5 .
Figure 6: Streaming-vLLM for persistent multi-environment inference. Each game client maintains separate short-term history, long-term memory, and logical KV-cache state. Paged allocation and memory-aware KV-cache reuse avoid repeated context processing, while incremental rotary position embedding (RoPE) refresh corrects retained keys after compaction. The runtime combines architecture-aware execution, BF16-verified FSM speculation, and early action dispatch to serve concurrent real-time interactions.
Figure 7: Bottom-up construction of multi-level task supervision. The diagram illustrates the organization of aligned gameplay segments into local operations, intermediate subgoals, and extended task stages. Separate GPT-5.6-Luna passes produce sparse Think annotations, segment-level instructions and scene descriptions, and memory updates as detailed in Secs. 6.2 - 6.3 . The diagram focuses on hierarchy assembly rather than depicting every annotation call.
Figure 8: Multi-stage data curation for training. Gameplay trajectories and general vision-language data are cleaned, sampled, and filtered to construct the CPT corpus. Trajectories with aligned action annotations are further used to construct SFT examples at three levels: short-horizon operations, medium-horizon subgoals, and long-horizon tasks. For RL, online policy interaction provides trajectories and environment-verifiable outcomes.
Stage
Data
Supervision / Usage
Size
Sampling
CPT
VPT, D2E, OpenP2P
Cross-game generalization
∼ 2.4K hours
85%
Honkai: Star Rail (HSR)
LLaVA-OneVision, Zebra-CoT, GameQA
Vision–language
∼ 6.3B tokens
15%
Embodied-Reasoner, Cambrian-S
Embodied and egocentric
SFT
Operation-Level HSR
Local operations
∼ 261K samples
37.0%
Subgoal-Level HSR
Intermediate subgoals
∼ 211K samples
29.8%
Table 2: Overview of training data used for building Gamer. The table summarizes the major data sources, their role in each training stage, and the corresponding data scale.
Figure 9: Representative long-horizon side quests in the Honkai: Star Rail evaluation suite. (a) Opportunity Makes the Thief (Part I) contains 30 milestones with navigation, dialogue, combat, and UI interaction. (b) Bestial Ferocity contains 55 milestones and additionally includes three mechanical-repair puzzles. Both panels show ordered milestone sequences and configured step budgets rather than measured completion times.
Figure 10: Overview of the minigame benchmark in the Honkai: Star Rail evaluation suite. The benchmark contains 40 task instances across seven game types. The left panel shows representative gameplay examples, instance counts, and mechanics. The upper-right chart shows the number of instances per type, while the lower-right chart groups minigames by four descriptive primary capability tags, with instance totals in parentheses. Each minigame type is assigned one primary tag for counting and may require additional abilities. Screenshots illustrate game types and are not necessarily drawn from the evaluated instances.
Scaling Dimension
Controlled Configurations
Research Focus
Model size N
Qwen3.5 0.8B, 2B, 9B, 27B
Capacity and data efficiency
Gameplay budget D
50, 100, 200, 400, 800, 1,600 hours
Gains from additional interaction
Game mixture
S1-S5
Effect of target-game coverage
Table 3: Experimental configurations for the GameScaling CPT study. The study varies model size, gameplay budget, and game mixture across 120 configurations.
Validation
q
R2
RMSE
Within R2
Minecraft
wMC
0.982
0.0020
0.912
Training games
1−wMC
0.995
0.0012
0.959
Table 4: Fit accuracy of the composition-aware scaling relation. Within-mixture R2 measures variation explained by model size and data budget after removing mean differences across mixtures.
Model
MCU ↑
ViZDoom ↑
OmniGameArena ↑
GameWorld ↑
Average ↑
GPT-5.6-Sol
8.7 ( − 73.3 )
2.7 ( − 24.0 )
14.5 ( − 31.1 )
44.3 ( − 9.6 )
17.6 ( − 34.5 )
GPT-5.6-Luna
5.3 ( − 48.7 )
2.9 ( − 21.2 )
13.2 ( − 13.5 )
41.6 ( − 3.3 )
15.8 ( − 21.6 )
Claude-Opus-5.0
8.7 ( − 48.0 )
2.0 ( − 18.8 )
16.4 ( − 28.0 )
42.0 ( − 10.8 )
17.3 ( − 26.4 )
GLM-5.3-Flash
7.3 ( − 40.0 )
1.9 ( − 17.0 )
12.3 ( − 11.2 )
29.6 ( − 4.4 )
12.8 ( − 18.1 )
DeepSeek-V4-Flash-Vision
7.3 ( − 28.7 )
3.3 ( − 9.7 )
11.6 ( − 15.5 )
28.5 ( − 13.3 )
12.7 ( − 16.8 )
Qwen3.5-0.8B
0.0 ( + 0.0)
1.0 ( + 0.1 )
1.8 ( + 0.1 )
15.1 ( + 1.3 )
4.5 ( + 0.4 )
Table 5: Cross-game performance of CPT policies under real-time interaction. The final two rows report CPT checkpoints based on Qwen3-VL-8B Dense and Qwen3.5-35B-A3B MoE. The main value is the real-time (RT) score, while parentheses show its difference from idealized time-paused (TP) evaluation, computed as RT − TP. Green indicates a positive difference, red a negative difference, and zero denotes no change. Higher scores are better. The performance of controlled GameScaling-S3 checkpoints is not included in this table.
Model
Opportunity Makes the Thief (I)
Bestial Ferocity
Nav.
Dialogue
Combat
UI
Acc ↑
Nav.
Dialogue
Combat
UI
Puzzle
Acc ↑
Gemini-3.8-Flash
0.0
35.7
0.0
0.0
16.7
31.2
81.2
50.0
50.0
0.0
45.5
GPT-5.6-Sol
7.1
28.6
0.0
0.0
16.7
28.1
68.8
50.0
100.0
0.0
41.8
GPT-5.6-Luna
7.1
14.3
0.0
0.0
10.0
18.8
43.8
0.0
0.0
0.0
23.6
Claude-Opus-5.0
7.1
21.4
0.0
0.0
13.3
15.6
56.2
50.0
50.0
0.0
29.1
GLM-5.3-Flash
7.1
21.4
0.0
0.0
13.3
15.6
43.8
50.0
0.0
0.0
23.6
Table 6: Long-horizon instruction following across two side quests. To improve evaluation robustness, we evaluate each task three times and report the averaged success rate (%) of evaluated tasks for each interaction type. Overall accuracy (Acc) divides successful task executions by all evaluated task executions, pooling repeated attempts over the 30 or 55 milestone units. Opportunity Makes the Thief (Part I) and Bestial Ferocity contain 30 and 55 milestones, respectively.
Model
Minigame Types
Acc ↑
ML
NC
AC
HX
JHP
GS
JM
GPT-6.1-Sol
20.0
37.5
0.0
33.3
14.3
0.0
14.3
17.1
Gemini-3.8-Flash
0.0
0.0
0.0
0.0
0.0
0.0
14.3
2.0
GPT-5.6-Sol
0.0
12.5
0.0
0.0
57.1
0.0
14.3
12.0
GPT-5.6-Luna
0.0
37.5
0.0
0.0
28.6
0.0
14.3
11.5
Claude-Opus-5.0
0.0
12.5
0.0
0.0
57.1
0.0
0.0
9.9
Table 7: Instruction generalization across seven minigame types. To improve evaluation robustness, we evaluate each task three times and report the averaged task success rates (%) on a reduced benchmark of 40 task instances. Each task score is the mean of its valid evaluation attempts. Acc is the unweighted average across the seven game types. ML, NC, AC, HX, JHP, GS, and JM denote Magflow Link, Navigation Compass, Abacus Circuitry, Hexanexus, Janus’s Hidden Passage, Golden Scapegoat, and Janus’s Maze, respectively.
Figure 11: Real-time inference improves online RL learning across eight navigation tasks. Task labels indicate navigation targets. For example, “Pom-Pom” refers to the task of finding the NPC Pom-Pom. Orange solid lines show rewards under the original real-time Streaming-vLLM setting, while blue dashed lines show the 0.6-FPS control representing standard vLLM rollout speed. Real-time inference typically achieves earlier reward improvement and stronger learning over the first 50 plotted training iterations.
Method
Backbone
StreamingBench ↑
OVO-Bench ↑
Latency ↓
Memory
(Real-time)
(Real / Backward / Forward)
ReKV [ 10 ]
LLaVA-OV-7B
71.06
53.9 (62.0 / 48.5 / 51.2)
0.23s
21 GB
StreamingVLM [ 54 ]
76.92
55.6 (68.4 / 46.0 / 52.9)
0.21s
16 GB
DSCache [ 34 ]
79.12
57.5 (71.5 / 47.8 / 53.1)
0.30s
16 GB
InfiniPot-V [ 15 ]
Qwen2.5-VL-7B
76.40
53.6 (65.9 / 47.6 / 47.9)
0.37s
17 GB
StreamingVLM [ 54 ]
78.56
52.2 (67.8 / 42.3 / 46.6)
0.31s
17 GB
Table 8: Accuracy, latency, and memory consumption of streaming video understanding. StreamingBench Real-Time and OVO-Bench scores are reported with query latency and peak GPU memory. Latency and memory consumption are measured on an NVIDIA H200.
Configuration
ViT ↓
Prefill ↓
Decode ↓
Latency (ms) ↓
SR (Hz) ↑
CR (Hz) ↑
Baseline (TP1)
27.90
94.70
1087.00
1306.00
0.77
4.62
+ Stateful Streaming Engine (TP4)
68.89
121.85
874.07
1264.76
0.79
4.74
+ Basic Fusion ‡
60.67
154.80
630.51
1042.76
0.96
5.76
+ Parallel Overlap
55.74
144.48
507.37
831.82
1.20
7.20
+ CUDA Graph
64.06
140.31
179.79
536.66
1.86
11.16
+ Megakernel
16.61
24.19
35.97
87.68
11.41
68.46
Table 9: Cumulative Streaming-vLLM optimization on NVIDIA H200 (Hopper) with Qwen3.5-35B-A3B MoE. Latencies are in milliseconds. SR is the step rate, and CR is the theoretical six-block control rate derived from SR. The change from TP1 to TP4 incurs communication and synchronization overhead, so per-component speedups need not be monotonic.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Mixture
Worlds
MC share
Largest share
Eff. worlds
Inv. Simpson
S1
1
100.0%
100.0%
1.0
1.0
S2
80
51.4%
51.4%
14.5
3.7
S3
80
27.1%
27.1%
35.2
11.9
S4
80
2.8%
4.2%
60.1
52.6
S5
79
0.0%
4.3%
58.7
51.0
Appendix
Table 10: Measured diversity of the CPT game mixtures. World labels are canonicalized across sources. Effective worlds denotes exponential Shannon diversity, while the inverse Simpson index emphasizes high-probability worlds.
Figure 12: Early adaptation to the shared keyboard-mouse interface. The figure reports keyboard, mouse, and action-format losses together with directional prediction accuracy for 2B, 9B, and 27B models under the S1 and S4 mixtures. The horizontal axis denotes cumulative gameplay hours. Results from GameScaling [ 58 ] .
Figure 13: CPT scaling depends on model capacity, interaction budget, and game composition. The rows show training loss, Minecraft validation loss, validation loss on other training games, and validation loss on unseen games. Columns correspond to model scale, while S1-S5 denote different game mixtures. The final column compares fitted and measured losses. Results from GameScaling [ 58 ] .
Figure 14: Gameplay benchmark scaling across training budgets. Rows show MCU, ViZDoom, OmniGameArena, and GameWorld, and columns show the four Qwen3.5 Dense scales. Colored curves compare the S1-S5 training mixtures across 50-1,600 gameplay hours, with dashed lines denoting the unadapted backbone. Multi-game mixtures support transfer to related environments. Scores are measured under time-paused interaction. Results from GameScaling [ 58 ] .
Parameter
Minecraft
Training Games
q=wMC
q=1−wMC
E
0.047
0.136
CN
0.019
0.014
a
0.15
0.17
CD
0.106
0.078
b
0.16
0.14
Appendix
Table 11: Fitted parameters of the composition-aware CPT scaling relation. Separate parameter sets are estimated for Minecraft validation loss and validation loss over the remaining training-game distribution.
Held-Out Block
Configurations Used for Fitting
Minecraft RMSE
Training-Game RMSE
27B model
0.8B, 2B, 9B
0.0026
0.0016
1,600 hours
50-800 hours
0.0027
0.0018
50 hours
100-1,600 hours
0.0025
0.0022
Mixture S2
S1, S3, S4, S5
0.0020
0.0016
Mixture S3
S1, S2, S4, S5
0.0021
0.0008
Mixture S4
S1, S2, S3, S5
0.0057
0.0005
Appendix
Table 12: Block-wise holdout prediction error for the CPT scaling relation. Each row excludes the indicated block during fitting and evaluates RMSE only on that held-out configuration.
Figure 15: Scaling-law fits with predictive uncertainty. The panels show the training, Minecraft, and in-corpus loss laws across S1-S5, with measurements, predictions, and stratified-bootstrap bands. Prediction bands reflect variation across resampled experimental configurations within the observed grid, not seed-to-seed variance in model training.
Figure 16: Frontier-model progress on four public game benchmarks. Scores are plotted against model release date under time-paused evaluation. Circles indicate the GPT series, squares the Qwen series, and stars the GameScaling-S3 checkpoints at 1,600 gameplay hours (marked GS in the original plot). These results are from the controlled scaling study, not the HSR-augmented Gamer-CPT model. The comparison shows gains from gameplay adaptation alongside the contribution of general backbone capabilities, particularly on distant worlds.
Figure 17: Test-time scaling on MCU. Cumulative task success versus available interaction steps (up to 3,000) for CPT checkpoints and unadapted Qwen3.5 backbones. The primary MCU protocol uses a 600-step cap. This supplementary sweep illustrates how additional interaction opportunities affect task completion.
Figure 18: Incremental RoPE refresh for retained KV states. A cached key moved from logical position p to p is updated by an additional rotation determined by Δp=p−p . The same positional offset is applied independently to each RoPE frequency pair and written back in place, while the value cache remains unchanged. The figure illustrates Δp<0 , corresponding to a leftward shift after context compaction.
Figure 19: Cross-backbone request latency on NVIDIA A800 (Ampere) and H200 (Hopper). The existing sweep compares Qwen3-VL-8B (Dense) and Qwen3.5-35B-A3B (MoE) across tensor-parallel degrees and BF16 or W8A8 execution. Stacked values separate vision processing, context (prefill), generation (decode), and remaining overhead, measured as mean request latency. These results characterize representative hardware and backbone coverage rather than all supported model sizes. The theoretical control rate in Table 9 is derived from request time, not measured at the client.
Figure 20: Exploration diversity in Honkai: Star Rail . Map layouts vary in geometry, connectivity, and floor structure, and point icons indicate resources and other interactive locations. Red paths illustrate the possible exploration routes.
Figure 21: Combat mechanics and character roles in Honkai: Star Rail . The examples illustrate (1) the combat interface, including action order, HP and Toughness, team status, skill points, and available abilities, (2) a Weakness Break during a boss encounter, (3) the Action Value display, and (4) a character Data Bank snapshot. The lower panel summarizes the nine Paths by representative combat roles, including damage, support, sustain, and more specialized mechanics. Path groupings summarize representative combat roles, and the figure provides game-mechanics context rather than a measured policy rollout.
ID
Milestone
Type
s
1
Follow the quest marker to locate Pela.
N
60
2
Talk to Pela and transition into the museum.
D
60
3
Continue the conversation with Pela inside the museum.
D
60
4
Follow Pela to the exhibit.
N
120
5
Complete the conversation and transition to the Underworld.
D
60
Appendix
Table 13: Opportunity Makes the Thief (Part I). The 30 required milestones span 22 parent steps and cover navigation, dialogue, combat, and exhibit recovery. Read down the left block, then the right block.
ID
Milestone
Type
s
1
Locate the unidentified NPC in Central Starskiff Haven.
N
90
2
Complete the initial NPC dialogue.
D
90
3
Teleport to Exalting Sanctum.
N
90
4
Enter the Realm-Keeping Commission Chancery.
N
90
5
Locate Jingyan.
N
90
Appendix
Table 14: Bestial Ferocity. The 55 required milestones span 40 parent steps and include three automaton-repair puzzles. Read down the left block, then the right block.
Figure 22: Representative minigames and interaction challenges in the Honkai: Star Rail evaluation suite. This extended screenshot gallery illustrates the range of game interactions in Honkai: Star Rail. It is a qualitative visualization rather than the scored minigame inventory. The quantitative benchmark contains seven scored minigame types and 40 tasks, summarized in Figure 10 and Table 7 .
Figure 23: Navigation across RL checkpoints on a training task (Reach Pom-Pom). Each row shows one representative rollout with twelve temporally ordered observations, actions, and translated reasoning excerpts. Task successes reports successful attempts out of 16 for each checkpoint: 3, 5, 14, 13, 15, and 15 for CP 0-50, respectively. Red boxes mark scene details or interaction prompts, and blue shading marks reasoning excerpts. The illustrated CP 0 and CP 10 attempts time out, while CP 20-50 reach the target and complete the interaction. Frame counts and completion times describe only the displayed rollout.
Figure 24: Navigation across RL checkpoints on a development task (Reach March 7th). Each row presents one representative rollout from the same task and initial scene, with twelve ordered observations, actions, and translated reasoning excerpts. Task successes counts successful attempts out of 16: 1, 1, 8, 6, 8, and 10 for CP 0-50, respectively. The illustrated CP 0, CP 10, and CP 30 attempts time out, whereas CP 20, CP 40, and CP 50 complete the target dialogue. Aggregate successes and representative trajectories provide complementary evidence of navigation and interaction progress.
Stage I
Stage II
Stage III
Phase
CPT
SFT
RL
Model and Context Configuration
Qwen3-VL (supported)
Dense: 2B, 4B, 8B, 32B, MoE: 30B-A3B
Qwen3.5 (supported)
Dense: 0.8B, 2B, 9B, 27B, MoE: 35B-A3B
Controlled CPT Scales
0.8B / 2B / 9B / 27B
–
–
Image Resolution
854×480
854×480
854×480
Appendix
Table 15: Training recipes for Gamer. Stage I reports the controlled GameScaling reference on four Qwen3.5 Dense sizes. The HSR experiments use a separately trained Gamer-CPT checkpoint with additional Honkai: Star Rail data and the same 85/15 gameplay-VLM token mixture, followed by SFT and TA-GRPO. RL run-dependent settings are reported for navigation (nav) and puzzle training (puzzle). RL token totals count generated response tokens, whereas supervised target tokens per update follow a different accounting convention.
Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.
Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.
Mingyu Ouyang, Yiran Wang, Siyuan Hu +9
National University of Singapore · ARC Lab, Tencent IEG · Nanyang Technological University
We present ABot-World-0, an action-conditioned video world model for real-time, long-horizon closed-loop interaction, supported by a multi-source data infrastructure spanning AAA games, simulation engines, and internet videos to learn controllable world dynamics. WorldExplorer performs agent-driven collection guided by training feedback, while a unified pipeline applies 14 deterministic quality checks, VLM-based assessment, and synchronized action and text annotation. We progressively distill a bidirectional action-conditioned teacher into a causal student through teacher forcing and ODE distillation, and introduce LongForcing to align long student self-rollouts with an extended-horizon teacher, mitigating accumulated distribution shift and autoregressive drift. Raw keyboard actions provide a unified control interface for scene roaming and third-person character interaction, while reference-character memory provides persistent appearance cues for identity consistency during third-person rollouts. For deployment, we co-design a streaming inference stack with a lightweight VAE decoder, efficient attention, memory-aware scheduling, and low-bit DiT inference. Across optimized low-bit configurations, ABot-World-0 streams 720P video at up to 16 FPS on a single NVIDIA RTX 5090 desktop GPU, with 1.2s action-to-first-frame latency and approximately 19GiB peak VRAM. Experiments on WorldRoamBench and extended interactive rollouts demonstrate competitive controllability and coherent long-horizon world evolution.