Game-Enhanced Embodied Learning with Event and Foresight Pretraining
Organizations: Beijing Xinying Suixing Technology Co., Ltd.
Abstract
Human gameplay supplies paired observations, actions, and visual consequences that can supplement robot-policy training. We study event-and-foresight pretraining for InternVLA-A1.5, combining recorded keyboard/mouse event prediction with history-conditioned visual supervision through a frozen video model. Coverage-preserving anchor selection constructs one-second targets, and an attention mask prevents teacher-forced future actions from entering foresight queries. The selected corpus contains 394 recordings across 20 games, with 223.36 hours of recorded timestamp span and 2,072,084 training anchors. After 12,400 game-pretraining updates, we compare the resulting initialization with the public base under the same suite-specific robot fine-tuning protocol. Across four LIBERO suites, aggregate success increases from 1,718/2,000 (85.90%) to 1,811/2,000 (90.55%), a gain of 4.65 percentage points. Suite gains are 9.2, 6.8, 0.2, and 2.4 points on Spatial, Object, Goal, and Long. The 40-task breakdown shows 24 improvements, 10 declines, and six ties. These results support the practical value of the complete game-pretraining stage in this evaluated setting, while revealing heterogeneous task outcomes. The comparison includes additional optimization and does not isolate the effects of game data, the two objectives, or training variability.
Figures & tables
| Work | Established ingredient | Adaptation evaluated here |
|---|---|---|
| D2E ( Choi et al., 2026 ) | Desktop events and vision–action pretraining for embodied transfer | Recorded human events paired with future game frames for joint event-and-foresight training |
| InternVLA-A1.5 ( Ma et al., 2026 ) | Native robot expert, foresight queries, and frozen-video supervision | Causal game-event targets, history-only foresight conditioning, and transfer back to the robot path |
| This study | Both ingredients above | Coverage-preserving game samples, a joint training interface, and a fixed downstream initialization comparison |
| Quantity | Count / duration |
|---|---|
| Games / recorded episodes | 20 / 394 |
| Recorded span / active intervals (hours) | 223.36 / 197.85 |
| Raw event rows | 22,646,179 |
| Legal dense samples after validity checks | 11,118,648 |
| Samples after 250 ms anchor selection | 2,356,278 |
| Valid anchors restored for action coverage | 3,038 |
| Stage | Direct Base | Game-pretrained initialization |
|---|---|---|
| Starting model | InternVLA-A1.5-base | InternVLA-A1.5-base |
| Game supervision | None | Human event tokens and future frames |
| Additional updates | 0 | 12,400, global batch 128 |
| Robot objective | Native action, language, and video losses | Same downstream objectives |
| Robot updates | 20,000 per suite | 20,000 per suite |
| Evaluation | 500 episodes per suite | Same evaluation protocol |
| Suite | Base | Game initialization | pp | |
|---|---|---|---|---|
| Spatial | 417 (83.4%) | 463 (92.6%) | +46 | +9.2 |
| Object | 460 (92.0%) | 494 (98.8%) | +34 | +6.8 |
| Goal | 442 (88.4%) | 443 (88.6%) | +1 | +0.2 |
| Long | 399 (79.8%) | 411 (82.2%) | +12 | +2.4 |
| Overall / 2,000 | 1,718 (85.90%) | 1,811 (90.55%) | +93 | +4.65 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Game | Records | Span | Active | Train anchors | Val. anchors |
|---|---|---|---|---|---|
| Apex Legends | 36 | 24.95 | 20.11 | 234,160 | 21,541 |
| Barony | 11 | 9.25 | 9.20 | 101,077 | 12,558 |
| Brotato | 13 | 5.93 | 5.81 | 71,558 | 6,057 |
| Core Keeper | 19 | 8.87 | 8.13 | 85,467 | 14,486 |
| Counter-Strike 2 | 10 | 9.85 | 6.81 | 69,633 | 4,041 |
| Cyberpunk 2077 | 33 | 13.98 | 11.95 | 125,783 | 10,992 |
| Setting | Value |
|---|---|
| Initialization | Expanded public InternVLA-A1.5-base |
| Human-event anchors | 2,072,084 |
| Batch | 8 devices 1 sample 16 accumulation steps = 128 |
| Optimizer | AdamW; learning rate ; betas ; epsilon |
| Regularization | Weight decay .01; gradient norm clipping 1.0 |
| Schedule / seed | 200-update linear warmup, then constant; seed 42 |
| Setting | Value |
|---|---|
| Training organization | One policy per suite and initialization |
| Budget and batch | 20,000 updates; two GPUs 16 samples = 32 |
| Learning-rate schedule | Peak ; endpoint ; 400 effective warmup updates |
| Optimizer preset | AdamW; betas ; epsilon ; weight decay .01; clipping norm 1.0 |
| Training seed / arithmetic | 42; BF16; gradient checkpointing enabled |
| Robot targets | Absolute actions; state tokens; FAST/language supervision; video weight 1 |
| ID | Original task name | Base | Game | pp |
|---|---|---|---|---|
| S01 | pick up the black bowl between the plate and the ramekin and place it on the plate | 42 | 49 | +14 |
| S02 | pick up the black bowl next to the ramekin and place it on the plate | 47 | 47 | +0 |
| S03 | pick up the black bowl from table center and place it on the plate | 49 | 49 | +0 |
| S04 | pick up the black bowl next to the plate and place it on the plate | 49 | 47 | -4 |
| S05 | pick up the black bowl on the wooden cabinet and place it on the plate | 39 | 47 | +16 |
| S06 | pick up the black bowl on the cookie box and place it on the plate | 39 | 48 | +18 |
| ID | Original task name | Base | Game | pp |
|---|---|---|---|---|
| O01 | pick up the alphabet soup and place it in the basket | 45 | 49 | +8 |
| O02 | pick up the cream cheese and place it in the basket | 50 | 50 | +0 |
| O03 | pick up the salad dressing and place it in the basket | 45 | 50 | +10 |
| O04 | pick up the bbq sauce and place it in the basket | 43 | 49 | +12 |
| O05 | pick up the ketchup and place it in the basket | 43 | 49 | +12 |
| O06 | pick up the tomato sauce and place it in the basket | 42 | 48 | +12 |
| ID | Original task name | Base | Game | pp |
|---|---|---|---|---|
| G01 | open the middle drawer of the cabinet | 50 | 49 | -2 |
| G02 | put the bowl on the stove | 48 | 46 | -4 |
| G03 | put the wine bottle on top of the cabinet | 40 | 49 | +18 |
| G04 | open the top drawer and put the bowl inside | 41 | 33 | -16 |
| G05 | put the bowl on top of the cabinet | 41 | 48 | +14 |
| G06 | push the plate to the front of the stove | 46 | 40 | -12 |
| ID | Original task name | Base | Game | pp |
|---|---|---|---|---|
| L01 | put both the alphabet soup and the tomato sauce in the basket | 42 | 24 | -36 |
| L02 | put both the cream cheese box and the butter in the basket | 50 | 48 | -4 |
| L03 | turn on the stove and put the moka pot on it | 48 | 50 | +4 |
| L04 | put the black bowl in the bottom drawer of the cabinet and close it | 47 | 49 | +4 |
| L05 | put the white mug on the left plate and put the yellow and white mug on the right plate | 14 | 30 | +32 |
| L06 | pick up the book and place it in the back compartment of the caddy | 43 | 46 | +6 |
| Configuration | (%) | Kbd F1 | Events | ||
|---|---|---|---|---|---|
| H-Base | 100 | .611 | .723 | .750 | 759 |
| All additions disabled | 100 | .611 | .723 | .750 | 759 |
| LoRA only | 100 | .607 | .788 | .749 | 627 |
| Time only | 52 | .597 | .724 | .460 | 592 |
| LoRA + time | 0 | .008 | .316 | .204 | 351 |
| Input position | Teacher forcing | Cached generation |
|---|---|---|
| Timed action: START, TYPE, | Unknown | Unknown |
| Timed action: , payload, END | Known | Known |
| Held event: START | Known | Unknown |
| Held event: held TYPE, payload, END | Known | Known |
| Setting | T-Real / T-Unknown / T-Neutral | No-Time pilot |
|---|---|---|
| Train / validation rows | 4,096 / 512 | 4,096 / 512 |
| Optimizer updates / global batch | 500 / 128 | 500 / 128 |
| Learning rate | ||
| Train / inference context | 8,192 / 2,048 tokens | 8,192 / 2,048 tokens |
| Attention LoRA | Rank 8, scale 16, dropout .05 | Rank 8, scale 16, dropout .05 |
| Trainable parameters | 4,987,137 | 1,081,344 |
| Game | Reference events | Predicted events | ||
|---|---|---|---|---|
| Brotato | .9932 | .9924 | 1,535 | 55 |
| GTA V | .9924 | .9923 | 1,794 | 60 |
| Apex Legends | .4019 | 349 | 85 |
| Model | Loss | (%) | Kbd F1 | Mouse F1 | Events | |
|---|---|---|---|---|---|---|
| H-Base | – | 99.5 | .4942 | .2121 | .7662/.7423 | 3,427 |
| T-Real | .24934 | 0.0 | .0017 | .0000 | .2823/.1977 | 1,735 |
| T-Unknown | .26507 | 0.0 | .0000 | .0000 | Undef. | 0 |
| T-Neutral | .77015 | 53.7 | .3572 | .0448 | .0173/.1159 | 43 |
| Scale | (%) | Kbd F1 | Mouse F1 | Events | ||
|---|---|---|---|---|---|---|
| 0.00 | 99.65 | .4891 | .1875 | .7918 | .7167 | 2,625 |
| 0.25 | 99.53 | .4907 | .1875 | .7860 | .7068 | 2,858 |
| 0.50 | 99.07 | .4893 | .2258 | .8026 | .7372 | 3,257 |
| 1.00 | 0.00 | .0017 | .0000 | .2823 | .1977 | 1,735 |
| Model | (%) | Kbd F1 | Mouse F1 | Events | ||
|---|---|---|---|---|---|---|
| H-Base | 99.53 | .4942 | .2121 | .7662 | .7423 | 3,427 |
| T-Real, gate=0 | 99.65 | .4891 | .1875 | .7918 | .7167 | 2,625 |
| No-Time | 98.39 | .4633 | .1690 | .6839 | .7114 | 4,465 |
Robot Learning
Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.
See all 340 papers