GameRealm: Grounding Visual Learning in Interactive Games
Organizations: Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Zhejiang University
Abstract
Visual foundation models are central to physical intelligence, yet their development remains constrained by fragmented training datasets, inconsistent annotation quality, and limited multimodal supervision. To address these challenges, we introduce GameRealm, a large-scale, richly annotated dataset that grounds visual observations in synchronized, game-native supervision. GameRealm encompasses over 120 million RGB frames and 1,000 hours of gameplay footage across all views. Beyond its unprecedented scale, GameRealm provides rich multimodal supervision, including metric depth, camera parameters, synchronized audio, control inputs, and 3D point trajectories. Its diversity further extends to flexible view configurations, spanning single-view, stereo, multi-view, and panoramic recordings. To enable scalable acquisition across diverse game environments, we develop a unified data collection pipeline that combines human-operated interaction with agentic exploration. Extensive experiments across diverse visual learning tasks demonstrate that GameRealm consistently strengthens visual foundation models in both visual generation and geometric learning. These results establish GameRealm as a versatile and valuable data resource, unlocking new opportunities for advancing visual intelligence.
Figures & tables
| Dataset | # Frames | Data modality | View | ||||||||
| Depth | Camera | Text | Audio | Keypress | Tracking | Single-view | Stereo | Multi-view | Panoramic | ||
| MPI Sintel | 1K | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ | ✔ | ✘ | ✘ |
| FlyingThings++ | 28K | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ | ✔ | ✔ | ✘ | ✘ |
| TartanAir | 1,000K | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ | ✔ | ✘ | ✔ |
| BlendedMVS | 17K | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✘ | ✘ | ✔ | ✘ |
| HyperSim | 77K | ✔ | ✔ | ✘ | ✘ | ✘ | ✘ | ✔ | ✘ | ✘ | ✘ |
| VBench-I2V ( Huang et al., 2025 ) | iWorld-Bench ( Fang et al., 2026 ) | ||||||
| Method | Condition | MUSIQ | Rotation path ( ∘ ) | Endpoint rotation ( ∘ ) | MUSIQ | Rotation path ( ∘ ) | Endpoint rotation ( ∘ ) |
| Helios | Empty text | 0.6732 | 5.1107 | 1.4688 | 0.5504 | 4.1446 | 1.7275 |
| Helios | Stationary text | 0.6879 | 3.4294 | 0.6952 | 0.5701 | 0.3140 | 0.0735 |
| Helios ∗ | Zero action | 0.6949 | 3.3018 | 0.6421 | 0.5846 | 0.2835 | 0.0461 |
| Method | Condition | Trans. cos. | Rot. cos. | Dir. succ. (%) |
| Helios | Empty text | 0.0020 | 0.0005 | 1.52 |
| Helios | Directional text | 0.0015 | 0.0047 | 2.40 |
| Helios ∗ | Action | 0.5490 | 0.8685 | 76.59 |
| Condition | Translation Cosine | MUSIQ |
| Correct action | 0.7783 | 0.5706 |
| Zero action | -0.0015 | 0.5867 |
| Reversed action | -0.7997 | 0.5674 |
| Cross-view comm. | Inter-view ( ∘ ) | Joint ( ∘ ) | Joint |
| Disabled (B1) | 39.93 | 13.94 | 2.88 |
| Enabled (B2) | 19.80 | 11.43 | 2.22 |
| Aligned | +1-frame shift | ||
| Configuration | P50 | P95 | P50 |
| Multi-view | 0.482 | 1.835 | 23.417 |
| Stereo | 0.458 | 2.193 | 22.780 |
| Method | Middlebury-H | Middlebury-Q | ETH3D | KITTI 2015 | |||||||||
| BP-1 | BP-2 | BP-3 | BP-1 | BP-2 | BP-3 | BP-1 | BP-2 | BP-3 | BP-1 | BP-2 | BP-3 | D1 | |
| Fast-FoundationStereo | 4.80 | 2.20 | 1.60 | 4.51 | 2.12 | 1.57 | 1.22 | 0.62 | 0.50 | 19.62 | 5.78 | 3.43 | 3.25 |
| baseline | 4.76 | 2.22 | 1.63 | 4.10 | 2.00 | 1.55 | 1.18 | 0.61 | 0.49 | 19.14 | 5.58 | 3.26 | 3.11 |
| Fast-FoundationStereo ∗ | 4.66 | 2.03 | 1.42 | 4.21 | 1.93 | 1.45 | 1.09 | 0.53 | 0.42 | 19.09 | 5.50 | 3.18 | 3.03 |
| Camera pose | Video depth | Multi-view recon. | ||||||||||
| Sintel (traj.) | Re10K (pairwise) | Bonn | KITTI | 7-Scenes | ETH3D | |||||||
| ATE | RPE-trans | AUC@5 | AUC@15 | AbsRel | AbsRel | Acc | Comp | Acc | Comp | |||
| 0.0733 | 0.0390 | 62.66 | 79.93 | 0.0438 | 0.9753 | 0.0366 | 0.9853 | 0.0158 | 0.0220 | 0.1871 | 0.2104 | |
| baseline | 0.0766 | 0.0384 | 62.43 | 79.79 | 0.0482 | 0.9763 | 0.0363 | 0.9858 | 0.0166 | 0.0220 | 0.1929 | 0.2125 |
| 0.0500 | 0.0343 | 62.96 | 80.18 | 0.0479 | 0.9778 | 0.0357 | 0.9856 | 0.0162 | 0.0219 | 0.1922 | 0.2112 | |
| Interaction | Locomotion | ||||
| Category | Seq. | Frames | Category | Seq. | Frames |
| General | 40 | 8,495 | General | 30 | 7,066 |
| Hard | 15 | 3,904 | Long-horizon | 20 | 61,718 |
| Embodied | 15 | 6,123 | Transportation | 30 | 11,492 |
| Plant | 15 | 3,690 | Group | 10 | 1,818 |
| Subtotal | 85 | 22,212 | Subtotal | 90 | 82,094 |
| Locomotion Methods | General | Long-horizon | Transportation | Group | ||||||||||||||||
| Without GT depth and camera pose | ||||||||||||||||||||
| Track4World † | 12.0 ‡ | 16.5 ‡ | 41.4 ‡ | 61.0 ‡ | 79.9 ‡ | 0.5 ‡ | 1.9 ‡ | 14.1 ‡ | 30.1 ‡ | 74.9 ‡ | 1.0 ‡ | 2.5 ‡ | 30.9 ‡ | 55.9 ‡ | 70.6 ‡ | 11.5 ‡ | 20.4 ‡ | 40.9 ‡ | 62.8 ‡ | 76.6 ‡ |
| MoSE3 † | 17.4 | 24.0 | 26.6 | 41.6 | 79.6 | 6.8 | 15.4 | 10.6 | 28.3 | 65.6 | 4.5 | 9.7 | 15.9 | 33.2 | 66.9 | 15.3 | 27.3 | 24.0 | 40.7 | 74.4 |
| TrackEverything + † | 19.0 | 25.1 | 25.4 | 34.0 | 72.4 | 0.4 | 1.7 | 1.6 | 3.6 | 61.7 | 2.6 | 5.3 | 5.6 | 11.1 | 59.0 | 15.7 | 30.9 | 23.6 | 46.4 | 66.6 |
| TrackEverything + ∗† | 21.3 | 29.1 | 30.4 | 44.0 | 79.0 | 1.7 | 4.8 | 5.4 | 14.5 | 66.9 | 5.1 | 10.2 | 14.4 | 29.2 | 65.2 | 17.3 | 32.2 | 29.3 | 53.6 | 70.9 |
| Methods | SynthVerse-mAverage | Dynamic Replica | LSFOdyssey | Avg. | ||||||||||||||||
| TAPIP3D-camera | ||||||||||||||||||||
| TAPIP3D | 33.3 | 46.2 | 47.1 | 66.9 | 79.4 | 53.7 | 70.8 | 64.6 | 84.7 | 84.7 | 68.3 | 83.2 | 76.0 | 91.2 | 86.2 | 51.8 | 66.7 | 62.6 | 80.9 | 83.4 |
| TAPIP3D ∗ | 36.4 | 48.0 | 48.1 | 65.9 | 79.9 | 61.5 | 75.4 | 69.8 | 85.4 | 87.8 | 73.5 | 86.0 | 80.2 | 92.4 | 90.4 | 57.2 | 69.8 | 66.0 | 81.2 | 86.1 |
| TAPIP3D-world | ||||||||||||||||||||
| Methods | ADT | DriveTrack | PStudio | Avg. | ||||||||
| TAPIP3D-camera +MegaSaM | ||||||||||||
| TAPIP3D | 21.6 | 31.0 | 90.4 | 14.6 | 21.3 | 82.2 | 18.1 | 27.7 | 85.5 | 18.1 | 26.7 | 86.0 |
| TAPIP3D ∗ | 24.0 | 32.6 | 93.6 | 14.8 | 20.7 | 84.3 | 19.1 | 28.5 | 88.0 | 19.3 | 27.2 | 88.6 |
| TAPIP3D-world +MegaSaM | ||||||||||||