Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics

Abhishek Pillai; Ekta Prashnani; Joohwan Kim; Iuri Frosio

cs.AISep 29, 2026

Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics

Authors: Abhishek Pillai, Ekta Prashnani, Joohwan Kim, Iuri Frosio

Organizations: NVIDIA

Abstract

Video games offer scalable environments for studying perception and control in embodied agents. Abundant online gameplay videos could supply demonstrations, but they rarely include player inputs for training. Inverse Dynamics Models (IDMs) have thus been proposed to infer inputs from frames. Large (up to 1B parameters) IDMs trained on $\sim$ 1K-2K gameplay hours demonstrate feasibility and cross-environment generalization at this scale, but researchers do not clarify what the key components are to recover individual actions and often report only aggregate accuracy that can mask rare-action failures. We study the problem in a data-constrained scenario to evaluate how spatial motion features, model architectures, and training objectives affect an IDM's outcome and we analyse our models on per-key and balanced metrics such as $F_1^{macro}$ . Our experiments on Trackmania highlight the importance of factors like the model architecture and motion flow extraction in preprocessing, while also showing the limits of evaluation through unbalanced metrics. The application of the same architecture and training recipe to Cyberpunk 2077 reveals uneven performance across game mechanics. Our per-action evaluation and failure analysis highlight ambiguities from camera motion, delayed effects and imbalanced key-press frequencies that call for explicit modeling of 3D scene structure, long-term state and the adoption of proper losses in future implementations.

Figures & tables

Panel of Figure 1 — Figure 1 : Sample sequences and key-press distributions for Trackmania (a) and Cyberpunk 2077 (b). Distributions are on a log scale: key usage is highly imbalanced.

Game	Training sequences	Validation sequences	Testing sequences
Trackmania	$30{,}000$ ( $\sim 25$ mins)	$20{,}294$ ( $\sim 17$ mins)	$27{,}984$ ( $\sim 23$ mins)
Cyberpunk 2077	$72{,}192$ ( $\sim 60$ mins)	$25{,}600$ ( $\sim 21$ mins)	$24{,}576$ ( $\sim 20$ mins)

Table 1 : Number of overlapping 5-frame sequences per dataset. Each split’s sequences belong to different recordings that are disjoint.

Model	Configuration							Acc.	$F_{1}$
Model	$T_{\mathrm{out}}$	Arch.	Res.	Flow	Initialization	Loss	Epochs	Acc.	micro	macro	W	A	S	D	Sp
CNN-512	$5$	CNN	$512^{2}$	N	none	soft-F1	100	.489	.499	.367	.902	.434	.075	.415	.008
ViT-E2E-512	$5$	ViT	$512^{2}$	N	none	soft-F1	100	.887	.817	.637	.907	.698	.567	.755	.256
HT-RGB-E2E	$1$	HT	$512^{2}$	N	none	soft-F1	50	.929	.880	.746	.921	.837	.648	.823	.502
HT-RGBF-E2E	$1$	HT	$512^{2}$	Y	none	soft-F1	50	.929	.878	.791	.920	.829	.655	.822	.728
HT-RGBF-PT-F1	$1$	HT	$512^{2}$	Y	contrastive	soft-F1	$20+25$	.938	.895	.793	.920	.869	.688	.870	.618

Table 2 : Accuracy Acc. and

F_{1}

metrics on the test datasets;

20+25

denotes 20 contrastive and 25 supervised epochs. All rows refer to Trackmania except the last one marked ∗ , which uses Cyberpunk 2077. Bold and underlined mark the highest and second-highest distinct Trackmania values in each metric column, respectively. For Cyberpunk 2077 we report for comparison only navigation keys that are in common with Trackmania, with Accuracy,

F_{1}^{\mathrm{macro}}

, and

F_{1}^{\mathrm{micro}}

over that set.

Panel of Figure 2 — Figure 2 : Correctly predicted Trackmania sequences for HT-RGBF-PT-F1 (attention in the lower row). Pred matches GT for the supervised frame $t$ . Better seen at 6x zoom.

Panel of Figure 3 — Figure 3 : Incorrectly predicted Trackmania sequences for HT-RGBF-PT-F1 (attention in the lower row). Reverse-right ( $S,D$ ) is predicted as forward-left ( $W,A$ ), indicating lack of 3D understanding, while airborne braking and steering ( $S,D$ ) is predicted as ( $W,A$ ) as it is visually ambiguous. Better seen at 6x zoom.

Panel of Figure 4 — Figure 4 : Correctly classified Cyberpunk 2077 sequences for CP-HT-RGBF-PT-F1 (attention in the lower row). Exact backward-motion match and partial right-movement match after being shot. Better seen at 6x zoom.

Panel of Figure 5 — Figure 5 : Incorrectly classified Cyberpunk 2077 sequences for CP-HT-RGBF-PT-F1 (attention in the lower row). Both examples invert backward motion; in the second a false left movement is also predicted. Better seen at 6x zoom.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Configuration	Purpose
CNN-128	$128^{2}$ , 15-channel RGB, soft-F1	resolution baseline
CNN-512	$512^{2}$ , otherwise CNN-128	resolution ablation
ViT-E2E-512	$512^{2}$ , RGB, $d/L/H=384/2/4$ , soft-F1	ViT baseline
ViT-DINO-512	ViT-E2E with DINOv3 features	spatial-pretraining ablation
HT-RGB-E2E	$512^{2}$ , 15-channel RGB, $d/L/H=256/4/8$ , soft-F1	HT baseline
HT-RGB-BCE	HT-RGB-E2E with BCE	loss ablation

Appendix

Table 3 : Experiment configurations for Tables 2 and 4 . E2E is end-to-end;

d/L/H

is transformer embedding dimension, number of blocks, and number of attention heads.

Model	Configuration							Acc.	$F_{1}$
Model	$T_{\mathrm{out}}$	Arch.	Res.	Flow	Initialization	Loss	Epochs	Acc.	micro	macro	W	A	S	D	Sp
CNN-128	$5$	CNN	$128^{2}$	N	none	soft-F1	100	.469	.492	.361	.902	.430	.074	.392	.009
CNN-512	$5$	CNN	$512^{2}$	N	none	soft-F1	100	.489	.499	.367	.902	.434	.075	.415	.008
ViT-E2E-512	$5$	ViT	$512^{2}$	N	none	soft-F1	100	.887	.817	.637	.907	.698	.567	.755	.256
ViT-DINO-512	$5$	ViT	$512^{2}$	N	DINOv3	soft-F1	100	.870	.790	.624	.898	.696	.452	.664	.411
HT-RGB-E2E	$1$	HT	$512^{2}$	N	none	soft-F1	50	.929	.880	.746	.921	.837	.648	.823	.502

Appendix

Table 4 : Accuracy Acc. and

F_{1}

metrics on the Trackmania test split (for the secondary ablations reported in Table 3 );

20+25

denotes 20 contrastive and 25 supervised epochs. Bold and underlined mark the highest and second-highest distinct values in each metric column, respectively.

Explore similar work

StableIDM: Stabilizing Inverse Dynamics Model against Manipulator Truncation via Spatio-Temporal Refinement
Apr 20, 2026Kerui Li, Zhe Jing, Xiaofeng Wang +6Inverse Dynamics Embodied Artificial Intelligence
1GigaAI · 3Beijing Institute of Technology · Institute of Automation, Chinese Academy of Sciences
KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding
Jul 22, 2026Zeyu Liu, Zhangzhe Zhu, Yang Zhang +3Articulated Kinematics World Models
Institute of Artificial Intelligence (TeleAI), China Telecom, China · National University of Singapore, Singapore · Fudan University, China +2
World Models as Group Actions
May 23, 2026Zijie Wang, Wei Zhang, Weiming Zhang +4Efficient World-Action Model Video World Models
Sun Yat-sen University · Shenzhen Loop Area Institute · Baidu Inc. +2

Apr 20, 2026, cs.ROJ/K move · Enter open · S save

StableIDM: Stabilizing Inverse Dynamics Model against Manipulator Truncation via Spatio-Temporal Refinement

Kerui Li, Zhe Jing, Xiaofeng Wang, Zheng Zhu+5

1GigaAI · 3Beijing Institute of Technology · Institute of Automation, Chinese Academy of Sciences

Inverse Dynamics Embodied Artificial Intelligence

Open paper AskPDF arXiv

Inverse Dynamics Models (IDMs) map visual observations to low-level action commands, serving as central components for data labeling and policy execution in embodied AI. However, their performance degrades severely under manipulator truncation, a common failure mode that makes state recovery ill-posed and leads to unstable control. We present StableIDM, a spatio-temporal framework that refines features from visual inputs to stabilize action predictions under such partial observability. StableIDM integrates three complementary components: (1) auxiliary robot-centric masking to suppress background clutter, (2) Directional Feature Aggregation (DFA) for geometry-aware spatial reasoning, which extracts anisotropic features along directions inferred from the visible arm and (3) Temporal Dynamics Refinement (TDR) to smooth and correct predictions via motion continuity. Extensive evaluations validate our approach: StableIDM improves strict action accuracy by 12.1% under severe truncation on the AgiBot benchmark, and increases average task success by 9.7% in real-robot replay. Moreover, it boosts end-to-end grasp success by 11.5% when decoding video-generated plans, and improves downstream VLA real-robot success by 17.6% when functioning as an automatic annotator. These results demonstrate that StableIDM provides a robust and scalable backbone for both policy execution and data generation in embodied artificial intelligence.

Abstract

Explore similar work

StableIDM: Stabilizing Inverse Dynamics Model against Manipulator Truncation via Spatio-Temporal Refinement

KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding

World Models as Group Actions