Video action models (VAMs) have emerged as a promising paradigm for robot learning, owing to their powerful visual foresight for complex manipulation tasks. However, current VAMs, typically relying on either slow multi-step video generation or noisy one-step feature extraction, cannot simultaneously guarantee real-time inference and high-fidelity foresight. To address this limitation, we propose S-VAM, a shortcut video-action model that foresees coherent geometric and semantic representations via a single forward pass. Serving as a stable blueprint, these foreseen representations significantly simplify the action prediction. To enable this efficient shortcut, we introduce a novel self-distillation strategy that condenses structured generative priors of multi-step denoising into one-step inference. Specifically, vision foundation model (VFM) representations extracted from the diffusion model's own multi-step generated videos provide teacher targets. Lightweight decouplers, as students, learn to directly map noisy one-step features to these targets. Extensive experiments in simulation and the real world demonstrate that our S-VAM outperforms state-of-the-art methods, enabling efficient and precise manipulation in complex environments. Our project page is https://haodong-yan.github.io/S-VAM/
Figures & tables
Figure 1: Motivation and Overview of our Shortcut Video-Action Model. (a) Current video-action models struggle with a trade-off: one-step feature extraction is fast but yields noisy and entangled representations, whereas multi-step video generation predicts precise future states but is too slow for real-time control. (b) To address this, we propose a shortcut video-action model that foresees coherent geometric and semantic representations via a single forward pass. Specifically, we introduce a self-distillation strategy that extracts vision foundation model (VFM) representations (DPAv3 [ 25 ] and DINOv2 [ 28 ] ) from the diffusion model’s own multi-step generated videos to serve as teacher supervision exclusively during training (dashed path). By employing lightweight decouplers as students to map entangled one-step features to these geometry and semantics-oriented targets, our approach condenses structured generative priors of multi-step denoising into one-step inference, thereby enabling real-time and precise action prediction.
Figure 2: Architecture of our S-VAM. The core technical novelty lies in establishing a shortcut that bypasses the prohibitive latency of iterative video generation. Specifically, specialized decouplers disentangle highly entangled one-step diffusion features into coherent geometric and semantic foresight. This foresight is then aggregated with original features by a Uni-Perceiver, providing a holistic condition context for the downstream diffusion policy to predict precise robot action.
Figure 3: Qualitative comparison on CALVIN [ 27 ] . VPP [ 14 ] utilizes entangled one-step features, resulting in an erratic attention trajectory that explicitly contradicts the language instruction and leads to failed actuation. In contrast, our S-VAM foresees geometric and semantic representations. This decoupled future blueprint enables the action expert to anchor a coherent attention trajectory that perfectly aligns with the language instruction, thereby ensuring successful execution. Note: Geometric foresight is visualized as probe-based depth maps [ 19 , 41 ] ; one-step and semantic features are visualized via PCA computed globally across the entire sequence.
Category
Method
ith Task Success Rate
Avg. Len. ↑
1
2
3
4
5
Direct Action Learning Methods
OpenVLA [ 18 ]
91.3
77.8
62.0
52.1
43.5
3.27
CLOVER [ 7 ]
96.0
83.5
70.8
57.5
45.4
3.53
π0 [ 3 ]
93.7
83.2
74.0
62.9
51.0
3.65
Spatial Forcing [ 19 ]
93.6
85.8
78.4
72.0
64.6
3.94
SuSIE [ 4 ]
87.0
69.0
49.0
38.0
26.0
2.69
Table 1: Quantitative comparison on CALVIN [ 27 ] . We report stage-wise success rates (1-5) and average sequence length (Avg. Len.). Best and second-best results are highlighted accordingly.
Figure 4: Qualitative comparison on MetaWorld [ 45 ] . VPP [ 14 ] utilizes entangled one-step features, resulting in a diverging attention trajectory that completely misses the target “nut”. In contrast, our S-VAM foresees explicit geometric and semantic representations. This decoupled future blueprint enables the action expert to anchor a coherent attention trajectory that accurately guides to the instructed target, ensuring successful grasping. Note: Visualization settings follow Fig. 3 .
Category
Method
Easy
Middle
Hard
Average ↑
(28 tasks)
(11 tasks)
(11 tasks)
(50 tasks)
Direct Action Learning Methods
RT-1 [ 6 ]
0.605
0.042
0.015
0.346
Diffusion Policy [ 9 ]
0.442
0.062
0.095
0.279
Spatial Forcing [ 19 ]
0.737
0.436
0.451
0.609
SuSIE [ 4 ]
0.560
0.196
0.255
0.410
GR-1 [ 42 ]
0.725
0.327
0.451
0.574
Table 2: Quantitative comparison on Metaworld [ 45 ] . We report success rates on easy, middle, and hard tasks, along with the overall average. Best and second-best results are highlighted accordingly.
Method Variant
ith Task Success Rate
Avg. Len. ↑
1
2
3
4
5
w/o Geometric Distillation
94.1
87.1
79.3
73.5
66.5
4.01
w/o Semantic Distillation
94.0
87.1
80.4
73.2
64.1
3.99
w/o Self-Distillation
94.2
85.2
75.9
67.8
59.0
3.82
w/o Uni-Perceiver
94.0
84.6
74.7
65.1
53.8
3.72
w/o Original Diffusion Feature
95.3
86.6
77.6
70.9
62.5
3.93
Table 3: Ablation study on key components of our S-VAM.
Type
Representation Source
ith Task Success Rate
Avg. Len. ↑
1
2
3
4
5
Semantic Targets
CLIP [ 31 ]
94.1
82.0
72.5
65.0
58.0
3.72
SigLIP [ 47 ]
94.5
84.8
74.7
65.5
57.0
3.77
DINOv2 [ 28 ]
94.1
87.1
79.3
73.5
66.5
4.01
DINOv3 [ 35 ]
95.6
87.7
78.8
70.3
62.1
3.95
Geometric Targets
DPAv3 [ 25 ]
94.0
87.1
80.4
73.2
64.1
3.99
Table 4: Quantitative comparison of different VFM representations as teacher targets.
Figure 5: Multi-task real-world experiments. (a) We deploy our system on a dual-arm Cobot using only monocular front-camera observations. (b) Our S-VAM demonstrates a significant success rate improvement over VPP [ 14 ] on all tasks without compromising real-time control capabilities.
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University +4