SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
Authors: Peiyan Li, Yixiang Chen, Yuan Xu, Jiabing Yang, Xiangnan Wu, Jun Guo, Nan Sun, Long Qian, Xinghang Li, Xin Xiao, Jing Liu, Nianfeng Liu, Tao Kong, Yan Huang, Liang Wang, Tieniu Tan
Organizations: New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University · Xi’an Jiaotong University · Wuhan University · FiveAges · Nanjing University
Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies neglect one or both aspects. They often rely on 2D visual observations or backbones pretrained on static image--text pairs, which leads to high data requirements and limited comprehension of environment dynamics. To address this, we introduce SpatialVAM, the first 3D Video Action Model that simultaneously predict spatial-aware multi-view heatmap videos and RGB videos. Our key insight is that this design naturally injects 3D information into video foundation models while aligning the representation format between video pretraining and action finetuning. Extensive experiments demonstrate that SpatialVAM enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and no additional pretraining, SpatialVAM handles challenging long-horizon and contact-rich tasks, generalizes to out-of-distribution settings, and predicts realistic future videos. Evaluations on Meta-World (22%↑), RoboCasa (15%↑) and real-world robotic platforms (16%↑) show that SpatialVAM consistently outperforms other video action models, vision language action models and 3D-based policies, establishing a new state-of-the-art in data-efficient multi-task manipulation.
Figures & tables
Figure 1: Overview. We introduce SpatialVAM, a spatial-aware multi-view video diffusion policy that jointly models spatial structure and temporal evolution. Compared to prior manipulation policies, our approach: (1) processes spatial-aware multi-view images rather than independent 2D views; (2) represents robot actions as multi-view heatmaps , aligning the action space with video pretraining representations; (3) leverages a video foundation model , instead of a vision–language backbone, to jointly model future RGB and heatmap sequences. As a result, SpatialVAM achieves state-of-the-art on Meta-World, RoboCasa and real-world benchmarks, outperforming vision language action models, video action models and 3D-based policies.
Figure 2: Overview of SpatialVAM’s pipeline . (a) Point clouds and the end-effector pose are projected into spatial-aware multi-view RGB images and heatmaps, which are encoded to jointly predict future multi-view RGB and heatmap videos via a video diffusion model. Predicted heatmaps are back-projected to recover 3D end-effector positions. (b) The multi-view video diffusion transformer augments a pretrained backbone with view-attention to enable cross-view interaction. (c) A lightweight action decoder predicts end-effector rotation and gripper states from the denoised video latents. Final action chunks combine the predicted positions, rotations, and gripper states.
Method
Meta-World Tasks
Avg. Succ. (%) ↑
D-Open
D-Close
Btn
Btn-Top
Fct-Cls
Fct-Open
Handle
UniPi [ 26 ]
0/25
9/25
3/25
0/25
1/25
3/25
4/25
11.4
BC-Scratch [ 54 ]
6/25
9/25
9/25
3/25
5/25
5/25
9/25
26.2
BC-R3M [ 54 ]
1/25
15/25
9/25
1/25
6/25
17/25
13/25
35.4
DP [ 22 ]
12/25
12/25
10/25
5/25
6/25
15/25
6/25
37.7
AVDC [ 29 ]
18/25
23/25
15/25
6/25
14/25
6/25
21/25
58.9
Table 1: Success rates on seven Meta-World tasks under a low-data regime (5 demonstrations per task). Each entry reports the number of successful rollouts out of 25 trials. SpatialVAM achieves the highest average success rate, outperforming all baselines.
Figure 3: RoboCasa results. 10 demos/task for training and 50 rollouts for evaluation.
Figure 4: Real-world experimental setup and tasks. We evaluate SpatialVAM on three manipulation tasks using a Franka Research 3 robot with three ZED2i cameras. We further assess generalization under variations in background, object height, lighting, and object category.
Method
Basic Tasks
Unseen Tasks
Avg. Succ. (%)
Put Lion
Push-T
Scoop Tort.
Put-B
Put-H
Push-L
Scoop-C
DP3 [ 43 ]
0/10
0/10
0/10
0/10
0/10
0/10
0/10
0.00
π0.5 [ 2 ]
1/10
0/10
0/10
0/10
0/10
0/10
0/10
1.40
UVA [ 32 ]
2/10
0/10
0/10
1/10
1/10
0/10
0/10
5.70
BridgeVLA [ 49 ]
9/10
0/10
4/10
8/10
7/10
0/10
1/10
41.42
SpatialVAM (Ours)
10/10
4/10
7/10
5/10
6/10
3/10
5/10
57.10
Table 2: Real-world manipulation results under limited demonstrations. Success rates over 10 trials per task on three basic and four unseen tasks. All methods use 10 expert trajectories.
Figure 5: Robustness and ablation on Meta-World. Left: Avg. success rate vs. inference denoising steps N ; SpatialVAM stays strong even with N=1 . Right: Avg. success rate vs. number of additionally predicted RGB views; predicting more views yields monotonic improvements.
Table 8
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Multi-View Video Diffusion Transformer
Rotation & Gripper Predictor
Optimizer
AdamW
AdamW
Learning rate
1.0×10−4
1.0×10−4
Epochs
100
100
Batch size
64
64
Weight decay
0
1.0×10−5
Image resolution
256×256
256×256
Appendix
Table 4: Hyperparameters used for training and inference in SpatialVAM.
Setting
Basic Tasks
Unseen Tasks
Avg. Succ. (%)
Put Lion
Push-T
Scoop Tort.
Put-B
Put-H
Push-L
Scoop-C
Three Cameras
10/10
4/10
7/10
5/10
6/10
3/10
5/10
57.1
One Camera Original Views
8/10
3/10
7/10
5/10
5/10
3/10
4/10
50.0
One Camera Adapted Views
10/10
5/10
8/10
5/10
7/10
3/10
4/10
60.0
Appendix
Table 5: Robustness to physical-camera coverage and virtual-view selection. Success rates over 10 trials per task on three basic and four unseen tasks.
Figure 7: Projected observations under different physical-camera and virtual-view configurations. The three rows show, from top to bottom: three physical cameras with the original virtual projection views, one physical camera with the original views, and one physical camera with adapted views. Although using one camera reduces point-cloud coverage, adapting the virtual views better exposes the workspace observable from the remaining camera.
Setting
Basic Tasks
Unseen Tasks
Avg. Succ. (%)
Put Lion
Push-T
Scoop Tort.
Put-B
Put-H
Push-L
Scoop-C
Original Calibration
10/10
4/10
7/10
5/10
6/10
3/10
5/10
57.1
Perturbed Calibration Setting 1
8/10
2/10
6/10
6/10
5/10
4/10
5/10
51.4
Perturbed Calibration Setting 2
10/10
4/10
8/10
4/10
5/10
3/10
4/10
54.3
Appendix
Table 6: Robustness to camera calibration errors. Success rates over 10 trials per task under the original calibration and two perturbed-calibration settings.
Hyperparameter
RGB Loss Weight ( λ )
Heatmap Standard Deviation ( σ )
0.1
0.5
0.9
1.5
2.5
3.5
Avg. Suc. (%)
92
89.1
89.1
89.1
89.7
86.9
Appendix
Table 7: Average Success Rate On Meta-World with different RGB Loss Weights and Heatmap Standard Deviations.
Figure 8: Visualization of the predicted RGB sequences and heatmap sequences for the Button-Press-Top task in Meta-World. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.
Figure 9: Visualization of the predicted RGB sequences and heatmap sequences for the Door-Open task in Meta-World. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.
Figure 10: Visualization of the predicted RGB sequences and heatmap sequences for the Push-T task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.
Figure 11: Visualization of the predicted RGB sequences and heatmap sequences for the Scoop Tortilla task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.
Figure 12: Visualization of the predicted RGB sequences and heatmap sequences for the Put Lion task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.
Figure 13: Visualization of video predictions under different denoising steps (Part I). Predicted RGB videos and heatmap videos under different denoising step settings. Lower denoising steps lead to visibly lower RGB video quality, while the predicted heatmaps remain relatively stable.
Figure 14: Visualization of video predictions under different denoising steps (Part II). Predicted RGB videos and heatmap videos under different denoising step settings. Lower denoising steps lead to visibly lower RGB video quality, while the predicted heatmaps remain relatively stable.
Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/
Yuxin Jiang, Chang Yu, Yunuo Chen +4
University of California, Los Angeles · University of California, San Diego · University of Utah +1
Video action models are an appealing foundation for Vision--Language--Action systems because they can learn visual dynamics from large-scale video data and transfer this knowledge to downstream robot control. Yet current diffusion-based video predictors are trained with likelihood-surrogate objectives, which encourage globally plausible predictions without explicitly optimizing the precision-critical visual dynamics needed for manipulation. This objective mismatch often leads to subtle errors in object pose, spatial relations, and contact timing that can be amplified by downstream policies. We propose VAMPO, a post-training framework that directly improves visual dynamics in video action models through policy optimization. Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising policy with rewards defined over expert visual dynamics in latent space. To make this optimization practical, we introduce an Euler Hybrid sampler that injects stochasticity only at the first denoising step, enabling tractable low-variance policy-gradient estimation while preserving the coherence of the remaining denoising trajectory. We further combine this design with GRPO and a verifiable non-adversarial reward. Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization. The homepage is https://vampo-robot.github.io/VAMPO/.
Zirui Ge, Pengxiang Ding, Baohua Yin +16
Zhejiang University · OpenHelix Robotics · Westlake University +10
This paper tackles spatial perception and manipulation challenges in Vision-Language-Action (VLA) models. To address depth ambiguity from monocular input, we leverage a pre-trained multi-view diffusion model to synthesize latent novel views and propose a Geometry-Guided Gated Transformer (G3T) that aligns multi-view features under 3D geometric guidance while adaptively filtering occlusion noise. To improve action learning efficiency, we introduce Action Manifold Learning (AML), which directly predicts actions on the valid action manifold, bypassing inefficient regression of unstructured targets like noise or velocity. Experiments on LIBERO, RoboTwin 2.0, and real-robot tasks show our method achieves superior success rate and robustness over SOTA baselines. Project page: https://junjxiao.github.io/Multi-view-VLA.github.io/.