SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy

Peiyan Li; Yixiang Chen; Yuan Xu; Jiabing Yang; Xiangnan Wu; Jun Guo; Nan Sun; Long Qian

cs.ROApr 3, 2026

SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy

Authors: Peiyan Li, Yixiang Chen, Yuan Xu, Jiabing Yang, Xiangnan Wu, Jun Guo, Nan Sun, Long Qian,

Organizations: New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University · Xi’an Jiaotong University ·

Abstract

Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies neglect one or both aspects. They often rely on 2D visual observations or backbones pretrained on static image--text pairs, which leads to high data requirements and limited comprehension of environment dynamics. To address this, we introduce SpatialVAM, the first 3D Video Action Model that simultaneously predict spatial-aware multi-view heatmap videos and RGB videos. Our key insight is that this design naturally injects 3D information into video foundation models while aligning the representation format between video pretraining and action finetuning. Extensive experiments demonstrate that SpatialVAM enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and no additional pretraining, SpatialVAM handles challenging long-horizon and contact-rich tasks, generalizes to out-of-distribution settings, and predicts realistic future videos. Evaluations on Meta-World (22% $\uparrow$ ), RoboCasa (15% $\uparrow$ ) and real-world robotic platforms (16% $\uparrow$ ) show that SpatialVAM consistently outperforms other video action models, vision language action models and 3D-based policies, establishing a new state-of-the-art in data-efficient multi-task manipulation.

Figures & tables

Figure 1: Overview. We introduce SpatialVAM, a spatial-aware multi-view video diffusion policy that jointly models spatial structure and temporal evolution. Compared to prior manipulation policies, our approach: (1) processes spatial-aware multi-view images rather than independent 2D views; (2) represents robot actions as multi-view heatmaps , aligning the action space with video pretraining representations; (3) leverages a video foundation model , instead of a vision–language backbone, to jointly model future RGB and heatmap sequences. As a result, SpatialVAM achieves state-of-the-art on Meta-World, RoboCasa and real-world benchmarks, outperforming vision language action models, video action models and 3D-based policies.

Figure 2: Overview of SpatialVAM’s pipeline . (a) Point clouds and the end-effector pose are projected into spatial-aware multi-view RGB images and heatmaps, which are encoded to jointly predict future multi-view RGB and heatmap videos via a video diffusion model. Predicted heatmaps are back-projected to recover 3D end-effector positions. (b) The multi-view video diffusion transformer augments a pretrained backbone with view-attention to enable cross-view interaction. (c) A lightweight action decoder predicts end-effector rotation and gripper states from the denoised video latents. Final action chunks combine the predicted positions, rotations, and gripper states.

Method	Meta-World Tasks							Avg. Succ. (%) $\uparrow$
	D-Open	D-Close	Btn	Btn-Top	Fct-Cls	Fct-Open	Handle
UniPi [ 26 ]	0/25	9/25	3/25	0/25	1/25	3/25	4/25	$11.4$
BC-Scratch [ 54 ]	6/25	9/25	9/25	3/25	5/25	5/25	9/25	$26.2$
BC-R3M [ 54 ]	1/25	15/25	9/25	1/25	6/25	17/25	13/25	$35.4$
DP [ 22 ]	12/25	12/25	10/25	5/25	6/25	15/25	6/25	$37.7$
AVDC [ 29 ]	18/25	23/25	15/25	6/25	14/25	6/25	21/25	$58.9$

Table 1: Success rates on seven Meta-World tasks under a low-data regime (5 demonstrations per task). Each entry reports the number of successful rollouts out of 25 trials. SpatialVAM achieves the highest average success rate, outperforming all baselines.

Panel of Figure 3 — Figure 3: RoboCasa results. 10 demos/task for training and 50 rollouts for evaluation.

Figure 4: Real-world experimental setup and tasks. We evaluate SpatialVAM on three manipulation tasks using a Franka Research 3 robot with three ZED2i cameras. We further assess generalization under variations in background, object height, lighting, and object category.

Method	Basic Tasks			Unseen Tasks				Avg. Succ. (%)
	Put Lion	Push-T	Scoop Tort.	Put-B	Put-H	Push-L	Scoop-C
DP3 [ 43 ]	0/10	0/10	0/10	0/10	0/10	0/10	0/10	0.00
$\pi_{0.5}$ [ 2 ]	1/10	0/10	0/10	0/10	0/10	0/10	0/10	1.40
UVA [ 32 ]	2/10	0/10	0/10	1/10	1/10	0/10	0/10	5.70
BridgeVLA [ 49 ]	9/10	0/10	4/10	8/10	7/10	0/10	1/10	41.42
SpatialVAM (Ours)	10/10	4/10	7/10	5/10	6/10	3/10	5/10	57.10

Table 2: Real-world manipulation results under limited demonstrations. Success rates over 10 trials per task on three basic and four unseen tasks. All methods use 10 expert trajectories.

Figure 5: Robustness and ablation on Meta-World. Left: Avg. success rate vs. inference denoising steps

N

; SpatialVAM stays strong even with

N{=}1

. Right: Avg. success rate vs. number of additionally predicted RGB views; predicting more views yields monotonic improvements.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Multi-View Video Diffusion Transformer	Rotation & Gripper Predictor
Optimizer	AdamW	AdamW
Learning rate	$1.0\times 10^{-4}$	$1.0\times 10^{-4}$
Epochs	100	100
Batch size	64	64
Weight decay	0	$1.0\times 10^{-5}$
Image resolution	$256\times 256$	$256\times 256$

Appendix

Table 4: Hyperparameters used for training and inference in SpatialVAM.

Setting	Basic Tasks			Unseen Tasks				Avg. Succ. (%)
	Put Lion	Push-T	Scoop Tort.	Put-B	Put-H	Push-L	Scoop-C
Three Cameras	10/10	4/10	7/10	5/10	6/10	3/10	5/10	57.1
One Camera Original Views	8/10	3/10	7/10	5/10	5/10	3/10	4/10	50.0
One Camera Adapted Views	10/10	5/10	8/10	5/10	7/10	3/10	4/10	60.0

Appendix

Table 5: Robustness to physical-camera coverage and virtual-view selection. Success rates over 10 trials per task on three basic and four unseen tasks.

Figure 7: Projected observations under different physical-camera and virtual-view configurations. The three rows show, from top to bottom: three physical cameras with the original virtual projection views, one physical camera with the original views, and one physical camera with adapted views. Although using one camera reduces point-cloud coverage, adapting the virtual views better exposes the workspace observable from the remaining camera.

Setting	Basic Tasks			Unseen Tasks				Avg. Succ. (%)
	Put Lion	Push-T	Scoop Tort.	Put-B	Put-H	Push-L	Scoop-C
Original Calibration	10/10	4/10	7/10	5/10	6/10	3/10	5/10	57.1
Perturbed Calibration Setting 1	8/10	2/10	6/10	6/10	5/10	4/10	5/10	51.4
Perturbed Calibration Setting 2	10/10	4/10	8/10	4/10	5/10	3/10	4/10	54.3

Appendix

Table 6: Robustness to camera calibration errors. Success rates over 10 trials per task under the original calibration and two perturbed-calibration settings.

Hyperparameter	RGB Loss Weight ( $\lambda$ )			Heatmap Standard Deviation ( $\sigma$ )
	0.1	0.5	0.9	1.5	2.5	3.5
Avg. Suc. (%)	92	89.1	89.1	89.1	89.7	86.9

Appendix

Table 7: Average Success Rate On Meta-World with different RGB Loss Weights and Heatmap Standard Deviations.

Figure 9: Visualization of the predicted RGB sequences and heatmap sequences for the Door-Open task in Meta-World. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.

Figure 10: Visualization of the predicted RGB sequences and heatmap sequences for the Push-T task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.

Figure 11: Visualization of the predicted RGB sequences and heatmap sequences for the Scoop Tortilla task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.

Figure 12: Visualization of the predicted RGB sequences and heatmap sequences for the Put Lion task. For each view, the first and third rows show predictions from SpatialVAM, while the second and fourth rows show the corresponding ground truth. The peak locations of both predicted and ground-truth heatmaps are overlaid on the predicted and ground-truth RGB images, respectively. The results show that (1) the predicted RGB sequences are visually realistic, and (2) the heatmap peaks closely follow the end effector in the RGB images, indicating strong consistency between the predicted heatmaps and RGB sequences.

Panel of Figure 13 — Figure 13: Visualization of video predictions under different denoising steps (Part I). Predicted RGB videos and heatmap videos under different denoising step settings. Lower denoising steps lead to visibly lower RGB video quality, while the predicted heatmaps remain relatively stable.

Panel of Figure 14 — Figure 14: Visualization of video predictions under different denoising steps (Part II). Predicted RGB videos and heatmap videos under different denoising step settings. Lower denoising steps lead to visibly lower RGB video quality, while the predicted heatmaps remain relatively stable.

Multi-Task Robotic Manipulation

Multi-task robotic manipulation studies how robots can learn reusable, reliable behaviors across diverse objects, tasks, and environments.

See all 51 papers

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
Jun 13, 2026Yuxin Jiang, Chang Yu, Yunuo Chen +4Long-Horizon Robotic Manipulation World Models for Robotics
University of California, Los Angeles · University of California, San Diego · University of Utah +1
VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models
Mar 19, 2026Zirui Ge, Pengxiang Ding, Baohua Yin +16Video Diffusion Models Video Prediction
Zhejiang University · OpenHelix Robotics · Westlake University +10
Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation
May 12, 2026Junjin Xiao, Dongyang Li, Yandan Yang +9Multi-View Learning Robotic Manipulation
Shenzhen University of Advanced Technology

Jun 13, 2026J/K move · Enter open · S save

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

Yuxin Jiang, Chang Yu, Yunuo Chen, Xiang Feng+3

University of California, Los Angeles · University of California, San Diego · University of Utah+1

Long-Horizon Robotic Manipulation World Models for Robotics Visuomotor Policy Learning Robotic Manipulation Memory-Augmented Agents

Open paper AskPDF arXiv

Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/

Abstract

Multi-Task Robotic Manipulation

Explore similar work

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models

Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation