VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models

Zirui Ge; Pengxiang Ding; Baohua Yin; Qishen Wang; Zhiyong Xie; Yemin Wang; Jinbo Wang; Hengtao Li

cs.ROMar 19, 2026

VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models

Authors: Zirui Ge, Pengxiang Ding, Baohua Yin, Qishen Wang, Zhiyong Xie, Yemin Wang, Jinbo Wang, Hengtao Li,

Organizations: Zhejiang University · OpenHelix Robotics · Westlake University · University of Sussex ·

Abstract

Video action models are an appealing foundation for Vision--Language--Action systems because they can learn visual dynamics from large-scale video data and transfer this knowledge to downstream robot control. Yet current diffusion-based video predictors are trained with likelihood-surrogate objectives, which encourage globally plausible predictions without explicitly optimizing the precision-critical visual dynamics needed for manipulation. This objective mismatch often leads to subtle errors in object pose, spatial relations, and contact timing that can be amplified by downstream policies. We propose VAMPO, a post-training framework that directly improves visual dynamics in video action models through policy optimization. Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising policy with rewards defined over expert visual dynamics in latent space. To make this optimization practical, we introduce an Euler Hybrid sampler that injects stochasticity only at the first denoising step, enabling tractable low-variance policy-gradient estimation while preserving the coherence of the remaining denoising trajectory. We further combine this design with GRPO and a verifiable non-adversarial reward. Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization. The homepage is https://vampo-robot.github.io/VAMPO/.

Figures & tables

Figure 1: Overall of VAMPO . Our post-training framework introduces reinforcement learning from verified rewards in place of the surrogate objective in video action models, enabling direct optimization of task-specific goals in training video prediction model (VPM). This approach improves the accuracy of VPM’s predictive visual representations, leading to enhanced action generation and task performance. Notably, our method demonstrates significant improvements not only in simulated environments but also in real-world scenarios, showcasing its robustness and versatility across diverse settings.

Figure 2: Overview of the VAMPO training paradigm. In the pretraining stage, the video prediction model (VPM) and action generation model (AGM) are trained on expert demonstrations. In the policy optimization stage, the VPM generates future latents via a hybrid denoising process, using SDE-style stochasticity only at the first step and ODE-based denoising for the remaining steps. Verified rewards are computed by comparing predicted latents with expert latents, and GRPO is used to optimize the VPM toward more precise, control-relevant visual dynamics for downstream action generation.

Figure 3: Evaluation on Visual Dynamics. The figure reports the L1 evaluation between predicted latents and ground-truth latents over training steps, and VAMPO exhibits improved alignment with expert dynamics, leading to hallucination suppression, planning correction, and action rectification.

CALVIN ABC $\to$ D		Task completed in a row $\uparrow$					Avg. Len $\uparrow$	Avg. ER $\uparrow$	Avg. ERR $\uparrow$
CALVIN ABC $\to$ D		1	2	3	4	5
	Base Policy	96.0	91.3	86.4	80.4	74.7	4.28	29.28	0.0603
+	post-training VPM + original AGM	96.3	91.4	87.0	82.9	77.2	4.35	–	–
+	post-training VPM + post-training AGM	98.0	94.8	91.3	88.3	83.1	4.56	43.88	0.0814

Table 1: Effectiveness of improved visual dynamics on CALVIN ABC

\to

D. The table reports task completion in a row (1–5), average trajectory length (Avg. Len), and vision–action coupling metrics (Avg. ER, Avg. ERR). Post-training both VPM and AGM with VAMPO achieves the best performance; improvements over the base policy are in bold .

CALVIN ABC $\to$ D		Task completed in a row $\uparrow$					Avg. Len $\uparrow$
CALVIN ABC $\to$ D		1	2	3	4	5
VLM-based VLA	OpenVLA ( Kim et al., 2024 ) (CoRL)	91.3	77.8	62.0	52.1	43.5	3.27
	OpenVLA-OFT ( Kim et al., 2025 ) (RSS)	96.3	89.1	82.4	75.8	66.5	4.10
	$\pi_{0}$ ( Black et al., 2024 ) (RSS)	93.7	83.2	74.0	62.9	51.0	3.65
	$\pi_{0.5}$ ( Intelligence et al., 2025 ) (CoRL)	92.7	84.3	76.7	68.8	61.3	3.84
VPM-based VLA (pixel-level)	UniPi ( Du et al., 2023 ) (NeurIPS)	56.0	16.0	8.0	8.0	4.0	0.92

Table 2: Performance on the CALVIN ABC

\to

D benchmark. The table reports task completion in a row (1–5) and average trajectory length (Avg. Len) for VLM-based and VPM-based VLAs. VAMPO (Ours) achieves the best performance; our method’s row is highlighted.

Table 3: Performance on L-CALVIN (long-horizon). The table reports tasks completed in sequence (1–10) and average trajectory length (Avg. Len) under the ABC $\to$ D protocol. VAMPO (Ours) yields the best results across all task lengths; improvements are highlighted.

Figure 4: Qualitative visualization across multiple benchmarks. The figure presents improved action prediction and manipulation quality achieved by VAMPO on real-world platforms (Agibot Genie 01, Flexiv dual-arm robot, VidowX).

Figure 5: Real-world evaluation across multiple task benchmarks. The figure reports performance on three manipulation benchmarks—grasping in clutter, single-arm pick-and-place, and bimanual grasp-and-place—on the Agibot Genie 01 platform. VAMPO (Ours) achieves the best performance across all tasks.

Figure 6: Ablation studies on core components. The figure visualizes the effects of different design choices on CALVIN ABC

\to

D: (a) post-training steps (GRPO iterations), (b) optimization algorithm (DDPO vs. GRPO), (c) hybrid denoising strategy (5-step vs. 1-step SDE), and (d) reward type (pixel-level vs. latent-level). Our final configuration (1400-step GRPO, 1-step SDE, latent-space reward) consistently achieves the best performance.

Type	Name	Parameters
Prediction	Video length	16
Prediction	Action shape	$10\times 7$
TVP	Language shape	$20\times 512$
TVP	Image shape	$256\times 256$
Video Former	Token shape	$16\times 14\times 384$
	Input dim	1280

Table S1: Model Configuration Parameters. Detailed architectural specifications for our model, including tensor shapes for the Video Former and Diffusion Transformer components

Category	Parameter	Value
Optimization	Group size ( $G$ )	8
	PPO clipping parameter ( $\epsilon_{\text{clip}}$ )	0.2
	Batch size	8
Learning Rate	Fine-tuning	$1\times 10^{-4}$
Learning Rate	Post-training	$1\times 10^{-6}$
Reward Weights	$L_{1}$ reward	1.0

Table S2: Summary of hyperparameters and optimization settings.

Figure S1 : Real-world robot execution examples.

		Task completed in a row $\uparrow$					Avg. Len $\uparrow$
		1	2	3	4	5
	Base Policy	96.0	91.3	86.4	80.4	74.7	4.28
+	Pixel	96.8	91.9	88.2	83.6	78.0	4.39 (+ 0.08)
+	Latent (Ours)	98.0	94.8	91.3	88.3	83.1	4.56 (+ 0.28)

Table S3: Ablation on reward type: pixel-level vs. latent-level reward on CALVIN ABC

\to

D. Latent-space reward (Ours) achieves the best performance.

		Task completed in a row $\uparrow$					Avg. Len $\uparrow$
		1	2	3	4	5
	Base Policy	96.0	91.3	86.4	80.4	74.7	4.28
+	5-step SDE	95.9	91.9	88.1	82.3	76.4	4.34 (+ 0.06)
+	1-step SDE (Ours)	98.0	94.8	91.3	88.3	83.1	4.56 (+ 0.28)

Table S4: Ablation on hybrid denoising: SDE at 5 steps vs. 1 step. Applying SDE only at the first step (Ours) yields the best performance and mitigates reward hacking.

		Task completed in a row $\uparrow$					Avg. Len $\uparrow$
		1	2	3	4	5
	Base Policy	96.0	91.3	86.4	80.4	74.7	4.28
+	400	97.4	93.8	90.2	87.0	81.8	4.50 (+ 0.22)
+	600	95.6	90.6	85.6	81.8	77.2	4.31 (+ 0.03)
+	1000	96.5	92.8	89.0	84.9	78.2	4.42 (+ 0.14)
+	1400 (Ours)	98.0	94.8	91.3	88.3	83.1	4.56 (+ 0.28)

Table S5: Ablation on post-training steps (CALVIN ABC

\to

D). Success rates and average length for different GRPO step counts. Best performance is achieved at 1400 steps; best values are in bold .

		Task completed in a row $\uparrow$					Avg. Len $\uparrow$
		1	2	3	4	5
	Base Policy	96.0	91.3	86.4	80.4	74.7	4.28
+	DDPO	96.1	91.4	87.5	83.4	77.6	4.36 (+ 0.08)
+	GRPO (Ours)	98.0	94.8	91.3	88.3	83.1	4.56 (+ 0.28)

Table S6: Ablation on optimization algorithm. Comparison of DDPO and GRPO on CALVIN ABC

\to

D. GRPO (Ours) achieves the best performance.

Figure S2 : Visualization of One-step Direct Prediction. As illustrated, our model accurately captures the underlying visual dynamics of the robot’s motion during the prediction process. Although the textures and fine-grained details are not perfectly precise, the consistent trajectory and physical interactions demonstrate the model’s effectiveness

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 735 papers

Video Prediction Policy 2: Predict Better, Act Better
Oct 7, 2026Yanjiang Guo, Haodong Yan, Zhide Zhong +15Visuomotor Policy Learning World Action Models
Robotera · Tsinghua University · HKUST (GZ) +2
SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
Apr 3, 2026Peiyan Li, Yixiang Chen, Yuan Xu +13Multi-Task Robotic Manipulation Data-Efficient Robot Learning
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tsinghua University +4
Vid2WAM: Distilling Video Diffusion Priors into World Action Models
Aug 9, 2026Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7Cross-Modal Knowledge Distillation World Model Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute

Oct 7, 2026J/K move · Enter open · S save

Video Prediction Policy 2: Predict Better, Act Better

Yanjiang Guo, Haodong Yan, Zhide Zhong, Zhongru Zhang+14

Robotera · Tsinghua University · HKUST (GZ)+2

Visuomotor Policy Learning World Action Models Robot Learning from Video Zero-Shot Robotic Manipulation Video Prediction

Open paper AskPDF arXiv

World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform event-level video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.

Abstract

Video Diffusion Models

Explore similar work

Video Prediction Policy 2: Predict Better, Act Better

SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy

Vid2WAM: Distilling Video Diffusion Priors into World Action Models