ViVa: A Video-Generative Value Model for Robot Reinforcement Learning

Jindi Lv; Hao Li; Jie Li; Fankun Kong; Yang Wang; Pengfei Yi; Yifei Nie; Xiaofeng Wang

cs.ROApr 9, 2026

ViVa: A Video-Generative Value Model for Robot Reinforcement Learning

Authors: Jindi Lv, Hao Li, Jie Li, Fankun Kong, Yang Wang, Pengfei Yi, Yifei Nie, Xiaofeng Wang,

Organizations: GigaAI · Sichuan University · Tsinghua University

Abstract

Vision-language-action (VLA) models have advanced robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback. Reinforcement learning addresses this via value functions, which assess task progress and guide policy improvement. However, existing value models built on vision-language models (VLMs) struggle to capture temporal dynamics and physical interactions, undermining reliable value estimation in long-horizon tasks. In this paper, we propose ViVa, a video-generative value model that repurposes a pretrained video generator to jointly predict future proprioception and a scalar value. By grounding value estimation in anticipated embodiment dynamics, ViVa leverages spatiotemporal priors to intrinsically couple value with foresight beyond static snapshots. ViVa achieves state-of-the-art results in metric-based evaluation across three tasks, producing reliable value signals that accurately track task progress and detect execution errors. Integrated into RECAP, it achieves an average success rate of 80%, highlighting the promise of video-generative models for value estimation.

Figures & tables

Figure 2: Overall architecture of ViVa. Left: Proprioception and the scalar value are embedded into latent frames via repeat-padding and broadcast. Right: The latent sequence comprises clean conditioning frames (blank token, proprioception, multi-view images) and noisy target frames (future proprioception and value). The diffusion Transformer denoises the targets conditioned on the clean prefix, jointly predicting future embodiment state and value.

Figure 3: Illustration of the three real-world tasks. For each task, we show the initial state (left), key intermediate stages (middle), and the final successful state (right).

Figure 4: Error type distribution for shirt folding, box packaging, and paper roll organization. Each pie chart shows the relative frequency of failure modes, based on 50 held-out episodes per task.

	Shirt		Box		Paper
Method	Pearson	Spearman	Pearson	Spearman	Pearson	Spearman
Sparse-sampled
GVL	0.276	0.263	-0.081	-0.114	-0.099	-0.096
TopReward	0.118	0.090	0.503	0.499	0.318	0.241
Dense-supervised
$\pi^{*}_{0.6}$	0.740	0.720	0.946	0.945	0.926	0.936

Table 1: Comparison with robotics SOTA value models. Sparse - sampled methods show low or negative correlation. Dense - supervised approaches perform much better, and ViVa establishes new SOTA on two of three tasks.

Figure 5: Detection of subtle manipulation errors by ViVa. The model produces clear value drops at failure moments, confirming sensitivity to fine-grained execution errors.

Table 6

Figure 6: Value trajectory visualization across all tasks. Trajectories for

\pi^{*}_{0.6}

and ViVa are smoothed by 20%. ViVa tracks task progress with clear responses at key events and maintains sensitivity on the unseen task, demonstrating stronger generalization.

Method	Pearson	Spearman	MS	MDR
$\pi^{*}_{0.6}$	0.740	0.721	0.016	0.397
ViVa	0.819	0.799	0.037	0.643

Table 4: Object generalization on the out-of-domain pants folding task. ViVa consistently outperforms

\pi^{*}_{0.6}

across all evaluation metrics.

Table 9

Figure 7: Rollout-wise performance evolution across three manipulation tasks. Both methods improve over successive rounds, with RECAP (ViVa) achieving the best overall performance.

Table 11

Method	Pretrained	MS	ES	MDR	ErrDR
ViVa w/o pretrain	✗	0.009	0.023	0.523	0.524
ViVa	✓	0.034	0.026	0.733	0.612

Table 9: Ablation on pretrained weights for box packaging. Removing pretrained weights degrades performance across all metrics, confirming the importance of spatiotemporal priors.

Table 13

Figure 8: Loss weight analysis for balancing value and future proprioception predictions. Setting

\lambda_{\text{prop}}=0.5

achieves the best trade-off and is adopted across all experiments.

Figure 9: Hyperparameter analysis on the prediction horizon

K

for future proprioception prediction. Setting

K=50

achieves the best trade-off across tasks and is adopted for all experiments.

Method	Backbone	Training cost (GPU $\cdot$ days)	Inference cost (s / frame)
$\pi^{*}_{0.6}$	VLM	6	0.32
ViVa w/o future-state	VG	3	0.11
ViVa	VG	4	0.18

Table 12: Computational cost comparison. Training time (GPU

\cdot

days) and inference time (s) are reported. ViVa achieves faster training and inference than the VLM-based baseline.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 299 papers

World Value Models for Robotic Manipulation
Jun 23, 2026Zhihao Wang, Jianxiong Li, Yu Cui +4Robot Policy Learning World Models for Robotics
ByteDance Seed · Tsinghua University · Peking University
RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
Sun Yat-sen University · South China University of Technology · Peng Cheng Laboratory +3
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Apr 13, 2026Liaoyuan Fan, Zetian Xu, Chen Cao +3World Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China

Jun 23, 2026J/K move · Enter open · S save

World Value Models for Robotic Manipulation

Zhihao Wang, Jianxiong Li, Yu Cui, Yuan Gao+3

ByteDance Seed · Tsinghua University · Peking University

Robot Policy Learning World Models for Robotics Robot Manipulation Benchmarks Robot Learning

Open paper AskPDF arXiv

Generalist value models play a pivotal role in scaling robotic policy learning from large-scale, mixed-quality data. Mathematically, accurate value estimation demands deep temporal understanding, requiring models to both ground the current belief using historical context and plan over future outcomes. However, most existing robotic value models are built on Vision-Language Model (VLM) backbones that are pretrained primarily on static or temporally sparse visual observations, lacking the requisite temporal modeling capabilities for value estimation. Unlike VLMs, world models naturally excel at temporal modeling and future planning, making them ideal foundations for learning generalizable value functions. Driven by this insight, we marry world models with value estimation to construct a new generalist robotic value model, World Value Model (WVM), that offers accurate task progressions to assess data quality. On standard benchmarks, WVM delivers state-of-the-art (SOTA) Value-Order Correlation (VOC) results. Complementing standard evaluation suites that contains only expert data, we further introduce Suboptimal-Value-Bench, a multi-embodiment benchmark consisting of 800 suboptimal trajectories with high-fidelity, human-labeled frame annotations. Our evaluations show that WVM maintains its SOTA performance on Suboptimal-Value-Bench, establishing its robustness in handling both expert and suboptimal data. When deployed for policy learning, WVM improves manipulation performance across various policy extraction approaches in both simulated and real-world deployment, providing robust guidance for learning from mixed-quality data.

Abstract

Robotic RL

Explore similar work

World Value Models for Robotic Manipulation

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps