VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Zhaochong An; Orest Kupyn; Théo Uscidda; Andrea Colaco; Karan Ahuja; Serge Belongie; Mar Gonzalez-Franco; Marta Tintore Gazulla

cs.CVMar 27, 2026

VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Authors: Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla

Organizations: Google · University of Copenhagen · University of Oxford · CREST-ENSAE, Institut Polytechnique de Paris

Abstract

Large-scale video diffusion models achieve impressive visual quality, yet often fail to preserve geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve the pretrained capacity while improving geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware video post-training. VGGRPO introduces a Latent Geometry Model (LGM) that stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view geometric coherence. Experiments on both static and dynamic benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent video generation.

Figures & tables

Figure 1: World-consistent Video Generations with VGGRPO. We compare the baseline video diffusion model ( left , orange) with the VGGRPO-aligned model ( right , green). Each example depicts a challenging dynamic scene; we visualize representative keyframes from the generated video and reconstructed scene geometry from the inferred 4D scene representation. VGGRPO produces markedly more coherent scene structure and smoother camera motion over time, reducing geometric drift and structural artifacts in challenging dynamic settings.

Figure 2: Method Overview. (a) Latent Geometry Model. We connect latents from the diffusion VAE encoder to a geometry foundation model via a lightweight connector, yielding a Latent Geometry Model that predicts 4D scene geometry directly from video latents. (b) VGGRPO training. We perform latent-space GRPO using two complementary rewards, camera motion smoothness and geometry reprojection consistency, computed entirely in latent space with the latent geometry model. Together, these components align the video diffusion model toward 4D world-consistent generation on both static and dynamic scenes.

Method	Static			Dynamic		Sub. Cons. $\uparrow$	Bg. Cons. $\uparrow$	Aes. Qual. $\uparrow$	Img. Qual. $\uparrow$	Mot. Smooth. $\uparrow$	Dyn. Deg. $\uparrow$
	VQ $\uparrow$	MQ $\uparrow$	Epi. $\downarrow$	VQ $\uparrow$	MQ $\uparrow$
Base Model: Wan2.1-1B
Base	-	-	0.133	-	-	0.7941	0.8930	0.5233	0.6178	0.9552	0.9231
SFT	45.26	46.84	0.137	40.00	39.00	0.8032	0.8896	0.5472	0.6256	0.9646	0.8795
Epipolar-DPO	54.21	55.79	0.098	45.50	43.00	0.8125	0.8916	0.5578	0.6461	0.9671	0.8816
VideoGPA	53.68	56.32	0.105	42.50	41.00	0.8068	0.8931	0.5562	0.6507	0.9650	0.8734

Table 1: Quantitative Comparison. The best and runner-up results are shown in bold and underlined , respectively. In both static- and dynamic-scene benchmarks, our method consistently improves geometric consistency and overall video quality over baselines across different base models, demonstrating the effectiveness of latent-space geometry-aware post-training.

Figure 3: Qualitative Comparison on Static and Dynamic Scenes. We show the first, middle, and last frames of video generations for a static-scene prompt ( left ) and a dynamic-scene prompt ( right ), with a representative segment of each prompt shown at the top. All baselines exhibit inconsistent artifacts, including geometric drift, temporal flicker, and unstable camera motion. In contrast, VGGRPO produces more coherent scene structure with smoother camera trajectories across frames.

Table 2: Additional Studies. (a) Impact of geometry foundation models: VGGRPO generalizes across latent geometry models constructed from different geometry foundation models. (b) Impact of reward components:

r_{\mathrm{motion}}

and

r_{\mathrm{geo}}

provide complementary gains, and their combination yields the best overall performance. (c) Test-time reward guidance: Differentiable latent-space guidance through the latent geometry model improves geometric consistency at test time with modest runtime overhead and no training. (d) Generalization: Beyond improving world consistency, VGGRPO transfers to standard VBench captions, indicating improved generation quality and demonstrating robustness in diverse scenarios. (e) Efficiency study: Latent-space reward computation reduces both runtime and peak GPU memory compared to RGB-based rewarding. VQ/MQ denote VideoReward win rates for Visual and Motion Quality, Epi. denotes the Sampson epipolar error on static scenes. Time is measured in seconds, and Mem denotes peak GPU memory (GB). Best results are in bold .

Figure 4: Reward Components Ablation. The reconstructed scene visualizes the estimated camera trajectory (red curve), with the first and last frames shown below each reconstruction. Compared to the Baseline, optimizing the motion reward $r_{\mathrm{motion}}$ stabilizes camera motion, but geometric artifacts persist (green circle). Adding the reprojection consistency reward ( $r_{\mathrm{motion}}{+}r_{\mathrm{geo}}$ ) further improves scene geometry while preserving smooth camera motion, proving the complementarity of both components.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Analysis of the Latent Geometry Model. We compare the latent geometry model with the original RGB-based geometry model on 50 RealEstate10K test sequences under controlled perturbations applied in the video latent space. The top row reports camera pose estimation performance as the perturbation scale $\alpha$ increases, measured by relative rotation accuracy ( left , Racc@5), area under the accuracy curve ( middle , AUC@5), and relative translation accuracy ( right , Tacc@5). Our latent geometry model maintains stable performance across all noise levels, whereas the RGB-based geometry model degrades substantially as the perturbation grows. The bottom row shows a decoded RGB frame from the perturbed latents at different values of $\alpha$ . Even when perturbations produce only barely perceptible visual changes in RGB space, the RGB-based geometry model already degrades, reflecting the distribution gap when applied to generated content rather than real images. Our latent geometry model, trained directly on generated latents, avoids this gap and remains robust.

Model	Sub. Cons. $\uparrow$	Bg. Cons. $\uparrow$	Aes. Qual. $\uparrow$	Img. Qual. $\uparrow$	Mot. Smooth. $\uparrow$	Dyn. Deg. $\uparrow$
Baseline	0.9542	0.9528	0.5966	0.6733	0.9841	0.4237
SFT	0.9578	0.9556	0.5968	0.6304	0.9872	0.3988
Epipolar-DPO	0.9601	0.9564	0.5977	0.6353	0.9847	0.3325
VideoGPA	0.9605	0.9565	0.5973	0.6338	0.9835	0.3419
Ours	0.9644	0.9583	0.5991	0.6861	0.9895	0.3962

Appendix

Table 3: Generalization Performance on Standard VBench Captions. Compared with the base model, prior post-training methods provide only limited gains and even degrade certain metrics, such as Imaging Quality . In contrast, VGGRPO achieves the best overall performance on most metrics among the base model and prior post-training baselines. These results show that VGGRPO transfers effectively to the standard VBench caption set, improving generation quality while maintaining robust generalization across diverse scenarios. The best and second-best results are highlighted in bold and underlined , respectively.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 364 papers

VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment
Mar 17, 2026Tengjiao Yin, Jinglei Shi, Heng Guo +1Diffusion Model Alignment Multi-View Consistency
VCIP & TMCC & DISSec, College of Computer Science, Nankai University · Beijing University of Posts and Telecommunications · LIX, École Polytechnique, IP Paris
GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation
May 18, 2026Jan Ackermann, Shengqu Cai, Boyang Deng +3Physical Consistency in Video Generation Video Diffusion Models
Stanford University · Google DeepMind
GeoFix: Geometry Fixing for 4D Consistent Generation
Oct 9, 2026Yeobin Hong, Hyojun Go, Dohun Lee +7Geometric Consistency in Video Generation 4D Scene Generation
KAIST · ETH Zürich · Google +1

Mar 17, 2026J/K move · Enter open · S save

VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment

Tengjiao Yin, Jinglei Shi, Heng Guo, Xi Wang

VCIP & TMCC & DISSec, College of Computer Science, Nankai University · Beijing University of Posts and Telecommunications · LIX, École Polytechnique, IP Paris

Diffusion Model Alignment Multi-View Consistency Physical Consistency in Video Generation Video Diffusion Models Video Generation Geometric Consistency in Video Generation

Open paper AskPDF arXiv

Video diffusion models lack explicit geometric supervision during training, leading to inconsistency artifacts such as object deformation, spatial drift, and depth violations in generated videos. To address this limitation, we propose a geometry-based reward model that leverages pretrained geometric foundation models to evaluate multi-view consistency through cross-frame reprojection error. Unlike previous geometric metrics that measure inconsistency in pixel space, where pixel intensity may introduce additional noise, our approach conducts error computation in a pointwise fashion, yielding a more physically grounded and robust error metric. Furthermore, we introduce a geometry-aware sampling strategy that filters out low-texture and non-semantic regions, focusing evaluation on geometrically meaningful areas with reliable correspondences to improve robustness. We apply this reward model to align video diffusion models through two complementary pathways: post-training of a bidirectional model via SFT or Reinforcement Learning and inference-time optimization of a Causal Video Model (e.g., Streaming video generator) via test-time scaling with our reward as a path verifier. Experimental results validate the effectiveness of our design, demonstrating that our geometry-based reward provides superior robustness compared to other variants. By enabling efficient inference-time scaling, our method offers a practical solution for enhancing open-source video models without requiring extensive computational resources for retraining.

Abstract

Video Generation

Explore similar work

VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment

GeoFlow: Enforcing Implicit Geometric Consistency in Video Generation

GeoFix: Geometry Fixing for 4D Consistent Generation