We present a novel method for generating geometrically realistic and consistent orbital videos from a single image of an object. Existing video generation works mostly rely on pixel-wise attention to enforce view consistency across frames. However, such mechanism does not impose sufficient constraints for long-range extrapolation, e.g. rear-view synthesis, in which pixel correspondences to the input image are limited. Consequently, these works often fail to produce results with a plausible and coherent structure. To tackle this issue, we propose to leverage rich shape priors from a 3D foundational generative model as an auxiliary constraint, motivated by its capability of modeling realistic object shape distributions learned from large 3D asset corpora. Specifically, we prompt the video generation with two scales of latent features encoded by the 3D foundation model: (i) a denoised global latent vector as an overall structural guidance, and (ii) a set of latent images projected from volumetric features to provide view-dependent and fine-grained geometry details. In contrast to commonly used 2.5D representations such as depth or normal maps, these compact features can model complete object shapes, and help to improve inference efficiency by avoiding explicit mesh extraction. To achieve effective shape conditioning, we introduce a multi-scale 3D adapter to inject feature tokens to the base video model via cross-attention, which retains its capabilities from general video pretraining and enables a simple and model-agonistic fine-tuning process. Extensive experiments on multiple benchmarks show that our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods, and robustly generalizes to complex camera trajectories and in-the-wild images.
Figures & tables
Figure 2: Overview of the method. Given an input image I , we aim to generate a realistic and consistent orbital video V in a base video diffusion model V(⋅) , conditioned on shape priors from a pretrained 3D foundation model F(⋅) . Specifically, we forward the input image to the foundation model to generate ( i ) a denoised global latent vector p^ as overall structural guidance, and ( ii ) a set of latent images L^ for view-dependent and fine-grained geometry details. We then prompt the video generation with both features through a multi-scale 3D adapter via alternating cross-attention to produce the resulting video. We omit conditions of camera pose e and timestep t for simplicity.
Methods
Objaverse-XL [ 9 ]
GSO [ 10 ]
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP-S ↑
MEt3R ↓
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP-S ↑
MEt3R ↓
Wonder3D [ 25 ]
19.53
0.89
0.15
89.03
-
19.06
0.89
0.19
84.59
-
Era3D [ 21 ]
21.08
0.90
0.13
91.06
-
21.72
0.91
0.16
88.11
-
Ours (6 frames)
23.20
0.93
0.08
94.95
-
25.05
0.93
0.07
92.86
-
Trellis [ 46 ]
19.43
0.89
0.14
92.05
-
20.36
0.91
0.15
89.70
-
Hunyuan [ 55 ]
20.25
0.91
0.11
93.44
-
20.88
0.91
0.14
91.03
-
Table 1: Quantitative comparison with baseline methods. Best results are marked as bold . Our method achieves superior results than baseline methods [ 25 , 21 , 42 , 47 , 55 , 46 ] in all visual quality, shape realism and view consistency metrics on multiple benchmarks [ 9 , 10 ] .
Figure 3: Qualitative comparison with baselines. Compared to NVS [ 25 , 21 ] and video [ 42 , 47 ] generation works, our method produces more realistic and consistent results with less artifacts, e.g. distorted shapes and unnatural structures. Moreover, we generate more accurate object colors and lighting effects compared to rendered textured meshes from 3D generation methods [ 46 , 55 ] .
Figure 4: Results on in-the-wild images. Our method robustly generalizes to in-the-wild examples and outperforms baseline video generation methods [ 42 , 47 ] in both visual fidelity and shape realism.
Figure 5: Extension to dynamic orbits. Our method can follow complex camera trajectories with non-zero elevations, i.e. dynamic orbits defined by [ 42 ] to achieve accurate camera controls.
Figure 6: Effects of the 3D foundation priors. The shape priors from 3D foundation models (visualized after decoding and mesh extraction at the last column) help to provide complementary guidance for unobserved and occluded object parts.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP-S ↑
MEt3R ↓
w/o Priors
20.06
0.90
0.14
91.26
0.08
w/ Global
21.86
0.91
0.11
93.51
0.06
Global + Local
22.78
0.92
0.09
94.19
0.05
Feature Cat.
20.53
0.90
0.16
91.88
0.08
Frame Stack
19.27
0.89
0.18
89.96
0.08
CrossAttn
22.78
0.92
0.09
94.19
0.05
Table 2: Ablation studies. In top rows, we observe leveraging both scales of features achieves the best overall performance. In bottom rows, we show proposed cross-attention based adapter outperforms alternative feature conditioning approaches as it better preserves pretrained temporal priors.
Recent video foundation models demonstrate impressive visual synthesis but frequently suffer from geometric inconsistencies. While existing methods attempt to inject 3D priors via architectural modifications, they often incur high computational costs and limit scalability. We propose World-R1, a framework that aligns video generation with 3D constraints through reinforcement learning. To facilitate this alignment, we introduce a specialized pure text dataset tailored for world simulation. Utilizing Flow-GRPO, we optimize the model using feedback from pre-trained 3D foundation models and vision-language models to enforce structural coherence without altering the underlying architecture. We further employ a periodic decoupled training strategy to balance rigid geometric consistency with dynamic scene fluidity. Extensive evaluations reveal that our approach significantly enhances 3D consistency while preserving the original visual quality of the foundation model, effectively bridging the gap between video generation and scalable world simulation.
Weijie Wang, Xiaoxuan He, Youping Gu +9
Zhejiang University · Microsoft Research · Independent Researcher
Novel View Synthesis (NVS) aims to generate unseen views of a 3D object given a limited number of known views. Existing methods often struggle to synthesize plausible views for unobserved regions, particularly under single-view input, and still face challenges in maintaining geometry- and appearance-consistency. To address these issues, we propose OrbitNVS, which reformulates NVS as an orbit video generation task. Through tailored model design and training strategies, we adapt a pre-trained video generation model to the NVS task, leveraging its rich visual priors to achieve high-quality view synthesis. Specifically, we incorporate camera adapters into the video model to enable accurate camera control. To enhance two key properties of 3D objects, geometry and appearance, we design a normal map generation branch and use normal map features to guide the synthesis of the target views via attention mechanism, thereby improving geometric consistency. Moreover, we apply a pixel-space supervision to alleviate blurry appearance caused by spatial compression in the latent space. Extensive experiments show that OrbitNVS significantly outperforms previous methods on the GSO and OmniObject3D benchmarks, especially in the challenging single-view setting (\eg, +2.9 dB and +2.4 dB PSNR).
Jinglin Liang, Zijian Zhou, Rui Huang +2
South China University of Technology · Agentic Labs, XGTech · King’s College London +1
Generating coherent videos grounded in a unified underlying world remains a major challenge, particularly under large viewpoint changes, where existing methods often struggle to maintain stable representations of individual objects. To this end, we introduce Totem, an object-centric 3D memory framework for world-consistent video generation. Totem first represents each selected object as a persistent 3D memory aligned to a shared world frame, assigning it a dedicated state. Based on these memories, we design a dual-stream 3D conditioning architecture that combines object memories with scene geometry for target-view synthesis. Moreover, these object memories constrain the model's attention with 3D structural information, thereby preserving object surfaces and relative spatial layouts. Extensive experiments on world-generation and object-centric benchmarks demonstrate that Totem generates videos depicting a more object-consistent 3D world under challenging camera trajectories, while supporting versatile applications.
Zichen Wan, Tianyu Huang, Fan Li +3
Harbin Institute of Technology · Huawei Technologies Ltd.