Understanding 4D scenes is a core challenge in visual intelligence, requiring models to jointly capture scene geometry, appearance, and dynamics from partial observations. However, existing approaches are subdivided into feedforward models for direct geometric prediction, and generative models for visual synthesis. We introduce UGG, a unified framework for 4D scene understanding that jointly models RGB video and geometric representations such as depth and camera pose. By assigning different noise levels across modalities and time, UGG formulates visual generation and geometric prediction as conditional completion within a single generative model. This flexible formulation enables the model to learn from diverse, heterogeneous datasets with different subsets of geometric and visual annotations. Furthermore, our generative formulation enables inference-time search over multiple plausible scene hypotheses, using geometric evidence extracted from observed RGB frames to improve geometric prediction without ground truth geometry. We demonstrate UGG's effectiveness over non-unified baselines across camera-controlled video generation, depth estimation, and camera pose prediction, including in challenging sparse observation settings. Code and visualizations are available at https://ugg-model.github.io/
Figures & tables
Figure 1: A unified model for generation and prediction. We formulate visual generation and geometric prediction as conditional completion in a single joint model over RGB, depth, and camera pose. Given arbitrary partial observations, ranging from a text prompt to RGB sequences with geometric cues, our model jointly uses these modalities to infer a complete 4D scene representation, enabling both generation from sparse inputs and geometric prediction from observations.
Figure 2: Generative-predictive transformer. We model RGB, depth, and camera pose as spatially aligned tokens, with camera pose represented as dense ray maps. Partial observations are zero-noise context tokens, while missing modalities are noisy tokens to be denoised. A unified transformer processes all tokens with modality-specific projections and feed-forward layers together with shared multimodal self-attention, enabling information exchange across modalities. The decoded RGB, depth, and ray maps form a complete 4D scene representation for cross-modal generation, geometric prediction, and inference-time consistency search.
Figure 3: Conditional completion with noise as masking. We specify each task by the clean/noisy pattern over RGB, depth, camera-ray, and text tokens. Clean tokens provide observed context, while noisy tokens are denoised as missing targets, enabling generation and prediction with the same model architecture.
Figure 4: Qualitative comparison of camera-controlled generation. Given the first and last RGB frames and a target camera trajectory, we compare generated intermediate frames against ground truth and baseline video generation methods. UGG better preserves scene identity and dynamic content compared with the generic video completion baselines.
Method
RealEstate10K
SpatialVID
DL3DV
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
2-view
Aether
20.02
0.196
0.624
13.84
0.489
0.334
13.14
0.399
0.243
SEVA
25.25
0.081
0.838
15.44
0.344
0.466
17.61
0.211
0.552
UGG
25.73
0.061
0.849
16.99
0.288
0.535
16.61
0.239
0.495
1-view
Aether
15.79
0.335
0.469
11.81
0.632
0.283
11.96
0.506
0.240
SEVA
19.58
0.194
0.703
13.35
0.472
0.343
13.50
0.393
0.335
Table 1: Camera-controlled generation results. Generation is conditioned on the first RGB frame (1-view) or the first and last frames (2-view), together with the camera trajectory. We report PSNR ( ↑ ), LPIPS ( ↓ ), and SSIM ( ↑ ). Darker and lighter blue shading indicate the best and second-best results within each setting.
Figure 5: Qualitative comparison of estimated camera trajectories. The temporal evolution of each trajectory is encoded by a color gradient, from purple (start) to red (end). Ground-truth trajectory is shown in black. The leftmost column shows representative RGB frames from the input video. Compared with Geo4D and RayDiffusion, our predicted trajectories more closely follow the ground-truth trajectory, with better overall structure in challenging cases.
(a) Dense observations
Method
RealEstate10K
Sintel
TUM
ScanNetV2
ScanNet++
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
Geo4D
0.026
0.028
0.147
0.160
0.097
0.911
0.052
0.029
1.414
0.074
0.044
1.762
0.190
0.090
5.591
RayDiffusion
0.139
0.081
1.809
0.471
0.293
14.495
0.088
0.058
2.942
0.170
0.090
8.909
0.344
0.148
25.941
Gen3R
0.030
0.011
0.141
0.233
0.066
1.538
0.046
0.016
0.943
0.075
0.024
1.018
0.232
0.066
5.760
Aether
0.018
0.014
0.851
0.180
0.867
1.354
0.041
0.016
1.462
0.075
0.030
1.617
0.225
0.068
5.944
Table 2: Camera pose estimation results. Dense: ATE, RPE t , and RPE r . Sparse: tAE and RPE r , evaluated only on the two observed frames. Angular errors are in degrees; lower is better for all metrics. Darker and lighter blue shading mark the best and second-best distinct results in each column, including oracle results.
Figure 6: Depth prediction qualitative comparison. The white box marks the region shown in the zoom-in columns, ordered as GT, Ours, ChronoDepth, and Geo4D from left to right. Our predictions preserve sharper local structures and object boundaries in the highlighted regions.
Method
ScanNet++
7-Scenes
KITTI
Sintel
Bonn
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Dense
ChronoDepth
0.177
0.724
0.116
0.871
0.120
0.857
0.394
0.530
0.092
0.920
Geo4D
0.106
0.894
0.080
0.923
0.097
0.923
0.247
0.676
0.057
0.972
Gen3R
0.157
0.772
0.079
0.937
0.185
0.777
0.414
0.520
0.168
0.805
Aether
0.166
0.769
0.109
0.898
0.056
0.975
0.351
0.607
0.095
0.911
Marigold
0.175
0.730
0.094
0.910
0.098
0.910
0.388
0.517
0.082
0.933
Table 3: Depth prediction results. We report Abs Rel ( ↓ ) and δ<1.25 ( ↑ ) on ScanNet++, 7-Scenes, KITTI, Sintel, and Bonn in dense and sparse settings. Darker and lighter blue shading indicate the best and second-best distinct results within each setting, including oracle results.
Figure 7: Test-time search with increasing sample budget. We sample up to 16 candidate completions and select one using the image-based depth or pose verifier. The dashed line shows the single-sample result, and the solid curve shows the selected result as L increases. Search improves several pose and depth settings, showing that the generative model produces useful candidate diversity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
ScanNetV2
ScanNet++
whist
ATE
RPE r
ATE
RPE r
0
0.0796
2.46
0.2170
5.77
1
0.0741
2.83
0.2089
6.09
2
0.0740
3.24
0.2091
6.45
4
0.0765
4.22
0.2141
7.70
Appendix
Table 5: History-guidance weight. We report ATE ( ↓ ) and RPE r ( ↓ , degrees) on ScanNetV2 and ScanNet++. Shading marks the best result in each column.
Asset
Use in this work
License / access terms
Training datasets
DL3DV-10K [ 29 ]
RGB and pose; generation evaluation
CC BY-NC 4.0 ; dataset terms; research and education only
RealEstate10K [ 74 ]
RGB and pose; generation and pose evaluation
CC BY 4.0
TartanAir / TartanAir V2 [ 54 ]
Synthetic RGB, depth, and pose
CC BY 4.0
Spring [ 33 ]
Synthetic RGB, depth, and pose
CC BY 4.0
BEDLAM [ 5 ]
Human-centric RGB, depth, and pose
BEDLAM license; noncommercial research and education
Appendix
Table 6: Datasets and pretrained assets. We summarize the role of each asset and its license or access terms. License names are abbreviated, original agreements and third-party restrictions apply.
We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. Together, these results demonstrate the potential of video generative pretraining for unified 4D perception. See https://gen4p.github.io for more visual results.
Yitong Xia, Zeren Jiang, Chuanxia Zheng
PVG, Nanyang Technological University · VGG, University of Oxford
Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization. To address these, we present MoGe4D (Motion and Geometry-Aware image-to-4D Synthesis), a geometry-conditioned framework for single-image 4D synthesis that models a scene as dense 4D point trajectories. Instead of treating geometry and dynamics as two disconnected stages, our method starts from an initial geometric prior inferred from the input image and predicts future time-varying trajectories in a diffusion process, improving spatiotemporal coherence while preserving structural stability. To support this task, we first introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense 4D point trajectories, addressing the scarcity of high-quality training data for scene-level 4D generation. Built on this, our diffusion-based 4D Scene Trajectory Generator (4D-STraG) predicts geometry-consistent and motion-plausible trajectory fields conditioned on the input image, with a depth-guided motion normalization strategy to reduce scale ambiguity and a Motion Perception Module (MPM) to inject motion-aware priors. We further propose a 4D View Synthesis Module (4D-ViSM) to render the generated 4D representation into videos under arbitrary camera trajectories. Experiments show that MoGe4D produces high-quality 4D scenes with strong temporal coherence, favorable geometry-aware consistency, and compelling novel-view synthesis from a single image. Code: https://github.com/Zhangyr2022/MoGe4D.
Yanran Zhang, Ziyi Wang, Wenzhao Zheng +3
Department of Automation, Tsinghua University, China · GigaAI
Recent 4D generation methods complete scene-level missing information using generative models and reconstruct the scene into radiance-based representations. However, these pipelines often present geometric inconsistencies in the generated content, and the radiance-based reconstruction requires expensive optimization. Furthermore, radiance-based representations often absorb these geometric inconsistencies into their view-dependent nature, failing to enforce the grounded geometric consistency. To address these issues, we propose Geometric 4D Stitching, an efficient framework that explicitly identifies missing geometric regions and complements them with geometrically grounded 4D stitches. As a result, our method constructs 4D scene representations in under 10 minutes on a single NVIDIA RTX 5090 GPU per one-step scene expansion, while improving geometric consistency. Moreover, we demonstrate that our explicit 4D stitching supports interative expansion of 4D mesh as well as 4D scene editing.