Understanding 4D scenes is a core challenge in visual intelligence, requiring models to jointly capture scene geometry, appearance, and dynamics from partial observations. However, existing approaches are subdivided into feedforward models for direct geometric prediction, and generative models for visual synthesis. We introduce UGG, a unified framework for 4D scene understanding that jointly models RGB video and geometric representations such as depth and camera pose. By assigning different noise levels across modalities and time, UGG formulates visual generation and geometric prediction as conditional completion within a single generative model. This flexible formulation enables the model to learn from diverse, heterogeneous datasets with different subsets of geometric and visual annotations. Furthermore, our generative formulation enables inference-time search over multiple plausible scene hypotheses, using geometric evidence extracted from observed RGB frames to improve geometric prediction without ground truth geometry. We demonstrate UGG's effectiveness over non-unified baselines across camera-controlled video generation, depth estimation, and camera pose prediction, including in challenging sparse observation settings. Code and visualizations are available at https://ugg-model.github.io/
Figures & tables
Figure 1: A unified model for generation and prediction. We formulate visual generation and geometric prediction as conditional completion in a single joint model over RGB, depth, and camera pose. Given arbitrary partial observations, ranging from a text prompt to RGB sequences with geometric cues, our model jointly uses these modalities to infer a complete 4D scene representation, enabling both generation from sparse inputs and geometric prediction from observations.
Figure 2: Generative-predictive transformer. We model RGB, depth, and camera pose as spatially aligned tokens, with camera pose represented as dense ray maps. Partial observations are zero-noise context tokens, while missing modalities are noisy tokens to be denoised. A unified transformer processes all tokens with modality-specific projections and feed-forward layers together with shared multimodal self-attention, enabling information exchange across modalities. The decoded RGB, depth, and ray maps form a complete 4D scene representation for cross-modal generation, geometric prediction, and inference-time consistency search.
Figure 3: Conditional completion with noise as masking. We specify each task by the clean/noisy pattern over RGB, depth, camera-ray, and text tokens. Clean tokens provide observed context, while noisy tokens are denoised as missing targets, enabling generation and prediction with the same model architecture.
Figure 4: Qualitative comparison of camera-controlled generation. Given the first and last RGB frames and a target camera trajectory, we compare generated intermediate frames against ground truth and baseline video generation methods. UGG better preserves scene identity and dynamic content compared with the generic video completion baselines.
Method
RealEstate10K
SpatialVID
DL3DV
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
PSNR
LPIPS
SSIM
2-view
Aether
20.02
0.196
0.624
13.84
0.489
0.334
13.14
0.399
0.243
SEVA
25.25
0.081
0.838
15.44
0.344
0.466
17.61
0.211
0.552
UGG
25.73
0.061
0.849
16.99
0.288
0.535
16.61
0.239
0.495
1-view
Aether
15.79
0.335
0.469
11.81
0.632
0.283
11.96
0.506
0.240
SEVA
19.58
0.194
0.703
13.35
0.472
0.343
13.50
0.393
0.335
Table 1: Camera-controlled generation results. Generation is conditioned on the first RGB frame (1-view) or the first and last frames (2-view), together with the camera trajectory. We report PSNR ( ↑ ), LPIPS ( ↓ ), and SSIM ( ↑ ). Darker and lighter blue shading indicate the best and second-best results within each setting.
Figure 5: Qualitative comparison of estimated camera trajectories. The temporal evolution of each trajectory is encoded by a color gradient, from purple (start) to red (end). Ground-truth trajectory is shown in black. The leftmost column shows representative RGB frames from the input video. Compared with Geo4D and RayDiffusion, our predicted trajectories more closely follow the ground-truth trajectory, with better overall structure in challenging cases.
(a) Dense observations
Method
RealEstate10K
Sintel
TUM
ScanNetV2
ScanNet++
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
ATE
RPEt
RPEr
Geo4D
0.026
0.028
0.147
0.160
0.097
0.911
0.052
0.029
1.414
0.074
0.044
1.762
0.190
0.090
5.591
RayDiffusion
0.139
0.081
1.809
0.471
0.293
14.495
0.088
0.058
2.942
0.170
0.090
8.909
0.344
0.148
25.941
Gen3R
0.030
0.011
0.141
0.233
0.066
1.538
0.046
0.016
0.943
0.075
0.024
1.018
0.232
0.066
5.760
Aether
0.018
0.014
0.851
0.180
0.867
1.354
0.041
0.016
1.462
0.075
0.030
1.617
0.225
0.068
5.944
Table 2: Camera pose estimation results. Dense: ATE, RPE t , and RPE r . Sparse: tAE and RPE r , evaluated only on the two observed frames. Angular errors are in degrees; lower is better for all metrics. Darker and lighter blue shading mark the best and second-best distinct results in each column, including oracle results.
Figure 6: Depth prediction qualitative comparison. The white box marks the region shown in the zoom-in columns, ordered as GT, Ours, ChronoDepth, and Geo4D from left to right. Our predictions preserve sharper local structures and object boundaries in the highlighted regions.
Method
ScanNet++
7-Scenes
KITTI
Sintel
Bonn
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Abs Rel
δ1.25
Dense
ChronoDepth
0.177
0.724
0.116
0.871
0.120
0.857
0.394
0.530
0.092
0.920
Geo4D
0.106
0.894
0.080
0.923
0.097
0.923
0.247
0.676
0.057
0.972
Gen3R
0.157
0.772
0.079
0.937
0.185
0.777
0.414
0.520
0.168
0.805
Aether
0.166
0.769
0.109
0.898
0.056
0.975
0.351
0.607
0.095
0.911
Marigold
0.175
0.730
0.094
0.910
0.098
0.910
0.388
0.517
0.082
0.933
Table 3: Depth prediction results. We report Abs Rel ( ↓ ) and δ<1.25 ( ↑ ) on ScanNet++, 7-Scenes, KITTI, Sintel, and Bonn in dense and sparse settings. Darker and lighter blue shading indicate the best and second-best distinct results within each setting, including oracle results.
Figure 7: Test-time search with increasing sample budget. We sample up to 16 candidate completions and select one using the image-based depth or pose verifier. The dashed line shows the single-sample result, and the solid curve shows the selected result as L increases. Search improves several pose and depth settings, showing that the generative model produces useful candidate diversity.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
ScanNetV2
ScanNet++
whist
ATE
RPE r
ATE
RPE r
0
0.0796
2.46
0.2170
5.77
1
0.0741
2.83
0.2089
6.09
2
0.0740
3.24
0.2091
6.45
4
0.0765
4.22
0.2141
7.70
Appendix
Table 5: History-guidance weight. We report ATE ( ↓ ) and RPE r ( ↓ , degrees) on ScanNetV2 and ScanNet++. Shading marks the best result in each column.
Asset
Use in this work
License / access terms
Training datasets
DL3DV-10K [ 29 ]
RGB and pose; generation evaluation
CC BY-NC 4.0 ; dataset terms; research and education only
RealEstate10K [ 74 ]
RGB and pose; generation and pose evaluation
CC BY 4.0
TartanAir / TartanAir V2 [ 54 ]
Synthetic RGB, depth, and pose
CC BY 4.0
Spring [ 33 ]
Synthetic RGB, depth, and pose
CC BY 4.0
BEDLAM [ 5 ]
Human-centric RGB, depth, and pose
BEDLAM license; noncommercial research and education
Appendix
Table 6: Datasets and pretrained assets. We summarize the role of each asset and its license or access terms. License names are abbreviated, original agreements and third-party restrictions apply.