Recovering metric-scale geometry from observations at a single viewpoint remains fundamentally ill-posed, limiting the reliability of novel view synthesis. Recent methods rely on learned monocular depth priors, which provide strong structural estimates but can suffer from scale ambiguity and dataset-dependent biases. We propose DfD-GS, a feed-forward framework that predicts a metric-scale 3D Gaussian scene representation from a single-view all-in-focus--defocused image pair. The key idea is to exploit defocus blur as a physical metric depth cue. DfD-GS first estimates a defocus-based metric disparity map from the all-in-focus--defocused image pair and the corresponding defocus strength. This estimate is used to align and refine a monocular depth prior to metric scale. Visible and hidden 3D Gaussians are then predicted for occlusion-aware novel view synthesis. Experiments on synthetic-defocus datasets constructed from Booster and Middlebury show that DfD-GS outperforms single-image baselines in metric depth estimation and novel view synthesis. On WildRGB-D, DfD-GS achieves lower AbsRel and LPIPS but higher depth RMSE than the single-image baselines. Additional analyses show that DfD-GS maintains a more stable metric scale under field-of-view changes. Qualitative results on a real dataset show recovery of the main scene structure under optical defocus.
Figures & tables
Fig. 1: Given a single-view all-in-focus–defocused image pair and the corresponding defocus strength, DfD-GS predicts metric-scale 3D Gaussians by exploiting defocus blur as a physical depth cue.
Fig. 2: Overview of DfD-GS. A defocus-conditioned U-Net estimates a metric disparity offset from the all-in-focus–defocused image pair and the defocus strength. This estimate is used to globally align the normalized disparity-offset prior from DA3. The aligned offset is converted to metric disparity and refined using a predicted scale map. The refined depth, all-in-focus image, and DA3 features are then used to predict visible and hidden 3D Gaussians for novel view synthesis.
Symbol
Definition
I , Idef , kdef
All-in-focus image, defocused image, and defocus strength.
zf , df
Known focus distance and its inverse ( df=1/zf ).
0pt , D
Metric depth and disparity ( D=1/0pt ).
ΔD
Disparity offset from the focus plane: ΔD=df−D .
FDA
Multi-scale DA3 feature maps.
DDA
DA3 normalized disparity-offset prior.
TABLE I: Main notation used in the proposed method.
Fig. 3: Depth comparisons on the Booster dataset. For better visualization, the brightness of the input images was increased by a factor of 1.2.
Fig. 4: Depth comparisons on the Middlebury dataset.
Fig. 5: Depth comparisons on the WildRGB-D dataset.
Method
AbsRel ↓
RMSE ↓
δ1↑
DA3 [ 23 ] (metric)
0.426 ± 0.110
0.500 ± 0.268
0.06 ± 0.22
Depth Pro [ 8 ]
0.271 ± 0.257
0.283 ± 0.220
0.63 ± 0.44
Metric3D v2 [ 6 ]
0.372 ± 0.302
0.392 ± 0.250
0.41 ± 0.43
MoGe [ 25 ]
0.291 ± 0.301
0.292 ± 0.265
0.51 ± 0.45
UniDepth v1 [ 24 ]
0.283 ± 0.243
0.339 ± 0.257
0.62 ± 0.40
UniDepth v2 [ 7 ]
0.230 ± 0.200
0.248 ± 0.177
0.64 ± 0.44
TABLE II: Depth comparisons on the Booster dataset.
Method
AbsRel ↓
RMSE ↓
δ1↑
DA3 [ 23 ] (metric)
0.430 ± 0.114
1.183 ± 0.665
0.08 ± 0.18
Depth Pro [ 8 ]
0.237 ± 0.254
0.662 ± 0.618
0.70 ± 0.40
Metric3D v2 [ 6 ]
0.236 ± 0.226
0.614 ± 0.509
0.80 ± 0.36
MoGe [ 25 ]
0.315 ± 0.321
0.737 ± 0.608
0.67 ± 0.41
UniDepth v1 [ 24 ]
0.189 ± 0.156
0.568 ± 0.437
0.64 ± 0.41
UniDepth v2 [ 7 ]
0.229 ± 0.162
0.636 ± 0.461
0.54 ± 0.45
TABLE III: Depth comparisons on the Middlebury dataset.
Method
AbsRel ↓
RMSE ↓
δ1↑
DA3 [ 23 ] (metric)
0.323 ± 0.101
0.740 ± 0.288
0.35 ± 0.28
Depth Pro [ 8 ]
0.561 ± 0.427
0.448 ± 0.260
0.36 ± 0.27
Metric3D v2 [ 6 ]
0.514 ± 0.288
0.387 ± 0.185
0.36 ± 0.24
MoGe [ 25 ]
0.503 ± 0.367
0.406 ± 0.217
0.39 ± 0.26
UniDepth v1 [ 24 ]
0.769 ± 0.392
0.493 ± 0.188
0.29 ± 0.20
Pentland [ 9 ]
0.735 ± 1.977
115.1 ± 502.5
0.72 ± 0.13
TABLE IV: Depth comparisons on the WildRGB-D dataset.
Fig. 6: Effect of f-number on depth estimation accuracy.
Fig. 7: Effect of field of view on depth estimation accuracy.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
AdaMPI [ 3 ]
22.86 ± 4.10
0.743 ± 0.137
0.285 ± 0.090
Flash3D [ 4 ]
22.78 ± 3.94
0.737 ± 0.134
0.313 ± 0.090
SHARP [ 5 ]
22.36 ± 4.08
0.738 ± 0.135
0.262 ± 0.097
Ours
24.37 ± 3.40
0.807 ± 0.086
0.257 ± 0.056
w/o DA3 Prior
24.31 ± 3.35
0.816 ± 0.083
0.267 ± 0.054
w/o Scale Map
22.03 ± 3.83
0.716 ± 0.120
0.347 ± 0.072
TABLE V: NVS comparisons on the Booster dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
AdaMPI [ 3 ]
17.48 ± 3.83
0.503 ± 0.174
0.331 ± 0.137
Flash3D [ 4 ]
16.07 ± 4.01
0.425 ± 0.168
0.430 ± 0.139
SHARP [ 5 ]
16.19 ± 3.64
0.454 ± 0.164
0.344 ± 0.121
Ours
18.44 ± 3.25
0.576 ± 0.153
0.310 ± 0.104
w/o DA3 Prior
17.56 ± 3.11
0.530 ± 0.155
0.360 ± 0.108
w/o Scale Map
15.87 ± 2.53
0.439 ± 0.120
0.422 ± 0.096
TABLE VI: NVS comparisons on the Middlebury dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
AdaMPI [ 3 ]
15.18 ± 2.80
0.464 ± 0.135
0.421 ± 0.098
Flash3D [ 4 ]
15.30 ± 2.63
0.472 ± 0.133
0.431 ± 0.097
SHARP [ 5 ]
7.29 ± 3.31
0.115 ± 0.140
0.674 ± 0.109
Ours
14.52 ± 2.80
0.470 ± 0.143
0.406 ± 0.084
w/o DA3 Prior
14.48 ± 2.75
0.457 ± 0.139
0.423 ± 0.084
w/o Scale Map
13.57 ± 2.77
0.436 ± 0.134
0.451 ± 0.091
TABLE VII: NVS comparisons on the WildRGB-D dataset.
Fig. 8: NVS comparisons on the Booster dataset. For better visualization, the image brightness was increased by a factor of 1.2.
Fig. 9: NVS comparisons on the Middlebury dataset.
Fig. 10: NVS comparisons on the WildRGB-D dataset.
Fig. 11: NVS and depth comparisons on the Booster dataset. For better visualization, the image brightness was increased by a factor of 1.2.
Fig. 12: NVS and depth comparisons on the Middlebury dataset.
Fig. 13: NVS and depth comparisons on the WildRGB-D dataset.
Fig. 14: Metric depth comparisons on five real scenes captured at 70 mm. The reference column shows the input images and measured distances. The remaining columns show estimated depth maps (normalized for visualization) and the corresponding estimated distances.
Fig. 15: Results on five real scenes captured at shorter focal lengths (24–34 mm, left) and 70 mm (right). Each group shows the all-in-focus–defocused input pair, source-view rendered depth (normalized for visualization), and a novel view. Blue annotations indicate estimated distances and signed errors (estimate minus ground truth). Focal lengths and focus distances are shown in each row.
Dataset
Prediction (ms)
Rendering (ms)
Booster
42.23
2.04
Middlebury
43.29
2.05
WildRGB-D
42.25
2.10
TABLE S1: Median Gaussian prediction and per-view rendering times.
Dataset
AbsRel ↓
RMSE ↓
δ1↑
Booster
0.084±0.008
0.156±0.009
0.91±0.01
Middlebury
0.114±0.013
0.467±0.044
0.90±0.02
WildRGB-D
0.313±0.048
1.011±0.258
0.56±0.03
Dataset
PSNR (dB) ↑
SSIM ↑
LPIPS ↓
Booster
24.10±0.16
0.802±0.008
0.272±0.016
Middlebury
18.41±0.24
0.579±0.011
0.317±0.008
TABLE S2: Mean ± standard deviation across five training seeds.
Diffusion-based approaches have recently demonstrated strong performance for single-image novel view synthesis by conditioning generative models on geometry inferred from monocular depth estimation. However, in practice, the quality and consistency of the synthesized views are fundamentally limited by the reliability of the underlying depth estimates, which are often fragile under low-texture, adverse weather, and occlusion-heavy real-world conditions. In this work, we show that incorporating sparse multimodal range measurements provides a simple yet effective way to overcome these limitations. We introduce a multimodal depth reconstruction framework that leverages extremely sparse range sensing data, such as automotive radar or LiDAR, to produce dense depth maps that serve as robust geometric conditioning for diffusion-based novel view synthesis. Our approach models depth in an angular domain using a localized Gaussian Process formulation, enabling computationally efficient inference while explicitly quantifying uncertainty in regions with limited observations. The reconstructed depth and uncertainty are used as a drop-in replacement for monocular depth estimators in existing diffusion-based rendering pipelines, without modifying the generative model itself. Experiments on real-world multimodal driving scenes demonstrate that replacing vision-only depth with our sparse range-based reconstruction substantially improves both geometric consistency and visual quality in single-image novel-view video generation. These results highlight the importance of reliable geometric priors for diffusion-based view synthesis and demonstrate the practical benefits of multimodal sensing even at extreme levels of sparsity. Code is publicly available at: https://github.com/importAmir/MultiModalNVS
Amirhosein Javadi, Chi-Shiang Gau, Konstantinos D. Polyzos +1
Department of Electrical and Computer Engineering, University of California San Diego
Gaussian Splatting (GS) has emerged as an efficient approach for high-quality novel view synthesis. While early GS variants struggled to accurately model the scene's geometry, recent advancements constraining the Gaussians' spread and shapes, such as 2D Gaussian Splatting, have significantly improved geometric fidelity. In this paper, we present Pixel-Aligned 1DoF Gaussian Splatting (PAGaS) that adapts the GS representation from novel view synthesis to the multi-view stereo depth task. Our key contribution is modeling a pixel's depth using one-degree-of-freedom (1DoF) Gaussians that remain tightly constrained during optimization. Unlike existing approaches, our Gaussians' positions and sizes are restricted by the back-projected pixel volumes, leaving depth as the sole degree of freedom to optimize. PAGaS produces highly detailed depths, as illustrated in Figure 1. We quantitatively validate these improvements on top of reference geometric and learning-based multi-view stereo baselines on challenging 3D reconstruction benchmarks. Code: davidrecasens.github.io/pagas
Depth from Defocus (DfD) is the task of estimating a dense metric depth map from a focus stack. Unlike previous works overfitting to a certain dataset, this paper focuses on the challenging and practical setting of zero-shot generalization. We first propose a new real-world DfD benchmark ZEDD, which contains 8.3x more scenes and significantly higher quality images and ground-truth depth maps compared to previous benchmarks. We also design a novel network architecture named FOSSA. FOSSA is a Transformer-based architecture with novel designs tailored to the DfD task. The key contribution is a stack attention layer with a focus distance embedding, allowing efficient information exchange across the focus stack. Finally, we develop a new training data pipeline allowing us to utilize existing large-scale RGBD datasets to generate synthetic focus stacks. Experiment results on ZEDD and other benchmarks show a significant improvement over the baselines, reducing errors by up to 55.7%. The ZEDD benchmark is released at https://zedd.cs.princeton.edu. The code and checkpoints are released at https://github.com/princeton-vl/FOSSA.
Yiming Zuo, Hongyu Wen, Venkat Subramanian +5
Department of Computer Science, Princeton University