Diffusion-based approaches have recently demonstrated strong performance for single-image novel view synthesis by conditioning generative models on geometry inferred from monocular depth estimation. However, in practice, the quality and consistency of the synthesized views are fundamentally limited by the reliability of the underlying depth estimates, which are often fragile under low-texture, adverse weather, and occlusion-heavy real-world conditions. In this work, we show that incorporating sparse multimodal range measurements provides a simple yet effective way to overcome these limitations. We introduce a multimodal depth reconstruction framework that leverages extremely sparse range sensing data, such as automotive radar or LiDAR, to produce dense depth maps that serve as robust geometric conditioning for diffusion-based novel view synthesis. Our approach models depth in an angular domain using a localized Gaussian Process formulation, enabling computationally efficient inference while explicitly quantifying uncertainty in regions with limited observations. The reconstructed depth and uncertainty are used as a drop-in replacement for monocular depth estimators in existing diffusion-based rendering pipelines, without modifying the generative model itself. Experiments on real-world multimodal driving scenes demonstrate that replacing vision-only depth with our sparse range-based reconstruction substantially improves both geometric consistency and visual quality in single-image novel-view video generation. These results highlight the importance of reliable geometric priors for diffusion-based view synthesis and demonstrate the practical benefits of multimodal sensing even at extreme levels of sparsity. Code is publicly available at: https://github.com/importAmir/MultiModalNVS
Figures & tables
Figure 1: Overview of the proposed multimodal single-image novel-view synthesis pipeline. Sparse range sensing measurements are first processed by the proposed GP-based depth reconstruction module to produce a dense depth map. The reconstructed depth and the input RGB image are used to form a colored 3D point cloud, which is rendered along a target camera trajectory to generate sparse novel-view conditioning frames. These rendered frames are provided as geometric conditioning to a diffusion model, which synthesizes a temporally consistent video at the target viewpoints.
Figure 2: Illustration of the proposed localized Gaussian Process depth reconstruction in the angular domain. Left: Sparse radar range measurements represented in azimuth–elevation space, with the current target viewing direction indicated. Right: Zoomed-in view around the target location, highlighting the local neighborhood used for Gaussian Process inference. Only range measurements within this localized region contribute to the depth estimation at the target direction, enabling efficient and spatially adaptive depth reconstruction from sparse observations.
Method
Pixel coverage
PSNR↑
SSIM↑
LPIPS↓
FID↓
t−LPIPS↓
GEN3C (Vision-Only Monocular Depth)
—
12.36
0.4561
0.5804
152.62
0.1117
GEN3C (Ours, Multi-Modal with radar)
0.02%
14.26
0.4860
0.4441
82.41
0.0790
GEN3C (Ours, Multi-Modal with LiDAR)
0.52%
14.69
0.4971
0.4230
71.91
0.0563
Table 1: Quantitative evaluation on the View-of-Delft dataset for single-image novel-view video generation with GEN3C ( Ren et al., 2025 ) under three depth-conditioning variants: (i) the default pipeline using the vision-only monocular depth estimator MoGe ( Wang et al., 2025 ) , (ii) the same pipeline with monocular depth replaced by our multimodal sparse range-sensor depth reconstruction module using synchronized radar measurements (approximately 0.02% pixel coverage), and (iii) the same replacement using synchronized LiDAR measurements (approximately 0.52% pixel coverage). Metrics are computed with respect to ground-truth target views, and the results show consistent improvements across all measures when using our reconstructed depth, emphasizing the role of reliable depth priors in diffusion-based novel-view synthesis from a single image.
Figure 3: Qualitative comparison on single-image novel-view synthesis on the View-of-Delft dataset. From left to right, we show the input image, the novel view generated by GEN3C ( Ren et al., 2025 ) using its default monocular depth estimator, MoGe ( Wang et al., 2025 ) , the novel view generated by GEN3C when replacing monocular depth with our sparse range-sensor depth reconstruction module, and the ground-truth target view. For each generated view, we report LPIPS with respect to the ground truth (lower is better). Across all examples, our depth reconstruction yields consistently lower LPIPS and improved geometric alignment, underscoring the importance of reliable geometry for diffusion-based rendering from single-view inputs.
Method
MAE↓
RMSElog↓
MoGe
14.25
2.23
Depth Anything V2
18.70
0.94
Ours (Sparse Radar Depth)
13.61
0.92
Table 2: Depth estimation accuracy on the View-of-Delft dataset evaluated on the first frame of each of the 26 selected segments. Our multimodal depth reconstruction uses synchronized radar measurements that cover approximately 0.02% of image pixels. Predicted depth is compared against LiDAR-derived depth at pixels where LiDAR measurements are available. Lower is better for both metrics.
Retention Ratio
PSNR↑
SSIM↑
LPIPS↓
FID↓
t−LPIPS↓
40%
14.36
0.4713
0.4143
78.46
0.0583
60%
14.72
0.4747
0.4136
69.82
0.0583
80%
14.80
0.4826
0.4029
67.91
0.0578
100%
14.70
0.4800
0.4139
78.44
0.0648
Table 3: Ablation on uncertainty-based masking for depth-conditioned novel-view synthesis. After reconstructing dense depth and predictive uncertainty, we retain only the p% most certain pixels and mask out the remaining high-uncertainty regions before rendering the conditioning frames. Results on 9 videos from the View-of-Delft dataset show that retaining 80% of the lowest-uncertainty pixels provides the best overall trade-off between geometric reliability and conditioning coverage, leading to the strongest downstream generation performance.
Locality Radius
MAE↓
RMSElog↓
r = 1
10.88
0.646
r = 2
10.67
0.627
r = 4
10.94
0.648
Table 4: Ablation on the angular locality radius used in dense depth estimation. Using the same 9 videos from the View-of-Delft dataset as in the previous ablation, we vary the local neighborhood radius r when predicting dense depth from sparse radar observations and evaluate the predictions against LiDAR ground truth on valid pixels. A radius of 2∘ achieves the best overall performance, while increasing the radius to 4∘ does not improve accuracy and incurs higher computational cost due to the larger number of local points considered during Gaussian Process inference.