cs.ROSep 24, 2026
SaveM3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis
Organizations: New York University, New York, USA · Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, USA · U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD 20783, USA
Abstract
Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned LiDAR content and that target-view LiDAR acts as a geometric query linking the requested view to source observations. Deployment on a ground robot demonstrates practical real-world operation, with a configurable quality--cost trade-off controlled by the number of Euler integration steps.
Figures & tables
Fig. 1: Projected RGB–LiDAR representation. Frozen Utonia per-point descriptors are projected and pooled into the same image-latent grid produced by the frozen DA3 encoder, so each cell indexes both and (first three PCA components; hatched cells have no LiDAR support, ).
Fig. 3: Clearpath Warthog platform used for the real-robot deployment.
| Feature pair | PCA ch. corr. | CKA | SVCCA- | Jaccard@ | Affinity corr. |
|---|---|---|---|---|---|
| DA3-Base feature level vs. Utonia | |||||
| DA3-Base level- –Utonia | |||||
| DA3-Base level- –Utonia | |||||
| DA3-Base level- –Utonia | |||||
| DA3-Base level- –Utonia | |||||
| Image backbone comparison vs. Utonia | |||||
TABLE I: Feature-structure similarity on GrandTour. All metrics are computed on shared image-grid cells with valid projected LiDAR support. The study covers successfully processed images from recordings and three HDR camera streams. Best and second-best are highlighted within each comparison block.
Fig. 4: Qualitative feature-structure visualization on GrandTour. For each camera image, we show the RGB input, the DA3-Base level- PCA pseudo-color map, projected Utonia features, and DINOv2 final-layer image features. For each image, DA3-Base, Utonia, and DINOv2 features are independently projected to three local PCA components on the shared valid grid cells. The Utonia and DINOv2 coordinates are aligned to DA3-Base level- using orthogonal Procrustes. Black regions in the Utonia column indicate cells without valid projected LiDAR support. Projected Utonia features recover large-scale geometric partitions similar to DA3-Base level- on LiDAR-supported cells, while DINOv2 provides an image-only reference.
| 2D Appearance | 2D Geometry (Depth) | 3D Reconstruction | ||||||
| Method | PSNR | SSIM | LPIPS | AbsRel | RMSE | MEt3R | Reproj. | |
| Diffusion-based methods | ||||||||
| Matrix3D [ 73 ] | ||||||||
| MVGenMaster [ 74 ] | ||||||||
| GLD [ 8 ] | ||||||||
| GLD ∗† [ 8 ] | ||||||||
TABLE II: Baseline comparison on the GrandTour evaluation split ( clips). / indicate that higher/lower is better; best and second-best per column are highlighted. ∗ fine-tuned on the GrandTour training set; † trained on the DA3 level- feature.
| Method | AbsRel | RMSE | |
|---|---|---|---|
| Native readout (GLD-default) | |||
| GLD | |||
| GLD ∗† | |||
| M3GD ∗† (Ours) | |||
| RGB readout (protocol of Table II ) | |||
| GLD | |||
TABLE III: Depth readout comparison, using identical latent predictions as Table II . The native readout scores depth decoded directly from the synthesized latent (GLD-default); the RGB readout first decodes to RGB and re-encodes through the DA3 encoder (protocol of Table II ). Both are scored against the same pseudo-ground truth. Notation follows Table II ; best within each readout.
| Method | Latency (s) | Memory (GiB) |
|---|---|---|
| Matrix3D [ 73 ] | ||
| MVGenMaster [ 74 ] | ||
| GLD [ 8 ] | ||
| M3GD ∗† (Ours) |
TABLE IV: Sampling cost of the diffusion methods on the GrandTour evaluation split. All methods use source views and target views. Latency is measured per target pair over the full sampling procedure, including the VAE decode to RGB and, for GLD, both stages of its native cascade; for M3GD it also includes generation of the LiDAR packet. Values are averaged over three runs ( standard deviation); peak reserved GPU memory is measured on a single NVIDIA RTX A5000, following the protocol of Table VII . Best per column.
Fig. 5: Qualitative RGB and depth comparison on the GrandTour evaluation split. For each scene, the two source images are shown on the left. The middle block compares RGB predictions, and is organized into three rows: feed-forward splatting baselines DepthSplat [ 16 ] , MVSplat [ 15 ] , and PixelSplat [ 14 ] ; diffusion-based baselines Matrix3D [ 73 ] and MVGenMaster [ 74 ] together with the original GLD model [ 8 ] ; and GLD ∗† , our GrandTour-fine-tuned image-only GLD baseline, followed by M3GD and the ground-truth target view. The right block shows the corresponding DA3 depth estimates for the same predicted views and the ground-truth target depth, using the same method layout. Compared with splatting and image-only generative baselines, M3GD better preserves target-view appearance and produces depth maps that more closely follow the ground-truth scene geometry in challenging robot-captured scenes.
Fig. 6: Qualitative visualization of M3GD on GrandTour evaluation samples. Each group follows the actual evaluation layout: two observed source views, two M3GD target-view predictions, and the corresponding ground-truth target views. M3GD synthesizes targets from sparse source images and target-view LiDAR geometry; target RGB is never provided. Examples are ordered from indoor scenes to semi-open structures and outdoor environments, showing that M3GD preserves dominant scene layout and reconstructs plausible target-view appearance across diverse and geometrically challenging robot-captured environments.
| Training mode | LiDAR | PSNR | SSIM | LPIPS |
|---|---|---|---|---|
| freeze | ✗ ‡ | |||
| ✓ § | ||||
| scratch | ✗ | |||
| ✓ | ||||
| fine-tune | ✗ | |||
| ✓ |
TABLE V: Effect of LiDAR conditioning across training modes (multi-step, level- latent). Best per column. ‡ Off-the-shelf: frozen released GLD checkpoint, no LiDAR adapter. § As ‡ , with the LiDAR adapter trained. tgt-only scores the loss on the target-view latents only, not on the jointly denoised source views. ¶ Classifier-free LiDAR (cfg-dropout ); at evaluation a single CFG scale of is applied to the joint camera+LiDAR condition.
| LiDAR input mode | PSNR | SSIM | LPIPS |
|---|---|---|---|
| Full (real packet) | |||
| Geom-only | |||
| Zero | |||
| Learnable param | |||
| No LiDAR (floor) |
TABLE VI: Effect of LiDAR input mode on the hard subset (maximum camera baseline between views m, ; Section V-B ). Each row is a separately trained model under the default recipe of Section VI-D1 ; only the LiDAR input changes. Highlighting follows Table II .
Fig. 7: Qualitative examples highlighting the effect of target-view LiDAR conditioning. Each row shows two source images, the image-only GLD ∗† baseline, the raw target-view LiDAR projection, our downsampled LiDAR representation, the geometry-only (geom-only) variant, M3GD, and the ground-truth target image. These examples contain large source-target viewpoint changes where RGB-only conditioning is ambiguous and GLD ∗† often fails to infer the target-facing surface. The LiDAR-conditioned geom-only model recovers the dominant scene layout more accurately, while M3GD further improves appearance fidelity and local details such as boxes, road surfaces, walls, and vehicles by combining LiDAR geometry with Utonia features.
Fig. 8: Qualitative results for two real-robot queries Q1 (rows – ) and Q2 (rows – ), using Euler steps, source views, and target views. Each query forms a two-row block: in the first two columns, the upper row shows the two source RGB images and the lower row shows their corresponding projected LiDAR point clouds. The remaining columns show the ground-truth RGB, M3GD synthesized RGB, our RGB readout depth, GLD synthesized RGB, and GLD RGB readout depth (defined in Section VI-B ) for target T1 in the upper row and target T2 in the lower row.
| Configuration | M3GD Quality | M3GD Computational Cost | |||||
|---|---|---|---|---|---|---|---|
| Steps | Targets | PSNR (dB) | SSIM | LPIPS | Latency (s) | Lat./Target (s) | Mem. (GiB) |
TABLE VII: On-robot quality, latency, and memory scaling on the Warthog platform with two fixed source views ( ). Quality is averaged over queries; latency and peak GPU reserved memory are measured over four synchronized post-warm-up queries.