Large-scale Digital Surface Models (DSMs) can be produced cost-effectively from satellite images via stereo-photogrammetry. However, the resulting 3D maps are often contaminated by noise, outliers, and voids. On the other hand, aerial LiDAR provides high-accuracy elevation measurements at a substantially higher cost. In this work, we study diffusion models conditioned both on photogrammetric DSMs and Pléiades imagery to refine vertically co-registered DSMs. We introduce a modified Stable Diffusion 3 architecture with a pruned text stream and a patch-wise normalization strategy, enabling stable training on LiDAR data and transfer from natural images to elevation maps. Experiments in French cities demonstrate that multimodal conditioning improves elevation accuracy, reducing Dense Urban RMSE from 6.00 to 3.45 m in the in-context cities and from 4.16 to 2.77 m in the held-out city of Bordeaux.
Figures & tables
Method
Dense DSM Correction
Void Filling
Sparse-Depth Completion
Resolution Change
LoD2-like Target
RGB Conditioning
Diff-DEM
×
✓
×
×
×
×
Dfilled
×
✓
×
×
×
✓
GAN void filling Gavriil et al. [2019] , Qiu et al. [2019] , Zhang et al. [2020] , Zhou et al. [2022]
×
✓
×
×
×
×
Marigold-DC
×
×
✓
×
×
✓
DSM super-resolution
×
×
×
✓
×
✓ *
DSM-to-LoD2
✓
×
×
×
✓
×
Table 1: Task comparison used to define the internal baseline. A checkmark denotes a primary or optional input/task association; a star marks a secondary/optional capability; a cross denotes a different target or input setting.
Figure 2
City
LiDAR
Pléiades
All Pléiades
Stereo DSM
Amiens
27,645
20,404
44,220
5077
Angers
5380
0
0
0
Arcachon
10,076
7132
21,515
1462
Biarritz
11,090
6571
23,299
4180
Bordeaux
26,191
14,376
65,273
3221
Clermont-Ferrand
9311
0
0
0
Table 2: Dataset constitution by city and modality in terms of 512×512 geographic patches. Pléiades counts locations with at least one acquisition, whereas All Pléiades counts acquisition–patch pairs across all dates.
Figure 3: Diagram of the partitioning strategy across modality-specific datasets. Patches from the same geographic location remain in the corresponding partition as modalities are added; all fully paired Bordeaux patches form the held-out test set. Adjacent patches may touch, and no geographic buffer is used.
Figure 4: Data acquisition dates by city and modality: LiDAR , Pléiades , and Stereo DSM .
Land Cover
In-Context
Bordeaux
Train
Val
Test
Test
Beaches and dunes
27.0
3.0
5.8
2.4
Coniferous forests
277.8
40.0
80.0
35.2
Deciduous forests
1183.9
171.2
328.0
106.9
Dense urban
329.3
47.0
93.3
14.6
Dispersed urban
1654.4
241.5
484.6
441.5
Table 3: Millions of pixels per Theia Land Cover 2021 class and dataset split for locations containing LiDAR-HD, Pléiades-HR, and stereo DSM data. In-context corresponds to all paired cities except Bordeaux; Bordeaux is the held-out test city.
Experiment
Trainable Modules
Frozen Modules
Steps
Batch Size
Learning Rate
A100 GPU-Hours
SD3-Pruned (Finetuned)
Backbone
–
50k
120
1×10−5
100
SD3-Pruned (Scratch)
Backbone
–
50k
120
2×10−4
100
DSM-Only
DSM CN
Backbone
20k + 30k
120 → 360
1×10−5
70 + 270
Multimodal (Stage 1)
Pléiades CN
Backbone
20k + 30k
120 → 360
1×10−5
70 + 270
Multimodal (Stage 2)
DSM CN
Backbone, Pléiades CN
20k + 30k
120 → 360
1×10−5
90 + 350
Table 4: Summary of training configurations and computational cost. CN stands for ControlNet.
Model Configuration
FD DINOv2 ↓
SD3-Pruned (pretrained)
169.4
SD3-Pruned (best scratch LR)
692.8
Table 5: Comparison of backbone initialization strategies using FD DINOv2 (lower is better). Bold indicates the best result.
Figure 5: Qualitative comparison of the two backbone configurations. From top to bottom: SD3-Pruned (Scratch), SD3-Pruned (Finetuned), and LiDAR elevation. Within each column, all generations use the same random seed. s and u are computed from the LiDAR patch and used to prompt the models without classifier-free guidance. Elevation maps are structural viridis min–max displays.
Land Cover
Method
Dense Urban
Croplands
Sparse Built-Up
Roads
Vegetation
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
In-context
Calibrated stereo DSM (CARS)
2.95
6.00
1.63
6.64
1.48
2.70
1.35
2.96
2.04
4.19
Ours—DSM only
2.14
3.85
0.47
1.04
1.50
2.68
1.23
2.51
1.93
3.63
Ours—DSM + RGB
1.91
3.45
0.47
0.98
1.32
2.37
1.06
2.14
1.75
3.32
Table 6: Conditioning results on the in-context test set and held-out Bordeaux test set. Metrics are computed where both LiDAR and the input DSM are valid, then aggregated by land-cover group. Metrics are averaged over 20 inference seeds; the calibrated DSM input is deterministic. The largest inference-seed standard deviation is 0.035 m (Appendix D , Table A8 ); these standard deviations describe generation variability for one fitted model, including the configured RGB-acquisition selection, not spatial/test uncertainty or training-run variability. All values are in meters (m); lower is better. Bold values indicate the lowest metric among the compared methods.
Figure 6: Difference in land-cover RMSE between DSM + RGB and each comparator, with paired 95% bootstrap intervals; negative values favor DSM + RGB. In-context intervals resample the eight paired cities, and Bordeaux intervals resample paired 2×2 -patch spatial blocks. In every resample, RMSE is calculated for each seed 0–19 and each learned method, then averaged across seeds before the paired difference is taken. The calibrated DSM is deterministic. Thus, both comparisons use the 20-seed learned-method estimand in Table 6 ; intervals describe spatial/test uncertainty, not inference-seed or training-run variability.
Figure 7: DSM-only refinement. Rows show the calibrated stereo DSM, DSM-only prediction, LiDAR reference, and the corresponding errors. Within each column, elevation maps share the LiDAR range; error maps use a fixed range of −10 to +10 m.
Table A1: Common training hyperparameters used across the reported experiments.
Setting
Value
Generated and reference samples
50,000 each
Precision
BF16
Sampling steps
50
Scheduler
FlowMatchEulerDiscreteScheduler
Scheduler shift
1
Classifier-free guidance
Disabled
Appendix
Table A2: Settings for comparing pretrained and scratch backbones using FD DINOv2 .
Pruned
Unpruned SD3 Medium
Configuration
Batch
Time/Batch (s)
Throughput (Samples/s)
GPU Used (GiB)
Time/Batch (s)
Throughput (Samples/s)
GPU Used (GiB)
Backbone + VAE
1
0.937
1.07
3.85
1.525
0.66
5.79
8
4.817
1.66
11.89
7.058
1.13
13.88
32
18.366
1.74
39.19
26.232
1.22
41.72
64
36.222
1.77
50.41
51.790
1.24
53.68
Backbone + 2 ControlNets + VAE
1
1.874
0.53
6.33
3.089
0.32
10.25
Appendix
Table A3: Runtime, throughput, and GPU memory for pruned and unpruned models on one NVIDIA A100 80 GB PCIe GPU. Both use BF16, 50 steps with FlowMatchEulerDiscreteScheduler, and the same VAE decoder. Times are averages of two runs after one warm-up run. Memory is measured with nvidia-smi after warm-up.
L
L+R
L+D+R
City
Train
Train
Val.
Train
Val.
In-Context Test
Bordeaux Test
Amiens
24,591
17,350
2039
3555
507
1015
0
Angers
5380
0
0
0
0
0
0
Arcachon
9071
6127
713
1024
146
292
0
Biarritz
9599
5080
656
2927
417
835
0
Bordeaux
0
0
0
0
0
0
3209
Appendix
Table A4: Numbers of geographic patches by city, available modalities, and split. L denotes LiDAR, D stereo DSM, and R Pléiades RGB. The L set is used for backbone training, L+R for the RGB stage, and L+D+R for paired training and evaluation. Multiple RGB dates at one location count as one patch. A zero indicates that no patches from that city are used in the corresponding set.
Group
Before Filtering (%)
Excluded Patches (%)
Ratio ( × )
Dense urban
5.56
10.03
1.81
Croplands
15.94
14.14
0.89
Sparse built-up
29.77
17.47
0.59
Roads
1.83
1.85
1.02
Vegetation
45.30
45.42
1.00
Other Theia classes
1.61
11.08
6.89
Appendix
Table A5: Land-cover composition before RMSE filtering and among excluded patches. Percentages are shares of pixels in each group. The ratio divides the excluded share by the original share: values above one indicate that a group is more common among exclusions. Other Theia classes includes all labels outside the five evaluation groups.
Figure A1: Patch-wise normalization statistics for 35,154 LiDAR–DSM pairs from nine cities. The top row shows the scale s=std/0.25 ; the bottom row shows the mean elevation u . LiDAR is shown on the left and the corresponding DSM on the right. Colors identify cities, weighted equally so that cities with more patches do not dominate the histograms. Each panel displays the central 98% of values, with omitted extremes noted above the plot.
Land Cover
Method
Dense Urban
Croplands
Sparse Built-Up
Roads
Vegetation
Patches excluded by the RMSE filter
Calibrated stereo DSM (CARS)
9.686
23.050
5.564
5.740
9.804
Ours—DSM only
6.179 ± 0.028
2.064 ± 0.019
4.776 ± 0.010
4.162 ± 0.015
6.849 ± 0.023
Ours—DSM + RGB
5.736 ± 0.032
2.240 ± 0.021
4.464 ± 0.017
3.743 ± 0.028
6.718 ± 0.035
Appendix
Table A6: RMSE on patches excluded by the P90 filter, calculated over all valid LiDAR/DSM pixels in each land-cover group. Values are in meters; model results are mean ± standard deviation over 20 inference seeds.
Land Cover
Method
Dense Urban
Croplands
Sparse Built-Up
Roads
Vegetation
In-context
Ours—DSM only
4.135 ± 0.038
1.179 ± 0.050
4.747 ± 0.056
3.720 ± 0.096
6.040 ± 0.029
Ours—DSM + RGB
3.576 ± 0.037
0.932 ± 0.023
3.801 ± 0.047
3.215 ± 0.083
5.362 ± 0.030
Held-out Bordeaux
Ours—DSM only
4.147 ± 0.083
3.856 ± 0.290
4.995 ± 0.029
3.309 ± 0.085
5.916 ± 0.030
Appendix
Table A7: RMSE within original DSM voids on the retained test sets. Errors are calculated over all evaluated void pixels in each land-cover group, excluding areas outside the source DSM image. Values are mean ± standard deviation in meters over 20 inference seeds.
Land Cover
Method
Dense Urban
Croplands
Sparse Built-Up
Roads
Vegetation
MAE SD
RMSE SD
MAE SD
RMSE SD
MAE SD
RMSE SD
MAE SD
RMSE SD
MAE SD
RMSE SD
In-context
Ours—DSM only
0.005
0.006
0.003
0.007
0.003
0.005
0.011
0.024
0.002
0.004
Ours—DSM + RGB
0.007
0.016
0.003
0.005
0.004
0.006
0.008
0.017
0.003
0.007
Held-out Bordeaux
Appendix
Table A8: Standard deviations of the MAE and RMSE reported in Table 6 , measured across 20 inference seeds. Evaluation uses pixels with valid LiDAR and input DSM elevations. All values are in meters.
Figure A2: Signed bias, NMAD, and P95 absolute error by land cover for the in-context test ( left ) and Bordeaux ( right ). Only pixels with valid LiDAR and input DSM elevations are evaluated. Bias uses all evaluated pixels; NMAD and P95 are estimated by sampling 256 pixels per patch, giving each patch equal weight before grouping by land cover. All values are in meters.
Figure A3: Change in RMSE after excluding land-cover boundaries and patch edges. Bars show RMSE for the remaining interior pixels minus RMSE for all evaluated pixels; negative values mean lower interior error. The top row evaluates valid DSM pixels and the bottom row evaluates original DSM voids within the source image. Percentages indicate the fraction of pixels retained in each group. Errors are in meters.
Multimodal diffusion models generate or interpret text, images, audio, video, motion, and other data by jointly denoising representations conditioned across modalities.
High-resolution digital surface models (DSMs) play an important role in urban analysis, 3D building reconstruction, and infrastructure monitoring, yet their availability remains limited due to the high cost and complexity of data acquisition. In contrast, coarse DSMs from commercial satellite missions are widely accessible, and high-resolution optical imagery is increasingly available from aerial and satellite platforms. We address the resulting mismatch in spatial resolution and propose a DSM superresolution approach that enhances 5 m DSMs to 0.5 m resolution, using guidance from high-resolution spectral images. Our method employs denoising diffusion to transfer information that is visible only in the image, like crisp outlines and detailed roof structures, into the elevation maps. In this way, surface details are reconstructed more accurately than with conventional interpolation or filtering techniques. Experiments on several cities in Central Europe demonstrate that the proposed approach produces high-quality DSMs with improved structural detail and accurate surface geometry. Our results highlight the potential of guided super-resolution with foundational image priors as a means of reconstructing high-resolution surface models.
Armand Mihai Nicolicioiu, Dominik Narnhofer, Nando Metzger +3
Photogrammetry and Remote Sensing, ETH Zürich, Zürich, Switzerland · Remote Sensing Technology Institute, German Aerospace Center (DLR), Wessling, Germany
Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.
Samed Doğan, Nico Leuze, Alfred Schöttl
Department of Electrical Engineering and Information Technology Munich University of Applied Sciences Munich, Bavaria 80335
High-beam-count LiDAR sensors are costly, yet many perception pipelines require dense angular sampling. Using a pretrained Stable Diffusion model as the backbone, we fine-tune a LiDAR-conditioned depth model with pseudo-depth targets from a 2D foundation model. During training, the LiDAR conditioning is randomly decimated at different beam budgets. We then investigate how much of a LiDAR scan can be recovered from heavily decimated input and characterize performance across the input beam budget. We evaluate against physically held-out real beams on nuScenes and report recovery separately from fit accuracy. Our model yields its largest advantage in very sparse regimes, achieving a δ1.25 accuracy of 66.8% from 4-beam input where scattered interpolation reaches only 45.1%. A class-stratified error breakdown further reveals that planar surfaces recover first while objects introducing depth discontinuities degrade earliest. Together, these results quantify the recovery/resolution trade-off for foundation-model-guided LiDAR enhancement.
Samed Doğan, Nico Leuze, Alfred Schöttl
Dept. of Electrical Engineering and Information Technology Munich University of Applied Sciences, 80335 Munich, Germany