Enhancing Photogrammetric Digital Surface Models with Pretrained Diffusion Models and Multimodal Conditioning

Antoine Lorentz; Stéphane May; Valentine Bellet; Dawa Derksen; Bastien Nespoulous

eess.IVSep 25, 2026

Enhancing Photogrammetric Digital Surface Models with Pretrained Diffusion Models and Multimodal Conditioning

Authors: Antoine Lorentz, Stéphane May, Valentine Bellet, Dawa Derksen, Bastien Nespoulous

Organizations: Thales, 92098 Paris, France · Centre National d’Études Spatiales (CNES), 75039 Paris, France

Abstract

Large-scale Digital Surface Models (DSMs) can be produced cost-effectively from satellite images via stereo-photogrammetry. However, the resulting 3D maps are often contaminated by noise, outliers, and voids. On the other hand, aerial LiDAR provides high-accuracy elevation measurements at a substantially higher cost. In this work, we study diffusion models conditioned both on photogrammetric DSMs and Pléiades imagery to refine vertically co-registered DSMs. We introduce a modified Stable Diffusion 3 architecture with a pruned text stream and a patch-wise normalization strategy, enabling stable training on LiDAR data and transfer from natural images to elevation maps. Experiments in French cities demonstrate that multimodal conditioning improves elevation accuracy, reducing Dense Urban RMSE from 6.00 to 3.45 m in the in-context cities and from 4.16 to 2.77 m in the held-out city of Bordeaux.

Figures & tables

Method	Dense DSM Correction	Void Filling	Sparse-Depth Completion	Resolution Change	LoD2-like Target	RGB Conditioning
Diff-DEM	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
Dfilled	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$
GAN void filling Gavriil et al. [2019] , Qiu et al. [2019] , Zhang et al. [2020] , Zhou et al. [2022]	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
Marigold-DC	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$
DSM super-resolution	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$ *
DSM-to-LoD2	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$

Table 1: Task comparison used to define the internal baseline. A checkmark denotes a primary or optional input/task association; a star marks a secondary/optional capability; a cross denotes a different target or input setting.

City	LiDAR	Pléiades	All Pléiades	Stereo DSM
Amiens	27,645	20,404	44,220	5077
Angers	5380	0	0	0
Arcachon	10,076	7132	21,515	1462
Biarritz	11,090	6571	23,299	4180
Bordeaux	26,191	14,376	65,273	3221
Clermont-Ferrand	9311	0	0	0

Table 2: Dataset constitution by city and modality in terms of

512\times 512

geographic patches. Pléiades counts locations with at least one acquisition, whereas All Pléiades counts acquisition–patch pairs across all dates.

Panel of Figure 3 — Figure 3: Diagram of the partitioning strategy across modality-specific datasets. Patches from the same geographic location remain in the corresponding partition as modalities are added; all fully paired Bordeaux patches form the held-out test set. Adjacent patches may touch, and no geographic buffer is used.

Figure 4: Data acquisition dates by city and modality: LiDAR , Pléiades , and Stereo DSM .

Land Cover	In-Context			Bordeaux
	Train	Val	Test	Test
Beaches and dunes	27.0	3.0	5.8	2.4
Coniferous forests	277.8	40.0	80.0	35.2
Deciduous forests	1183.9	171.2	328.0	106.9
Dense urban	329.3	47.0	93.3	14.6
Dispersed urban	1654.4	241.5	484.6	441.5

Table 3: Millions of pixels per Theia Land Cover 2021 class and dataset split for locations containing LiDAR-HD, Pléiades-HR, and stereo DSM data. In-context corresponds to all paired cities except Bordeaux; Bordeaux is the held-out test city.

Experiment	Trainable Modules	Frozen Modules	Steps	Batch Size	Learning Rate	A100 GPU-Hours
SD3-Pruned (Finetuned)	Backbone	–	50k	120	$1\times 10^{-5}$	100
SD3-Pruned (Scratch)	Backbone	–	50k	120	$2\times 10^{-4}$	100
DSM-Only	DSM CN	Backbone	20k + 30k	120 $\rightarrow$ 360	$1\times 10^{-5}$	70 + 270
Multimodal (Stage 1)	Pléiades CN	Backbone	20k + 30k	120 $\rightarrow$ 360	$1\times 10^{-5}$	70 + 270
Multimodal (Stage 2)	DSM CN	Backbone, Pléiades CN	20k + 30k	120 $\rightarrow$ 360	$1\times 10^{-5}$	90 + 350

Table 4: Summary of training configurations and computational cost. CN stands for ControlNet.

Model Configuration	FD DINOv2 $\downarrow$
SD3-Pruned (pretrained)	169.4
SD3-Pruned (best scratch LR)	692.8

Table 5: Comparison of backbone initialization strategies using FD DINOv2 (lower is better). Bold indicates the best result.

Figure 5: Qualitative comparison of the two backbone configurations. From top to bottom: SD3-Pruned (Scratch), SD3-Pruned (Finetuned), and LiDAR elevation. Within each column, all generations use the same random seed. $s$ and $u$ are computed from the LiDAR patch and used to prompt the models without classifier-free guidance. Elevation maps are structural viridis min–max displays.

	Land Cover
Method	Dense Urban		Croplands		Sparse Built-Up		Roads		Vegetation
	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
In-context
Calibrated stereo DSM (CARS)	2.95	6.00	1.63	6.64	1.48	2.70	1.35	2.96	2.04	4.19
Ours—DSM only	2.14	3.85	0.47	1.04	1.50	2.68	1.23	2.51	1.93	3.63
Ours—DSM + RGB	1.91	3.45	0.47	0.98	1.32	2.37	1.06	2.14	1.75	3.32

Table 6: Conditioning results on the in-context test set and held-out Bordeaux test set. Metrics are computed where both LiDAR and the input DSM are valid, then aggregated by land-cover group. Metrics are averaged over 20 inference seeds; the calibrated DSM input is deterministic. The largest inference-seed standard deviation is 0.035 m (Appendix D , Table A8 ); these standard deviations describe generation variability for one fitted model, including the configured RGB-acquisition selection, not spatial/test uncertainty or training-run variability. All values are in meters (m); lower is better. Bold values indicate the lowest metric among the compared methods.

Figure 6: Difference in land-cover RMSE between DSM + RGB and each comparator, with paired 95% bootstrap intervals; negative values favor DSM + RGB. In-context intervals resample the eight paired cities, and Bordeaux intervals resample paired $2\times 2$ -patch spatial blocks. In every resample, RMSE is calculated for each seed 0–19 and each learned method, then averaged across seeds before the paired difference is taken. The calibrated DSM is deterministic. Thus, both comparisons use the 20-seed learned-method estimand in Table 6 ; intervals describe spatial/test uncertainty, not inference-seed or training-run variability.

Figure 7: DSM-only refinement. Rows show the calibrated stereo DSM, DSM-only prediction, LiDAR reference, and the corresponding errors. Within each column, elevation maps share the LiDAR range; error maps use a fixed range of $-10$ to $+10$ m.

Figure 8: DSM + RGB refinement. Rows show the calibrated stereo DSM, Pléiades RGB image, multimodal prediction, LiDAR reference, and the corresponding errors. Within each column, elevation maps share the LiDAR range; error maps use a fixed range of $-10$ to $+10$ m. Pléiades ©CNES 2015/2023/2025, Distribution CNES PWH.

Figure 9: Detailed comparison of DSM-only and DSM + RGB refinement. Rows show the Pléiades image, enlarged image crop, calibrated stereo DSM, DSM-only prediction, DSM + RGB prediction, and LiDAR reference. Within each column, elevation maps share the LiDAR range. Pléiades ©CNES 2025, Distribution CNES PWH.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
Learning-rate schedule	Constant
Weight decay	0.01
Mixed precision	BF16
Gradient clipping	1
Timestep sampling	Uniform

Appendix

Table A1: Common training hyperparameters used across the reported experiments.

Setting	Value
Generated and reference samples	50,000 each
Precision	BF16
Sampling steps	50
Scheduler	FlowMatchEulerDiscreteScheduler
Scheduler shift	1
Classifier-free guidance	Disabled

Appendix

Table A2: Settings for comparing pretrained and scratch backbones using FD DINOv2 .

		Pruned			Unpruned SD3 Medium
Configuration	Batch	Time/Batch (s)	Throughput (Samples/s)	GPU Used (GiB)	Time/Batch (s)	Throughput (Samples/s)	GPU Used (GiB)
Backbone + VAE	1	0.937	1.07	3.85	1.525	0.66	5.79
	8	4.817	1.66	11.89	7.058	1.13	13.88
	32	18.366	1.74	39.19	26.232	1.22	41.72
	64	36.222	1.77	50.41	51.790	1.24	53.68
Backbone + 2 ControlNets + VAE	1	1.874	0.53	6.33	3.089	0.32	10.25

Appendix

Table A3: Runtime, throughput, and GPU memory for pruned and unpruned models on one NVIDIA A100 80 GB PCIe GPU. Both use BF16, 50 steps with FlowMatchEulerDiscreteScheduler, and the same VAE decoder. Times are averages of two runs after one warm-up run. Memory is measured with nvidia-smi after warm-up.

	$L$	$L+R$		$L+D+R$
City	Train	Train	Val.	Train	Val.	In-Context Test	Bordeaux Test
Amiens	24,591	17,350	2039	3555	507	1015	0
Angers	5380	0	0	0	0	0	0
Arcachon	9071	6127	713	1024	146	292	0
Biarritz	9599	5080	656	2927	417	835	0
Bordeaux	0	0	0	0	0	0	3209

Appendix

Table A4: Numbers of geographic patches by city, available modalities, and split.

L

denotes LiDAR,

D

stereo DSM, and

R

Pléiades RGB. The

L

set is used for backbone training,

L+R

for the RGB stage, and

L+D+R

for paired training and evaluation. Multiple RGB dates at one location count as one patch. A zero indicates that no patches from that city are used in the corresponding set.

Group	Before Filtering (%)	Excluded Patches (%)	Ratio ( $\times$ )
Dense urban	5.56	10.03	1.81
Croplands	15.94	14.14	0.89
Sparse built-up	29.77	17.47	0.59
Roads	1.83	1.85	1.02
Vegetation	45.30	45.42	1.00
Other Theia classes	1.61	11.08	6.89

Appendix

Table A5: Land-cover composition before RMSE filtering and among excluded patches. Percentages are shares of pixels in each group. The ratio divides the excluded share by the original share: values above one indicate that a group is more common among exclusions. Other Theia classes includes all labels outside the five evaluation groups.

Figure A1: Patch-wise normalization statistics for 35,154 LiDAR–DSM pairs from nine cities. The top row shows the scale $s=\operatorname{std}/0.25$ ; the bottom row shows the mean elevation $u$ . LiDAR is shown on the left and the corresponding DSM on the right. Colors identify cities, weighted equally so that cities with more patches do not dominate the histograms. Each panel displays the central 98% of values, with omitted extremes noted above the plot.

	Land Cover
Method	Dense Urban	Croplands	Sparse Built-Up	Roads	Vegetation
Patches excluded by the RMSE filter
Calibrated stereo DSM (CARS)	9.686	23.050	5.564	5.740	9.804
Ours—DSM only	6.179 $\pm$ 0.028	2.064 $\pm$ 0.019	4.776 $\pm$ 0.010	4.162 $\pm$ 0.015	6.849 $\pm$ 0.023
Ours—DSM + RGB	5.736 $\pm$ 0.032	2.240 $\pm$ 0.021	4.464 $\pm$ 0.017	3.743 $\pm$ 0.028	6.718 $\pm$ 0.035

Appendix

Table A6: RMSE on patches excluded by the P90 filter, calculated over all valid LiDAR/DSM pixels in each land-cover group. Values are in meters; model results are mean

\pm

standard deviation over 20 inference seeds.

	Land Cover
Method	Dense Urban	Croplands	Sparse Built-Up	Roads	Vegetation
In-context
Ours—DSM only	4.135 $\pm$ 0.038	1.179 $\pm$ 0.050	4.747 $\pm$ 0.056	3.720 $\pm$ 0.096	6.040 $\pm$ 0.029
Ours—DSM + RGB	3.576 $\pm$ 0.037	0.932 $\pm$ 0.023	3.801 $\pm$ 0.047	3.215 $\pm$ 0.083	5.362 $\pm$ 0.030
Held-out Bordeaux
Ours—DSM only	4.147 $\pm$ 0.083	3.856 $\pm$ 0.290	4.995 $\pm$ 0.029	3.309 $\pm$ 0.085	5.916 $\pm$ 0.030

Appendix

Table A7: RMSE within original DSM voids on the retained test sets. Errors are calculated over all evaluated void pixels in each land-cover group, excluding areas outside the source DSM image. Values are mean

\pm

standard deviation in meters over 20 inference seeds.

	Land Cover
Method	Dense Urban		Croplands		Sparse Built-Up		Roads		Vegetation
	MAE SD	RMSE SD	MAE SD	RMSE SD	MAE SD	RMSE SD	MAE SD	RMSE SD	MAE SD	RMSE SD
In-context
Ours—DSM only	0.005	0.006	0.003	0.007	0.003	0.005	0.011	0.024	0.002	0.004
Ours—DSM + RGB	0.007	0.016	0.003	0.005	0.004	0.006	0.008	0.017	0.003	0.007
Held-out Bordeaux

Appendix

Table A8: Standard deviations of the MAE and RMSE reported in Table 6 , measured across 20 inference seeds. Evaluation uses pixels with valid LiDAR and input DSM elevations. All values are in meters.

Figure A2: Signed bias, NMAD, and P95 absolute error by land cover for the in-context test ( left ) and Bordeaux ( right ). Only pixels with valid LiDAR and input DSM elevations are evaluated. Bias uses all evaluated pixels; NMAD and P95 are estimated by sampling 256 pixels per patch, giving each patch equal weight before grouping by land cover. All values are in meters.

Figure A3: Change in RMSE after excluding land-cover boundaries and patch edges. Bars show RMSE for the remaining interior pixels minus RMSE for all evaluated pixels; negative values mean lower interior error. The top row evaluates valid DSM pixels and the bottom row evaluates original DSM voids within the source image. Percentages indicate the fraction of pixels retained in each group. Errors are in meters.

Figure A4: Excluded patches with visible changes between RGB dates: Paris (2019/2025), Toulouse (2012/2022), Biarritz (2014/2025), and Arcachon (2021/2024). Rows show the calibrated DSM, conditioning RGB, later RGB, LiDAR, and DSM error. Within each column, DSM and LiDAR share the LiDAR elevation range; errors use a fixed $-10$ to $+10$ m scale. Pléiades ©CNES 2012/2014/2019/2021/2022/2024/2025, Distribution CNES PWH; LiDAR-HD ©IGN.

Panel of Figure 27 — Figure A5: Twenty-four patches drawn at random without replacement from the 6385 in-context test patches, without visual or error-based selection. Each panel contains eight examples, with rows showing the calibrated DSM, RGB, DSM + RGB prediction, and LiDAR. Within each column, all elevation maps share the LiDAR range. Pléiades ©CNES 2012–2024, Distribution CNES PWH; LiDAR-HD ©IGN.

Figure A6: Bordeaux examples where DSM + RGB increases RMSE relative to the calibrated input. The first two columns show the largest whole-patch increases; the last two show the largest increases within Dense Urban pixels. Selection uses seed-0 predictions and pixels with valid LiDAR and DSM elevations. Rows show the DSM, RGB, prediction, LiDAR, DSM error, and prediction error. Within each column, elevation maps share the LiDAR range; both error rows use a fixed $-10$ to $+10$ m scale. These are selected failure cases, not average results. Pléiades ©CNES 2016/2020/2025, Distribution CNES PWH; LiDAR-HD ©IGN.

Multimodal Diffusion Models

Multimodal diffusion models generate or interpret text, images, audio, video, motion, and other data by jointly denoising representations conditioned across modalities.

See all 175 papers

Guided Super-Resolution of Digital Elevation Models with Diffusion-Based Image Generators
Sep 10, 2026Armand Mihai Nicolicioiu, Dominik Narnhofer, Nando Metzger +3Diffusion Model Guidance Image Super-Resolution
Photogrammetry and Remote Sensing, ETH Zürich, Zürich, Switzerland · Remote Sensing Technology Institute, German Aerospace Center (DLR), Wessling, Germany
Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge
Sep 9, 2026Samed Doğan, Nico Leuze, Alfred Schöttl3D Semantic Segmentation Transfer Learning
Department of Electrical Engineering and Information Technology Munich University of Applied Sciences Munich, Bavaria 80335
LiDAR Resolution Recovery via Foundation-Model-Guided Diffusion
Oct 6, 2026Samed Doğan, Nico Leuze, Alfred SchöttlDiffusion Model Guidance Depth Completion
Dept. of Electrical Engineering and Information Technology Munich University of Applied Sciences, 80335 Munich, Germany

Sep 10, 2026J/K move · Enter open · S save

Guided Super-Resolution of Digital Elevation Models with Diffusion-Based Image Generators

Armand Mihai Nicolicioiu, Dominik Narnhofer, Nando Metzger, Daniel Panangian+2

Photogrammetry and Remote Sensing, ETH Zürich, Zürich, Switzerland · Remote Sensing Technology Institute, German Aerospace Center (DLR), Wessling, Germany

Diffusion Model Guidance Image Super-Resolution

Open paper AskPDF arXiv

High-resolution digital surface models (DSMs) play an important role in urban analysis, 3D building reconstruction, and infrastructure monitoring, yet their availability remains limited due to the high cost and complexity of data acquisition. In contrast, coarse DSMs from commercial satellite missions are widely accessible, and high-resolution optical imagery is increasingly available from aerial and satellite platforms. We address the resulting mismatch in spatial resolution and propose a DSM superresolution approach that enhances 5 m DSMs to 0.5 m resolution, using guidance from high-resolution spectral images. Our method employs denoising diffusion to transfer information that is visible only in the image, like crisp outlines and detailed roof structures, into the elevation maps. In this way, surface details are reconstructed more accurately than with conventional interpolation or filtering techniques. Experiments on several cities in Central Europe demonstrate that the proposed approach produces high-quality DSMs with improved structural detail and accurate surface geometry. Our results highlight the potential of guided super-resolution with foundational image priors as a means of reconstructing high-resolution surface models.

Abstract

Multimodal Diffusion Models

Explore similar work

Guided Super-Resolution of Digital Elevation Models with Diffusion-Based Image Generators

Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge

LiDAR Resolution Recovery via Foundation-Model-Guided Diffusion