Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack

Fabio Montello; Ronja Güldenring; Lucas Mogensen Sandby; Christoffer Koo Øhrstrøm; Lazaros Nalpantidis

cs.CVNov 2, 2026

Do Pretrained Backbones Transfer to Physical Sensors? A Systematic Study of Multimodal Initialization Across the Sensor Stack

Authors: Fabio Montello, Ronja Güldenring, Lucas Mogensen Sandby, Christoffer Koo Øhrstrøm, Lazaros Nalpantidis

Organizations: Technical University of Denmark

Abstract

Transfer learning from pretrained backbones has become standard practice in multi-sensor computer vision---yet two questions remain open: which pretrained weights transfer best to multi-sensor setups, and can in-domain pretraining outperform off-the-shelf solutions? Frameworks like MultiMAE demonstrate the utility of masked autoencoding in a multi-modal context, but rely entirely on dense, model-derived modalities. It is unclear how robustly these methods translate to real-world applications that use physical sensors, where the aggregation of diverse public datasets for large-scale pretraining inherently introduces modality sparsity. To investigate this systematically, we curate a multi-sensor autonomous driving collection of $2.82M$ images from 16 publicly available datasets, incorporating both physical hardware modalities (RGB, LWIR, NIR, Event) and model-derived modalities (Stereo Depth, Segmentation). We use this collection for in-domain pretraining based on the MultiMAE approach. We adapt MultiMAE's loss formulation to accommodate sparse samples, and study the effect of an enforced encoder bottleneck for modality alignment and of additional losses specific to dense reconstruction (SSIM and FFT). We then systematically compare against common pretrained backbones (ViT, MAE, DINO and MultiMAE) across downstream sensor fusion tasks. We find that MultiMAE is a surprisingly strong out-of-the-box baseline, transferring effectively even to modalities it has never seen, while our targeted modifications yield incremental but consistent improvements on multi-sensor transfer and overall robustness.

Figures & tables

Figure 1: Multimodal pretraining under sparse availability of sensor modalities. Reconstruction examples from our multi-sensor pretraining pipeline. The rows display the heavily masked sparse inputs (top), the dense model predictions (middle), and the ground-truth targets (bottom) across physically collected hardware sensors (RGB, NIR, LWIR, Event) and model-derived modalities (Depth, Segmentation). Empty panels denote missing modalities due to dataset sparsity (no sample contains all modalities).

Dataset

Samples Split

Resolution

Modalities

Total

Unique

Training

Val

Test

RGB

Depth

Event

LWIR

NIR

Segmentation

Samples

BDD100k [ 41 ]

140,000

20,000

40,000

640\times 720

✓

400,000

200,000

Cityscapes [ 12 ]

91,716

2,000

6,100

640\times 775

✓

299,448

99,816

DSEC [ 16 , 17 ]

10,350

211

2,245

600\times 430

✓

51,224

12,806

ECMD [ 8 ]

52,424

1,036

640\times 480

✓

267,300

53,460

FMB [ 28 ]

1,202

280

800\times 600

✓

4,500

1,500

Table 1: Our Multi-Sensor Pretraining Collection. Overview of the 16 datasets used and the modalities included in our pretraining collection. The Table details the sample distributions across training, validation, and test splits, alongside the standardized spatial resolutions and the sensor modalities present in each dataset.

Figure 2: Overview of our modified MultiMAE architecture designed for sparse physical sensors. Independent tokenizers process each available sensor modality before a fixed token budget is sent to a shared ViT encoder. Instead of relying on decoder-level cross-attention, an MLP bottleneck aligns and concatenates active sensor tokens along the channel dimension (zero-filling missing ones) to produce a unified spatial representation before decoding.

	C	LF	DF	S	C	LF	DF	S
	RGB				NIR
ViT	61.91	63.58	54.98	60.43	64.98	65.09	57.21	63.00
MAE	65.79	68.15	59.11	64.92	69.30	67.60	60.85	67.35
DINO	65.25	70.93	60.27	65.45	68.83	68.94	61.59	66.43
MultiMAE	70.41	70.98	66.91	68.42	71.72	71.59	69.24	68.61
Our-ViT	69.43	68.94	61.15	65.54	70.81	70.52	65.12	68.42

Table 2: Evaluation Results on STF. We report 2D bounding box AP R40 (%) using the standard KITTI evaluation protocol. The reported scores represent the average across all object classes for the easy, moderate, and hard difficulty levels. C = Clear; LF = Light Fog; DF = Dense Fog; S = Snow/Rain. Bold indicates the best result. Underline indicates the second best. This is a summarized version of Table 9 in Appendix.

	Seg.	Det.		Seg.	Det.
	mIoU	$\textbf{mAP}_{\mathbf{50:95}}$	$\textbf{mAP}_{\mathbf{50}}$	mIoU	$\textbf{mAP}_{\mathbf{50:95}}$	$\textbf{mAP}_{\mathbf{50}}$
	RGB			Event
ViT	54.45	35.50	63.48	42.79	28.20	49.86
MAE	55.71	40.80	71.70	43.46	33.89	57.33
DINO	54.29	37.37	66.66	43.49	32.30	55.26
MultiMAE	62.61	46.46	77.11	50.05	36.88	60.78

Table 3: Evaluation Results on DSEC. We report mean Intersection over Union (mIoU) for semantic segmentation and average precision thresholds (

mAP_{50:95}

mAP_{50}

) for object detection across isolated and fused modalities. Bold indicates the best result. Underline indicates the second best.

		STF		LLVIP
ID	Pretraining Setting	RGB + NIR + Depth + Radar	NIR + Depth + Radar	RGB + LWIR	LWIR
A0	Baseline ( Rand. init, MSE, Sample balancing, cross-attention, no semantic. )	35.77 $\pm$ 0.12	30.78 $\pm$ 0.34	62.03 $\pm$ 0.43	61.81 $\pm$ 0.20
Initialization Weights
A1	ViT weights init.	38.62 $\pm$ 0.30	35.66 $\pm$ 0.37	62.56 $\pm$ 0.46	62.94 $\pm$ 0.21
A2	MultiMAE weights init.	39.01 $\pm$ 0.18	36.67 $\pm$ 0.62	63.68 $\pm$ 0.46	62.99 $\pm$ 0.41
Reconstruction Losses

Table 4: Ablation studies on different pretraining settings. We report mAP (mean

\pm

std over 3 seeds) on downstream tasks for the STF and LLVIP datasets across various modality configurations. Bold indicates the selected configuration for the final pretrain.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Value
Optimization
Optimizer	AdamW
Base learning rate	2e-4
Weight decay	0.05
Optimizer momentum	$\beta_{1}=0.9,\beta_{2}=0.95$
Learning rate schedule	Cosine decay

Appendix

Table 5: Pretraining hyperparameters. These settings apply to the primary Base model variant.

Configuration	Semantic Segmentation	Object Detection
Optimization
Optimizer	AdamW	AdamW
Base learning rate	1e-3 2e-4 (DSEC, FMB MM, MFNet)	1e-4
Backbone learning rate	2e-4 1e-4 (DSEC, FMB MM, MFNet)	1e-5
Weight decay	0.05	0.1
Optimizer momentum	$\beta_{1}=0.9,\beta_{2}=0.999$	$\beta_{1}=0.9,\beta_{2}=0.999$

Appendix

Table 6: Fine-tuning hyperparameters for semantic segmentation and object detection tasks.

Dataset	Task	Modality	Metric	ViT	MAE	Dino	MultiMAE	Our-ViT	Our-MM
Cityscapes	Sem. Seg.	RGB	mIoU	70.26	65.74	69.45	73.88	66.74	70.75
BDD100K	Sem. Seg.	RGB	mIoU	53.01	50.31	52.33	54.95	52.41	53.34
LLVIP	Obj. Det.	RGB	$\text{mAP}_{50:95}$	46.15	48.11	45.34	47.97	45.53	49.04
			$\text{mAP}_{50}$	86.01	86.91	83.97	86.79	84.00	86.83
		LWIR	$\text{mAP}_{50:95}$	54.89	60.49	57.10	60.33	57.01	59.52
			$\text{mAP}_{50}$	91.98	95.74	93.76	94.67	92.00	93.91

Appendix

Table 7: Quantitative evaluation across single-modal and multimodal downstream tasks. Performance is measured via mean Intersection over Union (mIoU) for semantic segmentation on Cityscapes, BDD100K, FMB, and MFNet, and via Average Precision (

mAP_{50:95}

and

mAP_{50}

) for object detection on LLVIP and KITTI. Best results are in bold , and second best are underlined .

	C	LF	DF	S	C	LF	DF	S
	RGB				NIR
MultiMAE	69.99 ±0.56	70.98 ±0.51	67.08 ±0.72	68.46 ±0.06	71.86 ±0.21	71.19 ±0.61	68.49 ±1.23	68.78 ±0.27
Our-ViT	69.20 ±0.73	69.48 ±0.99	62.91 ±1.75	65.44 ±1.12	70.53 ±0.28	70.47 ±0.20	65.53 ±1.33	68.46 ±0.19
Our-MM	70.94 ±0.59	71.62 ±0.96	68.32 ±1.24	67.98 ±0.08	72.22 ±0.51	73.43 ±0.16	69.09 ±0.32	69.47 ±0.36
	Depth				RGB+NIR
MultiMAE	44.57 ±0.32	42.09 ±0.69	18.13 ±0.60	39.25 ±0.29	75.67 ±0.69	77.56 ±0.98	72.30 ±1.11	75.01 ±0.44

Appendix

Table 8: Multi-seed Evaluation Results on STF. 2D bounding box AP R40 (%) using the standard KITTI evaluation protocol, averaged across all object classes for the easy, moderate, and hard difficulty levels. We report mean

\pm

std over seeds 42, 43 and 44. C = Clear; LF = Light Fog; DF = Dense Fog; S = Snow/Rain. Bold indicates the best result. Underline indicates the second best.

Modality	Model	Clear			Light Fog			Dense Fog			Snow/Rain
	Model	E	M	H	E	M	H	E	M	H	E	M	H
RGB	ViT	68.50	60.55	56.69	68.92	62.33	59.49	59.38	54.03	51.52	67.29	58.72	55.29
	MAE	72.05	64.47	60.84	72.79	67.23	64.42	63.32	58.11	55.90	71.35	63.44	59.98
	DINO	71.38	64.12	60.25	75.48	70.10	67.21	64.68	59.37	56.77	72.22	63.92	60.20
	MultiMAE	76.00	69.39	65.85	74.59	70.43	67.91	70.47	66.22	64.05	74.10	66.99	64.16
	Our-ViT	75.02	68.30	64.96	72.71	68.32	65.80	64.77	60.15	58.54	71.00	64.37	61.26

Appendix

Table 9: Average Precision (AP) at 40 recall points (R40) averaged across all object classes, evaluated under four weather conditions: clear, light fog, dense fog, and snow/rain. Each condition reports three difficulty levels — E (Easy), M (Moderate), and H (Hard) — defined by the KITTI benchmark based on object size, occlusion level, and truncation degree, with Hard being the most challenging. Results are reported for single-modality and multi-modality input configurations across six backbone/pretraining methods: ViT (vanilla Vision Transformer), MAE (Masked Autoencoder), DINO (self-distillation with no labels), MultiMAE (multi-modal MAE baseline), and our proposed models Our-ViT and Our-MM. Best results per condition and difficulty are in bold , second best are underlined .

Modality	Model	Clear			Light Fog			Dense Fog			Snow/Rain
	Model	E	M	H	E	M	H	E	M	H	E	M	H
RGB - LiDAR	ViT	71.60	63.54	59.47	69.33	63.90	61.21	60.44	55.61	52.85	68.19	59.91	56.27
	MAE	73.41	66.31	62.63	72.66	67.50	64.53	64.26	59.51	57.09	72.01	63.94	60.52
	DINO	72.76	65.24	61.28	72.17	66.81	64.03	58.96	53.80	51.28	70.56	62.60	59.00
	MultiMAE	75.50	68.59	65.35	74.24	69.86	66.89	70.45	66.16	63.94	75.01	68.34	65.11
	Our-ViT	75.71	68.89	65.45	74.43	69.84	66.79	67.96	63.86	61.44	72.40	65.99	62.44

Appendix

(Continued)

	RGB			Depth			Event
Model	mIoU	$\text{AP}_{50:95}$	$\text{AP}_{50}$	mIoU	$\text{AP}_{50:95}$	$\text{AP}_{50}$	mIoU	$\text{AP}_{50:95}$	$\text{AP}_{50}$
ViT	18.40	17.59	35.70	1.42	0.42	0.50	4.99	0.80	1.26
MAE	33.67	21.05	40.92	9.72	0.00	0.00	5.67	1.15	1.83
DINO	27.98	17.29	31.41	9.22	0.42	0.50	3.02	1.05	1.65
MultiMAE	54.23	29.90	58.74	2.04	2.53	4.21	21.33	2.28	3.82
Our-ViT	55.92	34.63	64.63	2.58	9.81	15.52	11.17	2.21	4.13

Appendix

Table 10: Sensor Dropout Robustness on DSEC. We report mIoU (%) for semantic segmentation and

\text{mAP}_{50:95}

\text{mAP}_{50}

(%) for object detection when the model receives only the listed subset of modalities at inference time, simulating sensor dropout or failure. Bold indicates the best result per column; underline indicates the second best.

	C	LF	DF	S	C	LF	DF	S
	RGB				NIR
Base	69.43	68.94	61.15	65.54	70.81	70.52	65.12	68.42
Small	66.68	69.31	63.46	65.17	69.10	67.15	63.53	67.46
Tiny	65.22	65.53	60.63	62.59	67.00	68.39	58.66	65.35
	Depth				RGB+NIR
Base	46.68	41.48	19.60	40.98	75.08	76.77	69.71	73.92

Appendix

Table 11: Model Scale Comparison on STF. We report 2D bounding box AP R40 (%) using the standard KITTI evaluation protocol for our ViT model at Base , Small , and Tiny scales. The reported scores represent the average across all object classes for the easy, moderate, and hard difficulty levels. C = Clear; LF = Light Fog; DF = Dense Fog; S = Snow/Rain.

Dataset	License / Terms
BDD100K [ 41 ]	UC Berkeley custom (BDD100K License)
Cityscapes [ 12 ]	Custom non-commercial license
DSEC [ 16 , 17 ]	CC BY-SA 4.0
ECMD [ 8 ]	Not specified
FMB [ 28 ]	Not specified
Freiburg Thermal [ 38 ]	Custom (non-commercial)

Appendix

Table 12: Licensing terms of the 16 constituent datasets in our pretraining collection. Of the datasets marked “Not specified” we could not find a published license or terms-of-use document accompanying the public release.

Panel of Figure 3 — Figure 3: Extended qualitative cross-modal reconstruction results across ten distinct driving scenarios from the pretraining corpus. The first row represents the input, second row shows the model output, while the last row shows the target. For each image, from left to right: RGB, NIR, LWIR, Depth, Event, Segmentation. Grey images represent unavailable input/target modalities.

Figure 4: Convergence analysis across datasets. Comparison of learning curves between Our-MM (blue), Our-ViT (red) and MultiMAE (green) across different benchmarks: DSEC, LLVIP, KITTI for Object Detection and FMB for Semantic Segmentation. Our in-domain pretrained weights present steeper learning curves at early training steps, providing a stronger starting point for adapting to new sensor modalities. The performance gap narrows as training progresses, which suggests that once MultiMAE catches up to the target modality, it is able to use its strong cross attention capabilities.

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 188 papers

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens
Mar 25, 2026Ciem Cornelissen, Sam Leroux, Pieter SimoensMultimodal SSL Multimodal Learning
IDLab, Department of Information Technology, Ghent University - imec, Belgium
OctoSense: Self-Supervised Learning for Multimodal Robot Perception
Jun 25, 2026Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2Autonomous Driving Datasets Masked Autoencoders
1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University
CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications
Aug 16, 2026Yubo Zhang, Yiyao LiuCross-Modal Representation Learning Self-Supervised Pre-Training

Mar 25, 2026J/K move · Enter open · S save

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Ciem Cornelissen, Sam Leroux, Pieter Simoens

IDLab, Department of Information Technology, Ghent University - imec, Belgium

Multimodal SSL Multimodal Learning Multimodal Fusion Joint-Embedding Predictive Architecture Autonomous Driving Perception

Open paper AskPDF arXiv

Self-supervised learning has emerged as a powerful paradigm for learning visual representations without manual annotations, yet most methods still operate on a single modality and therefore miss the complementary structure available from heterogeneous sensors. We present Le MuMo JEPA, a self-supervised framework that learns unified representations from RGB images and aligned companion modalities. In our driving experiments, the second modality is camera-aligned LiDAR depth; we also evaluate RGB-thermal training and transfer on the Teledyne FLIR ADAS benchmark. Our approach extends LeJEPA to the multi-modal setting by learning fusion tokens that act as a latent bottleneck between modality-specific patch stems inside a shared transformer. Our default model employs a pruned fusion strategy: after an initial cross-modal attention layer, modality-specific tokens are dropped, forcing cross-modal information into the shared fusion-token grid as an efficient latent bottleneck before Sketched Isotropic Gaussian Regularization (SIGReg) is applied to the joint multimodal CLS embedding. On Waymo, Le MuMo JEPA gives the strongest performance-efficiency trade-off on downstream patch probes among the from-scratch multimodal baselines, improving CenterNet detection and dense depth while remaining competitive on segmentation. Under from-scratch training on nuScenes, Le MuMo JEPA remains the strongest model, and it also gives the best FLIR results, especially after Waymo-initialized fine-tuning. It also retains the best overall accuracy-efficiency balance in our study at substantially lower compute, memory, and estimated training time.

Abstract

Multimodal Pretraining

Explore similar work

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications