DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision

Carlo Sgaravatti; Daniele Kopyshevskiy; Davide Secco; Giacomo Boracchi

cs.CVNov 3, 2026

DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision

Authors: Carlo Sgaravatti, Daniele Kopyshevskiy, Davide Secco, Giacomo Boracchi

Organizations: Politecnico di Milano · Sorbonne University · Universit`a Bocconi

Abstract

Multimodal perception is central to many computer vision applications, including robotics and autonomous driving, where different sensing modalities provide complementary cues. Self-supervised learning offers a scalable solution for exploiting large amounts of unlabeled multimodal data. However, existing methods either neglect modality-specific information by enforcing feature-space alignment between modalities or learn only global representations without preserving spatial structure. We propose DenseTwins, a multimodal self-supervised framework that learns dense representations by explicitly decoupling the dense feature space into shared and modality-specific components. The proposed approach combines dense cross-modal correspondences with redundancy-reduction objectives, enabling the model to capture both shared semantics and modality-specific cues within a structured dense embedding space. DenseTwins jointly optimizes an intra-modal objective across augmented views and a cross-modal objective that aligns shared features while decorrelating modality-specific information. We evaluate DenseTwins across multiple multimodal benchmarks and sensing setups, achieving state-of-the-art or competitive performance across multimodal self-supervised baselines. Code available at: https://github.com/CarloSgaravatti/DenseTwins.

Figures & tables

Figure 1: The structure of the dense embeddings of DenseTwins. Two separate encoders $\mathcal{F}_{1}$ , $\mathcal{F}_{2}$ process the two input modalities $\mathbf{X}_{1},\mathbf{X}_{2}$ separately, producing two feature maps $\mathbf{F}_{1},\mathbf{F}_{2}$ . Finally, two projection heads project $\mathbf{F}_{1},\mathbf{F}_{2}$ into a common latent space $\mathbf{Z}_{1},\mathbf{Z}_{2}$ , each divided into modality-shared $\mathbf{S}_{1},\mathbf{S}_{2}$ and modality-specific $\mathbf{U}_{1},\mathbf{U}_{2}$ components.

Figure 2: DenseTwins architecture . For each modality $m\in\{1,2\}$ , two augmented views are processed by a modality-specific backbone $\mathcal{F}_{m}$ , followed by a projection head $\mathcal{P}_{m}$ that produces both global and dense embeddings. The embedding space is explicitly partitioned into a shared subspace (red) and a modality-specific subspace (blue). Training is based on dense correspondences between feature vectors within each modality and across modalities computed using cosine similarities. The shared components are aligned through redundancy reduction and invariance objectives, while modality-specific components are decorrelated across modalities to preserve complementary information. Intra-modal learning, instead, applies a redundancy-reduction objective, encouraging invariance between augmented views. The final objective combines intra-modal and cross-modal losses applied to both global and dense representations.

Matrix	Input pairs	Section
$\bar{I}^{(m)}$	global intra-modal embeddings	Sec. 4.1
$I^{(m)}$	dense intra-modal correspondences	Sec. 4.1
$\bar{C}$	global cross-modal embeddings	Sec. 4.2
$C$	dense cross-modal correspondences	Sec. 4.2

Table 1: Cross-correlation matrices in DenseTwins.

	SUN-RGBD mIoU		EventScape mIoU
Pretrain	10% labels	100% labels	100% labels
None	35.7	49.7	61.9
SimCLR	37.6	50.8	63.2
CLIP	37.6	50.6	63.5
BarlowTwins	37.8	51.0	63.0
VICReg	37.7	51.0	62.4

Table 2: SUN-RGBD and EventScape (CMX fine-tuning).

Pretrain	kNN	LP	FCN
SimCLR	22.4	25.1	30.4
CLIP	20.3	22.3	29.9
Barlow Twins	18.5	20.2	27.0
VICReg	18.6	22.7	28.0
DeCUR	22.0	23.7	35.8
DINO	25.6	31.0	37.6

Table 3: SUN-RGBD frozen probing (NYU-13 class label set).

		KAIST AP50		DSEC AP50
Pretrain	Backbone	All	Reasonable	All	Car	Pedestrian
DenseTwins	Swin-T	33.3	35.3	24.3	51.1	11.9
DenseTwins	ConvNeXtV2-T	31.2	32.9	21.3	47.1	9.7
DenseTwins	ViT-S/16	26.8	29.8	20.6	46.0	9.3
SimCLR	ViT-S/16	16.8	19.2	8.4	36.7	6.8
CLIP	ViT-S/16	16.5	18.8	8.2	31.0	5.1

Table 4: Object detection AP50 on KAIST and DSEC. Baselines use ViT-S/16; DenseTwins also uses Swin-T and ConvNeXtV2-T.

	GeoNRW (fine-tuned FCN)		GeoNRW (frozen probe)			BigEarthNet
Pretrain	1% mIoU	100% mIoU	kNN	LP	FCN	mAP
Random	26.2	35.4	16.1	19.5	30.1	0.601
SimCLR	40.6	55.8	22.5	27.9	42.9	0.809
CLIP	40.8	55.2	23.8	28.9	42.6	0.819
BarlowTwins	40.8	52.1	19.2	23.7	41.8	0.797
VICReg	41.0	56.3	24.5	27.9	43.9	0.803

Table 5: GeoNRW (fine-tuned FCN and frozen probing) and BigEarthNet (frozen-probing mAP).

	PMF	MinkUNet+ViT
Pretrain	mIoU	FOV mIoU	Full mIoU
Random	63.9	31.1	31.8
DeCUR	65.0	44.8	44.4
DenseTwins	65.2	47.1	44.1

Table 6: SemanticKITTI: PMF (fine-tuned) and MinkUNet+ViT (frozen probing, FOV/Full mIoU).

Table 9

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Subspace	Recall@1	Recall@5
Shared	0.62	0.85
Specific	0.06	0.13

Appendix

Table 9: Cross-modal patch retrieval using different subspaces.

Figure 5: t-SNE visualization of global embeddings extracted from the shared and modality-specific subspaces. The shared subspace exhibits stronger cross-modal alignment, whereas the modality-specific subspace remains separated according to modality.

Stage	ms	% forward	% step
Batch size 8/GPU
Backbones forward	69.9	56.8	17.7
Projection heads	9.3	7.6	2.4
Dense Correspondences	3.6	2.9	0.9
BT correlation and loss	41.0	33.4	10.4
Total forward	123.0	100.0	31.2

Appendix

Table 10: Per-step time breakdown of a DenseTwins training step (single A100 GPU) at two per-GPU batch sizes.

Configuration	Batch size/GPU	Training time
w/o projection heads	28	5.1 h
w/ projection heads	22	6.4 h

Appendix

Table 11: Effect of the projection heads on DSEC pretraining using

8\times

A100 GPUs. At a fixed memory budget, removing them allows a larger per-GPU batch size and reduces training time.

	RGB		Depth		Concat
Method	LP	FCN	LP	FCN	LP	FCN
DenseTwins	17.2	26.7	27.8	32.5	33.7	40.6
I-JEPA	19.6	27.5	27.2	32.0	31.1	35.2
DINO	15.0	22.8	26.4	33.9	31.0	37.6
iBOT	12.7	19.7	24.8	31.6	30.8	36.3
LeJEPA	9.5	19.1	17.6	30.0	23.9	32.7

Appendix

Table 12: Linear probing (LP) and FCN performance (mIoU) on SUN-RGBD when only RGB or only Depth is available at test time, compared to the multimodal (Concat) setting obtained via late fusion.

Misalignment	Recall@1	Recall@5
None (aligned)	0.62	0.85
Shift 16px (1 patch)	0.60	0.84
Shift 32px (2 patches)	0.61	0.82

Appendix

Table 13: Cross-modal patch retrieval recall on SUN-RGBD under synthetic misalignment (translation applied to one modality) using the shared subspace.

Figure 6: Qualitative Results on SUN-RGBD . From left to right we show the RGB image, the depth modality encoded as HHA, the ground-truth semantic segmentation, and the predictions obtained using DeCUR and DenseTwins pretraining. While both methods correctly segment the main scene structures, DenseTwins produces sharper object boundaries and more consistent predictions.

Figure 7: Qualitative Results on Eventscape . From left to right, we show the RGB frame, the event representation, the ground-truth segmentation, and the predictions obtained using DeCUR and DenseTwins. Highlighted regions show typical failure cases of DeCUR (red boxes), such as missed distant objects or fragmented thin structures, while DenseTwins provides more complete predictions and improved detection of small or distant regions (green boxes).

Multimodal SSL

See all 27 papers

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens
Mar 25, 2026Ciem Cornelissen, Sam Leroux, Pieter SimoensMultimodal SSL Multimodal Learning
IDLab, Department of Information Technology, Ghent University - imec, Belgium
Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning
Nov 5, 2026Liangjian Wen, Jiangtao Zheng, Jiang Duan +8Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd. +8
OctoSense: Self-Supervised Learning for Multimodal Robot Perception
Jun 25, 2026Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2Autonomous Driving Datasets Masked Autoencoders
1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University

Mar 25, 2026J/K move · Enter open · S save

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Ciem Cornelissen, Sam Leroux, Pieter Simoens

IDLab, Department of Information Technology, Ghent University - imec, Belgium

Multimodal SSL Multimodal Learning Multimodal Fusion Joint-Embedding Predictive Architecture Autonomous Driving Perception

Open paper AskPDF arXiv

Self-supervised learning has emerged as a powerful paradigm for learning visual representations without manual annotations, yet most methods still operate on a single modality and therefore miss the complementary structure available from heterogeneous sensors. We present Le MuMo JEPA, a self-supervised framework that learns unified representations from RGB images and aligned companion modalities. In our driving experiments, the second modality is camera-aligned LiDAR depth; we also evaluate RGB-thermal training and transfer on the Teledyne FLIR ADAS benchmark. Our approach extends LeJEPA to the multi-modal setting by learning fusion tokens that act as a latent bottleneck between modality-specific patch stems inside a shared transformer. Our default model employs a pruned fusion strategy: after an initial cross-modal attention layer, modality-specific tokens are dropped, forcing cross-modal information into the shared fusion-token grid as an efficient latent bottleneck before Sketched Isotropic Gaussian Regularization (SIGReg) is applied to the joint multimodal CLS embedding. On Waymo, Le MuMo JEPA gives the strongest performance-efficiency trade-off on downstream patch probes among the from-scratch multimodal baselines, improving CenterNet detection and dense depth while remaining competitive on segmentation. Under from-scratch training on nuScenes, Le MuMo JEPA remains the strongest model, and it also gives the best FLIR results, especially after Waymo-initialized fine-tuning. It also retains the best overall accuracy-efficiency balance in our study at substantially lower compute, memory, and estimated training time.

Abstract

Multimodal SSL

Explore similar work

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning

OctoSense: Self-Supervised Learning for Multimodal Robot Perception