Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Ciem Cornelissen; Sam Leroux; Pieter Simoens

cs.CVMar 25, 2026

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Authors: Ciem Cornelissen, Sam Leroux, Pieter Simoens

Organizations: IDLab, Department of Information Technology, Ghent University - imec, Belgium

Abstract

Self-supervised learning has emerged as a powerful paradigm for learning visual representations without manual annotations, yet most methods still operate on a single modality and therefore miss the complementary structure available from heterogeneous sensors. We present Le MuMo JEPA, a self-supervised framework that learns unified representations from RGB images and aligned companion modalities. In our driving experiments, the second modality is camera-aligned LiDAR depth; we also evaluate RGB-thermal training and transfer on the Teledyne FLIR ADAS benchmark. Our approach extends LeJEPA to the multi-modal setting by learning fusion tokens that act as a latent bottleneck between modality-specific patch stems inside a shared transformer. Our default model employs a pruned fusion strategy: after an initial cross-modal attention layer, modality-specific tokens are dropped, forcing cross-modal information into the shared fusion-token grid as an efficient latent bottleneck before Sketched Isotropic Gaussian Regularization (SIGReg) is applied to the joint multimodal CLS embedding. On Waymo, Le MuMo JEPA gives the strongest performance-efficiency trade-off on downstream patch probes among the from-scratch multimodal baselines, improving CenterNet detection and dense depth while remaining competitive on segmentation. Under from-scratch training on nuScenes, Le MuMo JEPA remains the strongest model, and it also gives the best FLIR results, especially after Waymo-initialized fine-tuning. It also retains the best overall accuracy-efficiency balance in our study at substantially lower compute, memory, and estimated training time.

Figures & tables

Figure 1: Overview of Le MuMo JEPA. The companion modality is represented as a spatially aligned signal and fused with RGB through learnable fusion tokens , which act as a latent bottleneck inside a shared transformer. The default training objective applies SIGReg to the joint multimodal CLS embedding.

Figure 2: Waymo dataset showcase. Example synchronized supervision used in our experiments: (a) RGB image, (b) camera-view segmentation, (c) the aligned companion-modality signal shown in depth form for the driving setting, and (d) projected 3D bounding boxes.

Method	mAP XY $\uparrow$	Depth MAE $\downarrow$	Seg. mIoU $\uparrow$
LeJEPA [ 4 ]	19.3	4.704	0.261
LeJEPA encoder frozen [ 4 ]	13.8	6.734	0.126
LiDAR-only	15.4	2.982	0.151
LiDAR-only encoder frozen	8.0	5.397	0.089

Table 1: Self-supervised single-modality encoders vs. encoder-frozen controls on Waymo. All rows train the probes. Rows labeled “encoder frozen” keep a randomly initialized encoder fixed, whereas the other rows train the encoder self-supervised before frozen-probe evaluation. We report XY mAP, Depth MAE, and Seg. mIoU.

Method	mAP XY $\uparrow$	Depth MAE $\downarrow$	Seg. mIoU $\uparrow$
Early Fusion RGBD	18.1	4.767	0.248
Late Fusion	18.7	4.802	0.251
FT-Pruned + VICReg [ 7 ]	22.8	2.911	0.248
FT-Persistent + SIGReg	23.1	2.846	0.271
FT-Pruned + SIGReg (3-pass)	23.2	2.777	0.274
Le MuMo JEPA	23.6	2.860	0.275

Table 2: Focused fusion comparison on Waymo for the fusion variants. We compare the default Le MuMo JEPA setting against early and late fusion, persistent routing, VICReg, and the 3-pass SIGReg variant.

Method	Training Data	mAP XY $\uparrow$	Depth MAE $\downarrow$	mAP XZ $\uparrow$	Seg. mIoU $\uparrow$
LeJEPA [ 4 ]	RGB	19.3	4.704	4.9	0.261
DINOv3 [ 22 ]	RGB	15.2	5.314	3.5	0.239
LiDAR-only	Depth	15.4	2.982	4.0	0.151
MultiMAE-SS [ 2 ]	RGB+Depth	13.5	4.441	2.7	0.221
MultiMAE-MT [ 2 ]	RGB+Depth	13.7	3.583	2.9	0.262
ImageBind [ 13 ]	RGB+Depth	13.4	4.309	3.0	0.243

Table 3: Patch-probe comparison on Waymo with all encoders trained from scratch. CenterNet and dense prediction heads are trained on frozen patch embeddings taken from models that were all self-supervised from scratch on Waymo. “Training Data” denotes the modalities seen during SSL pretraining; for dual-stream baselines, the frozen patch probes use the camera-aligned RGB patch readout described in Section 4.1 . Best results are in bold ; second best are underlined .

\uparrow

means higher is better and

\downarrow

means lower is better.

Method	mAP XY $\uparrow$	mAP XZ $\uparrow$	Seg. mIoU $\uparrow$	Depth MAE $\downarrow$
MultiMAE-SS [ 2 ]	6.95	1.66	0.195	5.736
MultiMAE-MT [ 2 ]	6.67	1.62	0.192	5.624
ImageBind [ 13 ]	6.86	1.57	0.198	5.912
Le MuMo JEPA	9.52	2.53	0.228	2.031

Table 4: Results on the nuScenes dataset. We report CenterNet XY-match mAP, XZ-match mAP, semantic segmentation, and Depth MAE.

Setting	Method	mAP50 $\uparrow$	Car mAP50 $\uparrow$
Scratch	MultiMAE-SS [ 2 ]	0.51	3.86
Scratch	ImageBind [ 13 ]	0.24	1.90
Scratch	Le MuMo JEPA	0.41	3.00
Waymo $\rightarrow$ FLIR	MultiMAE-SS [ 2 ]	0.54	4.21
Waymo $\rightarrow$ FLIR	MultiMAE-MT [ 2 ]	0.69	5.26
Waymo $\rightarrow$ FLIR	ImageBind [ 13 ]	0.72	5.49

Table 5: FLIR results across training regimes. We report 2D CenterNet mAP50 and Car mAP50. Rows are ordered from training from scratch on FLIR, to Waymo

\rightarrow

FLIR transfer, to Waymo-pretrained fine-tuning on FLIR. Because FLIR does not provide segmentation labels, the scratch block excludes MultiMAE-MT.

Method	Time (min) $\downarrow$	SSL FLOPs (PFLOPs) $\downarrow$	VRAM (GB) $\downarrow$
MultiMAE-SS [ 2 ]	14.07	21.4	2.94
Early Fusion RGBD	18.70	72.9	6.15
LeJEPA [ 4 ]	18.73	72.7	6.01
ImageBind [ 13 ]	23.65	80.4	9.49
Late Fusion	34.72	144.9	11.41
DINOv3 [ 22 ]	37.72	108.8	8.70

Table 6: Encoder-side compute profile. Rows are sorted by estimated total encoder training time. We report time in minutes, total encoder SSL FLOPs in PFLOPs, and peak reserved VRAM in GB. Time is estimated for the encoder-side workload on an H200 system with an AMD EPYC 9275F 24-Core Processor. The fusion-token rows use the default single-pass joint-CLS objective, with the 3-pass SIGReg variant shown separately as an ablation.

Figure 3: Waymo patch-embedding visualization. t-SNE projections of final-layer patch embeddings are shown with class-oriented structure and depth-oriented structure, illustrating how the learned patch space organizes both semantic grouping and geometric variation across methods. The left column uses Le MuMo JEPA fusion-token embeddings, and the right column uses LeJEPA patch embeddings. A planar fit to the plotted depth gradient gives an $R^{2}$ score of $0.463$ for Le MuMo JEPA versus $0.086$ for LeJEPA.

Figure 4: Waymo qualitative probe output. The figure shows the RGB input together with predictions from the three probe families used in the paper: dense depth estimation, segmentation, and 3D detection boxes. It provides a direct qualitative view of the same patch-level capabilities summarized by the quantitative probe tables.

Multimodal SSL

See all 14 papers

OctoSense: Self-Supervised Learning for Multimodal Robot Perception
Jun 25, 2026Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2Autonomous Driving Datasets Masked Autoencoders
1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Oct 11, 2026Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu +4Multimodal Learning Multimodal Sensor Fusion
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Candour Ltd, Oulu, Finland
V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
Mar 15, 2026Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar +6Self-Supervised Visual Representation Learning Visual Representation Learning
FAIR at Meta · Universidad de Zaragoza

Jun 25, 2026J/K move · Enter open · S save

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero+1

1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University

Autonomous Driving Datasets Masked Autoencoders Multimodal SSL Robotic Perception Multimodal Learning Multimodal Sensor Fusion

Open paper AskPDF arXiv

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.

Abstract

Multimodal SSL

Explore similar work

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning