Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Ciem Cornelissen; Sam Leroux; Pieter Simoens

cs.CVMar 25, 2026

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

Authors: Ciem Cornelissen, Sam Leroux, Pieter Simoens

Organizations: IDLab, Department of Information Technology, Ghent University - imec, Belgium

Abstract

Self-supervised learning has emerged as a powerful paradigm for learning visual representations without manual annotations, yet most methods still operate on a single modality and therefore miss the complementary structure available from heterogeneous sensors. We present Le MuMo JEPA, a self-supervised framework that learns unified representations from RGB images and aligned companion modalities. In our driving experiments, the second modality is camera-aligned LiDAR depth; we also evaluate RGB-thermal training and transfer on the Teledyne FLIR ADAS benchmark. Our approach extends LeJEPA to the multi-modal setting by learning fusion tokens that act as a latent bottleneck between modality-specific patch stems inside a shared transformer. Our default model employs a pruned fusion strategy: after an initial cross-modal attention layer, modality-specific tokens are dropped, forcing cross-modal information into the shared fusion-token grid as an efficient latent bottleneck before Sketched Isotropic Gaussian Regularization (SIGReg) is applied to the joint multimodal CLS embedding. On Waymo, Le MuMo JEPA gives the strongest performance-efficiency trade-off on downstream patch probes among the from-scratch multimodal baselines, improving CenterNet detection and dense depth while remaining competitive on segmentation. Under from-scratch training on nuScenes, Le MuMo JEPA remains the strongest model, and it also gives the best FLIR results, especially after Waymo-initialized fine-tuning. It also retains the best overall accuracy-efficiency balance in our study at substantially lower compute, memory, and estimated training time.

Figures & tables

Figure 1: Overview of Le MuMo JEPA. The companion modality is represented as a spatially aligned signal and fused with RGB through learnable fusion tokens , which act as a latent bottleneck inside a shared transformer. The default training objective applies SIGReg to the joint multimodal CLS embedding.

Figure 2: Waymo dataset showcase. Example synchronized supervision used in our experiments: (a) RGB image, (b) camera-view segmentation, (c) the aligned companion-modality signal shown in depth form for the driving setting, and (d) projected 3D bounding boxes.

Method	mAP XY $\uparrow$	Depth MAE $\downarrow$	Seg. mIoU $\uparrow$
LeJEPA [ 4 ]	19.3	4.704	0.261
LeJEPA encoder frozen [ 4 ]	13.8	6.734	0.126
LiDAR-only	15.4	2.982	0.151
LiDAR-only encoder frozen	8.0	5.397	0.089

Table 1: Self-supervised single-modality encoders vs. encoder-frozen controls on Waymo. All rows train the probes. Rows labeled “encoder frozen” keep a randomly initialized encoder fixed, whereas the other rows train the encoder self-supervised before frozen-probe evaluation. We report XY mAP, Depth MAE, and Seg. mIoU.

Method	mAP XY $\uparrow$	Depth MAE $\downarrow$	Seg. mIoU $\uparrow$
Early Fusion RGBD	18.1	4.767	0.248
Late Fusion	18.7	4.802	0.251
FT-Pruned + VICReg [ 7 ]	22.8	2.911	0.248
FT-Persistent + SIGReg	23.1	2.846	0.271
FT-Pruned + SIGReg (3-pass)	23.2	2.777	0.274
Le MuMo JEPA	23.6	2.860	0.275

Table 2: Focused fusion comparison on Waymo for the fusion variants. We compare the default Le MuMo JEPA setting against early and late fusion, persistent routing, VICReg, and the 3-pass SIGReg variant.

Method	Training Data	mAP XY $\uparrow$	Depth MAE $\downarrow$	mAP XZ $\uparrow$	Seg. mIoU $\uparrow$
LeJEPA [ 4 ]	RGB	19.3	4.704	4.9	0.261
DINOv3 [ 22 ]	RGB	15.2	5.314	3.5	0.239
LiDAR-only	Depth	15.4	2.982	4.0	0.151
MultiMAE-SS [ 2 ]	RGB+Depth	13.5	4.441	2.7	0.221
MultiMAE-MT [ 2 ]	RGB+Depth	13.7	3.583	2.9	0.262
ImageBind [ 13 ]	RGB+Depth	13.4	4.309	3.0	0.243

Table 3: Patch-probe comparison on Waymo with all encoders trained from scratch. CenterNet and dense prediction heads are trained on frozen patch embeddings taken from models that were all self-supervised from scratch on Waymo. “Training Data” denotes the modalities seen during SSL pretraining; for dual-stream baselines, the frozen patch probes use the camera-aligned RGB patch readout described in Section 4.1 . Best results are in bold ; second best are underlined .

\uparrow

means higher is better and

\downarrow

means lower is better.

Method	mAP XY $\uparrow$	mAP XZ $\uparrow$	Seg. mIoU $\uparrow$	Depth MAE $\downarrow$
MultiMAE-SS [ 2 ]	6.95	1.66	0.195	5.736
MultiMAE-MT [ 2 ]	6.67	1.62	0.192	5.624
ImageBind [ 13 ]	6.86	1.57	0.198	5.912
Le MuMo JEPA	9.52	2.53	0.228	2.031

Table 4: Results on the nuScenes dataset. We report CenterNet XY-match mAP, XZ-match mAP, semantic segmentation, and Depth MAE.

Setting	Method	mAP50 $\uparrow$	Car mAP50 $\uparrow$
Scratch	MultiMAE-SS [ 2 ]	0.51	3.86
Scratch	ImageBind [ 13 ]	0.24	1.90
Scratch	Le MuMo JEPA	0.41	3.00
Waymo $\rightarrow$ FLIR	MultiMAE-SS [ 2 ]	0.54	4.21
Waymo $\rightarrow$ FLIR	MultiMAE-MT [ 2 ]	0.69	5.26
Waymo $\rightarrow$ FLIR	ImageBind [ 13 ]	0.72	5.49

Table 5: FLIR results across training regimes. We report 2D CenterNet mAP50 and Car mAP50. Rows are ordered from training from scratch on FLIR, to Waymo

\rightarrow

FLIR transfer, to Waymo-pretrained fine-tuning on FLIR. Because FLIR does not provide segmentation labels, the scratch block excludes MultiMAE-MT.

Method	Time (min) $\downarrow$	SSL FLOPs (PFLOPs) $\downarrow$	VRAM (GB) $\downarrow$
MultiMAE-SS [ 2 ]	14.07	21.4	2.94
Early Fusion RGBD	18.70	72.9	6.15
LeJEPA [ 4 ]	18.73	72.7	6.01
ImageBind [ 13 ]	23.65	80.4	9.49
Late Fusion	34.72	144.9	11.41
DINOv3 [ 22 ]	37.72	108.8	8.70

Table 6: Encoder-side compute profile. Rows are sorted by estimated total encoder training time. We report time in minutes, total encoder SSL FLOPs in PFLOPs, and peak reserved VRAM in GB. Time is estimated for the encoder-side workload on an H200 system with an AMD EPYC 9275F 24-Core Processor. The fusion-token rows use the default single-pass joint-CLS objective, with the 3-pass SIGReg variant shown separately as an ablation.

Figure 3: Waymo patch-embedding visualization. t-SNE projections of final-layer patch embeddings are shown with class-oriented structure and depth-oriented structure, illustrating how the learned patch space organizes both semantic grouping and geometric variation across methods. The left column uses Le MuMo JEPA fusion-token embeddings, and the right column uses LeJEPA patch embeddings. A planar fit to the plotted depth gradient gives an $R^{2}$ score of $0.463$ for Le MuMo JEPA versus $0.086$ for LeJEPA.

Figure 4: Waymo qualitative probe output. The figure shows the RGB input together with predictions from the three probe families used in the paper: dense depth estimation, segmentation, and 3D detection boxes. It provides a direct qualitative view of the same patch-level capabilities summarized by the quantitative probe tables.

Multimodal SSL

See all 14 papers

Jun 25, 2026

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.

Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2

1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University

Autonomous Driving Datasets Masked Autoencoders Multimodal SSL Robotic Perception Multimodal Learning Multimodal Sensor Fusion

Ask PDF

Oct 11, 2026

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of

75.23%\pm0.60

averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.

Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu +4

Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Candour Ltd, Oulu, Finland

Multimodal Learning Multimodal Sensor Fusion Joint-Embedding Predictive Architecture

Ask PDF

Mar 15, 2026

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.

Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar +6

FAIR at Meta · Universidad de Zaragoza

Self-Supervised Visual Representation Learning Visual Representation Learning Video Representation Learning Joint-Embedding Predictive Architecture

Ask PDF

Abstract

Multimodal SSL

Explore similar work

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning