cs.CVNov 11, 2026

DuoJEPA: Cross-Modal Translation Between Unaligned Self-Supervised Embedding Spaces

Authors: Mahmut S. Gokmen, Mitchell A. Klusty, Evan W. Damron, Caroline N. Leach, Samuel E. Armstrong, V. K. Cody Bumgardner

Organizations: Kentucky Center for Digital Innovation, University of Louisville, Louisville, KY, United States · University of Kentucky, Lexington, Office of Research, College of Medicine, United States

Abstract

Learning from paired modalities is commonly equated with aligning them: two encoders are trained until the embeddings of a pair coincide in a shared space. We study whether cross-modal retrieval and classification can instead be obtained while the two encoders' own embedding spaces remain directly unaligned. DuoJEPA trains one encoder per modality with the self-supervised LeJEPA objective and appends to each encoder a single transformer layer that translates its embedding into the other encoder's space, supervised by a contrastive objective computed entirely within that target space. No loss term compares the two own spaces directly, although the encoders are optimised jointly through the translation objective. In our experiments, the translator's residual connections determine where the cross-modal pressure is absorbed: with a residual translator the two encoders become directly aligned, whereas with a skipless (residual-free) translator a paired-cosine test detects no alignment in the final models, and at most a weak one in the ablations, at comparable cross-modal performance. The learned class-token translation is well approximated by a single scaled rotation of the source space. We evaluate DuoJEPA on two modality pairs that differ in nearly every respect. On chest radiographs and reports from CheXpert Plus, it attains a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, compared with 60.0 and 64.2 for our reimplementation of LeVLJEPA with the same encoders and data. On Sentinel-2 optical and Sentinel-1 radar imagery, the frozen encoders fall within the range of published Sentinel foundation models under our reproduction of the Copernicus-Bench linear-probe protocol, the translator retrieves the paired image of the other sensor among 4,000 candidates in 77-79% of cases, and a patch-level extension transfers the within-image similarity structure of the scene.

Figures & tables

Explore similar work

CardsList
  1. CR-JEPA: Cross-Modal Joint-Embedding Predictive Learning for Remote Sensing Image Retrieval

    May 30, 2026Md Aminur Hossain, Ayush V. Patel, Nitant Dube +1Cross-Modal Representation LearningJoint-Embedding Predictive Architecture

  2. CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA

    Nov 4, 2026Aniket Junghare, Biplab BanerjeeCross-Modal Representation LearningCross-Modal Retrieval