DuoJEPA: Cross-Modal Translation Between Unaligned Self-Supervised Embedding Spaces
Authors: Mahmut S. Gokmen, Mitchell A. Klusty, Evan W. Damron, Caroline N. Leach, Samuel E. Armstrong, V. K. Cody Bumgardner
Organizations: Kentucky Center for Digital Innovation, University of Louisville, Louisville, KY, United States · University of Kentucky, Lexington, Office of Research, College of Medicine, United States
Learning from paired modalities is commonly equated with aligning them: two encoders are trained until the embeddings of a pair coincide in a shared space. We study whether cross-modal retrieval and classification can instead be obtained while the two encoders' own embedding spaces remain directly unaligned. DuoJEPA trains one encoder per modality with the self-supervised LeJEPA objective and appends to each encoder a single transformer layer that translates its embedding into the other encoder's space, supervised by a contrastive objective computed entirely within that target space. No loss term compares the two own spaces directly, although the encoders are optimised jointly through the translation objective. In our experiments, the translator's residual connections determine where the cross-modal pressure is absorbed: with a residual translator the two encoders become directly aligned, whereas with a skipless (residual-free) translator a paired-cosine test detects no alignment in the final models, and at most a weak one in the ablations, at comparable cross-modal performance. The learned class-token translation is well approximated by a single scaled rotation of the source space. We evaluate DuoJEPA on two modality pairs that differ in nearly every respect. On chest radiographs and reports from CheXpert Plus, it attains a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, compared with 60.0 and 64.2 for our reimplementation of LeVLJEPA with the same encoders and data. On Sentinel-2 optical and Sentinel-1 radar imagery, the frozen encoders fall within the range of published Sentinel foundation models under our reproduction of the Copernicus-Bench linear-probe protocol, the translator retrieves the paired image of the other sensor among 4,000 candidates in 77-79% of cases, and a patch-level extension transfers the within-image similarity structure of the scene.
Figures & tables
Figure 1: A simplified skipless translator trained with an InfoNCE objective on 27 of 40 synthetic clusters recovers a rotation planted between two three-dimensional spaces (a, b), which acts as one turn of the whole space (c), and maps the 13 clusters it never saw onto their targets (d); the setup is given in Appendix D .
Figure 2: One pair of each kind: a frontal chest radiograph and the impression section of its report (a, b), and the same 2.64 km × 2.64 km area seen by the optical Sentinel-2 in natural colour and by the radar Sentinel-1 in its two polarisations (c, d).
Figure 3: DuoJEPA on chest radiographs and their reports: each tower’s own CLS ( h12 ) trains its LeJEPA loss, and its skipless translator layer ( h13 ) is trained by an InfoNCE in the other tower’s space.
Figure 4: DuoJEPA on Sentinel-2 and Sentinel-1 images of the same place, with the same CLS route as Figure 3 .
Figure 5: The patch InfoNCE in the space of a target modality m pulls a translated patch of the source modality mˉ towards the detached own patch of the same image at the same position and pushes it away from that position in the other images of the batch.
Space Applications Centre, Indian Space Research Organisation, Ahmedabad, India · Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India