Cross-Modal Representation Learning

Overview

The field studies how models can represent different kinds of data so that related observations can be matched, combined, or used to transfer knowledge. Common approaches include CLIP-style contrastive alignment, masked cross-modal reconstruction and predictive learning, teacher–student distillation, and methods that separate shared from modality-specific features; anchors and optimal transport can help when paired examples are scarce. These representations support image-text retrieval, cross-sensor recognition, and transfer across domains or missing-modality settings. A central tension is that shared spaces can improve transfer but discard useful modality-specific structure, and apparent semantic alignment does not always imply fine-grained correspondence; the best choice between alignment and prediction depends on data quality and cross-modal relationships.

Latest papers

CardsList

224 papers17 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.

  1. From Points to Trajectories: Rethinking Multimodal Alignment for Physical Events

    Nov 18, 2026Yiyang Zhu, Guangyuan Liu, Jingdi Lei +2Cross-Modal AlignmentCross-Modal Representation Learning

  2. Cardiac MRI-informed electrocardiography enables biologically grounded cardiomyopathy detection

    Nov 18, 2026Zhengyao Ding, Kewen Wang, Yujian Hu +27Electrocardiogram Representation LearningElectrocardiography

  3. Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation

    Nov 17, 2026Jihoo Jung, Ji-Hoon Kim, Suyeon Lee +2Audio-Video GenerationAudio-Visual Alignment

  4. CroCoMAE: Cross-modal self-supervised collaborative representation learning for SAR-optical remote sensing data

    Nov 10, 2026Vito Recchia, Giuseppe Guarino, Cássio Fraga Dantas +3Cross-Modal Representation LearningMultimodal SSL

  5. LUMIR: Decoupled Multimodal Representation Learning from Heterogeneous Data for Next-POI Recommendation

    Nov 10, 2026Zehui Wang, Wolfram Höpken, Dietmar JannachMultimodal RecommendationCross-Modal Representation Learning

  6. Flow-Guided Target-Space Alignment via Path Consistency

    Nov 6, 2026Ruizhi Yuan, Zeqiu Yu, Wei Gao +3Cross-Modal AlignmentCross-Modal Representation Learning

  7. RegNC-Mamba3: Shared Structural Feature Learning Using Non-Causal State Space Principles for Multi-Modal Image Registration

    Nov 6, 2026Zhikang Li, Yan Wu, Xin Hu +2Cross-Modal Image RegistrationMamba

  8. CAPE-JEPA - Cross-modal Adversarial Probabilistic Embedding JEPA

    Nov 4, 2026Aniket Junghare, Biplab BanerjeeCross-Modal Representation LearningCross-Modal Retrieval

  9. DenseTwins: Dense Modality Decoupling for Multimodal Self-Supervision

    Nov 3, 2026Carlo Sgaravatti, Daniele Kopyshevskiy, Davide Secco +1Multimodal SSLCross-Modal Representation Learning

  10. Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization

    Nov 2, 2026Anindya Nag, Ambuj Mehrish, Sebastiano VasconCross-Modal Representation LearningCross-Modal Retrieval

  11. Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment

    Nov 2, 2026Zohar Mashiah, Nir Ben-Ari, Ronen Talmon +1Cross-Modal AlignmentCross-Modal Representation Learning

  12. Text-Assisted Event Stream Visual Place Recognition via Multi-Scale Spatiotemporal Optimal Transport

    Nov 1, 2026Lan Chen, Xingxing Xiong, Xiao Wang +5Visual Place RecognitionEvent-Based Vision

  13. Parameter-Efficient Fine-Tuning of Transformer Encoders for Audio-Image-Symbolic Music Retrieval

    Oct 30, 2026Eliseo Fuentes-Martinez, Antonio Hidalgo-Centeno, Jorge Calvo-Zaragoza +1Multimodal IRParameter-Efficient Fine-Tuning

  14. Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking

    Oct 27, 2026Xi Chen, Xu Chen, Xiamgyang Jia +3Image-Text RetrievalCross-Modal Representation Learning

  15. AI Mindmeld: A Latent Interconnect Protocol

    Oct 21, 2026Siavash Golkar, Jake Kovalic, Shirley HoLarge Language Model-Guided Scientific DiscoveryCross-Modal Representation Learning

  16. Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

    Oct 20, 2026Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin +2Audio-Text AlignmentAudio-Language Models

  17. Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

    Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal ModelsMultimodal Large Language Models

  18. SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention

    Oct 19, 2026Saurav Raj Pandey, Harlin LeeMultimodal PretrainingCross-Modal Representation Learning

  19. Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection

    Oct 18, 2026Qiwu Wen, Jahangir AlamMultimodal Deepfake DetectionMultimodal Fusion

  20. MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

    Oct 17, 2026Haowen Bai, Rui Gong$, Meichen Liu +3Multimodal SSLJoint-Embedding Predictive Architecture

  21. Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

    Oct 16, 2026Zehui Feng, Weichuan Wang, Xiaohan Chen +2Neural DecodingBrain-to-Speech Decoding

  22. Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

    Oct 13, 2026Ding Ma, Jinyi Mi, Jiajun He +3Voice ConversionSpeech Enhancement

  23. EarthAlign: Learning Language-Aligned Representations of Earth System Data

    Oct 10, 2026Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma +4Cross-Modal Representation LearningMultimodal Embedding

  24. Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

    Oct 10, 2026Yuxin Jiang, Yunkang Cao, Weiming ShenMultimodal Anomaly DetectionMultimodal Fusion

  25. HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignment

    Oct 10, 2026Jiaqi Wang, Wei Chen, Xiaoyu Li +7Representation LearningPhotoplethysmography