Cross-Modal Representation Learning

Ask about this topic

Overview

The field studies how models can represent different kinds of data so that related observations can be matched, combined, or used to transfer knowledge. Common approaches include CLIP-style contrastive alignment, masked cross-modal reconstruction and predictive learning, teacher–student distillation, and methods that separate shared from modality-specific features; anchors and optimal transport can help when paired examples are scarce. These representations support image-text retrieval, cross-sensor recognition, and transfer across domains or missing-modality settings. A central tension is that shared spaces can improve transfer but discard useful modality-specific structure, and apparent semantic alignment does not always imply fine-grained correspondence; the best choice between alignment and prediction depends on data quality and cross-modal relationships.

Latest papers

CardsList

210 papers22 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

  1. Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking

    Oct 27, 2026Xi Chen, Xu Chen, Xiamgyang Jia +3Image-Text RetrievalCross-Modal Representation Learning

    School of Computer Science, Wuhan University, China · Beijing Institute for General Artificial Intelligence (BIGAI), China

  2. AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

    Oct 27, 2026Chaofei Hong, Ziyang Chen, Baochen Wang +2Joint-Embedding Predictive ArchitectureCross-Modal Representation Learning

    Zhejiang Lab

  3. AI Mindmeld: A Latent Interconnect Protocol

    Oct 21, 2026Siavash Golkar, Jake Kovalic, Shirley HoLarge Language Model-Guided Scientific DiscoveryCross-Modal Representation Learning

    Polymathic AI · Center for Data Science, New York University · Department of Applied Physics, Yale University +3

  4. Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

    Oct 20, 2026Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin +2Audio-Text AlignmentAudio-Language Models

    Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

  5. Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

    Oct 19, 2026Shiyun Xiong, Dongming Wu, Peiwen Sun +5Unified Multimodal ModelsMultimodal Large Language Models

    MMLab, The Chinese University of Hong Kong · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +2

  6. SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention

    Oct 19, 2026Saurav Raj Pandey, Harlin LeeMultimodal PretrainingCross-Modal Representation Learning

    Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · School of Data and Information Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

  7. Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection

    Oct 18, 2026Qiwu Wen, Jahangir AlamMultimodal Deepfake DetectionMultimodal Fusion

    Luqia Technologies, 405 Ogilvy Ave., Suite 101, Montréal, Québec H3N 1M3, Canada

  8. MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

    Oct 17, 2026Haowen Bai, Rui Gong$, Meichen Liu +3Multimodal SSLJoint-Embedding Predictive Architecture

    Nanyang Technological University · Hyundai Motor Group Innovation Center Singapore (HMGICS)

  9. Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling

    Oct 17, 2026Jiawei Guo, Meng ZhaoMultimodal LearningCross-Modal Representation Learning

    Department of Industrial and Systems Engineering, Lehigh University

  10. Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

    Oct 16, 2026Zehui Feng, Weichuan Wang, Xiaohan Chen +2Neural DecodingBrain-to-Speech Decoding

    Shanghai Jiao Tong University · Nanyang Technological University · Zhejiang University

  11. CIIR - Class-Indexed Interface Representation Learning

    Oct 16, 2026Doo Yon Kim, Christian WallravenCross-Modal Representation LearningCross-Modal Representation Transfer

  12. Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

    Oct 13, 2026Thinker Team, Peng Gao, Jichao Jiao +8World Models for RoboticsEmbodied World Models

    Ubtech

  13. Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

    Oct 13, 2026Ding Ma, Jinyi Mi, Jiajun He +3Voice ConversionSpeech Enhancement

    Graduate School of Informatics, Nagoya University, Nagoya, Japan · Alibaba Token Foundry, Alibaba Group, Hangzhou, China · Headquarters, TARVO, Inc., Nagoya, Japan +2

  14. EarthAlign: Learning Language-Aligned Representations of Earth System Data

    Oct 10, 2026Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma +4Cross-Modal Representation LearningMultimodal Embedding

    Mohamed bin Zayed University of Artificial Intelligence · ADIA Lab · University of California San Diego

  15. Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

    Oct 10, 2026Yuxin Jiang, Yunkang Cao, Weiming ShenMultimodal Anomaly DetectionMultimodal Fusion

    School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, 430000, Hubei, China

  16. HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignment

    Oct 10, 2026Jiaqi Wang, Wei Chen, Xiaoyu Li +7Representation LearningPhotoplethysmography

    Southeast University · OPPO · Shenzhen Technology University +1

  17. Representation Alignment from a Few Paired Data via Distilled Optimal Transport

    Oct 8, 2026Julien Kordes, Reuben DorentCross-Modal AlignmentCross-Modal Representation Learning

    Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, 75013 Paris, France

  18. Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

    Oct 8, 2026Youngtaek Oh, Qiyu Wu, Hiromi Wakaki +2Multimodal EmbeddingCross-Modal Representation Learning

    KAIST · Sony Group Corporation

  19. SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

    Oct 7, 2026Jeonghyeok Do, Munchurl KimRemote Sensing Image UnderstandingMultimodal Pretraining

    Korea Advanced Institute of Science and Technology (KAIST)

  20. MoTIF-X: A Multimodal Tokenized Framework for Interpretable and Extensible Molecular Representation Learning

    Sep 29, 2026Linqing Mo, Jiayu Zhou, Bin ChenMolecular Representation LearningCross-Modal Representation Learning

    Department of Computer Science and Engineering, Michigan State University, East Lansing, 48824, MI, USA. · School of Information, University of Michigan, Ann Arbor, 48109, MI, USA. · Department of Pediatrics and Human Development, Michigan State University, Grand Rapids, MI, 49503, USA. +2

  21. UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception

    Sep 29, 2026Yuhao Liu, Yiming Zhong, Hanqing Wang +9Cross-Modal Representation LearningUnified Multimodal Models

    ShanghaiTech University · Shandong University · HKUST(GZ) +3

  22. Physical Cross-Modal Masked Autoencoding for Seismic-to-Well Representation Learning

    Sep 28, 2026Meher Gajula, Keyla Gonzalez, Ben Lasscock +1Cross-Modal Representation LearningRepresentation Learning

  23. Do Emotion Concepts Generalize Across Sources, Modalities, and Architectures in Vision-Language Models?

    Sep 28, 2026Bohao Xing, Xin Liu, Kaishen Yuan +5Cross-Modal LearningVision-Language Models

    Lappeenranta-Lahti University of Technology LUT · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou) +3

  24. TRACE: Expert-Aligned ECG Representation Learning with Rigorous Benchmarking and Real-World Validation in Acute Cardiac Care

    Sep 28, 2026Lovely Yeswanth Panchumarthi, Andrew Lu, Saurabh Kataria +11Representation LearningCross-Modal Representation Learning

    Department of Computer Science, Emory University, GA, USA · School of Nursing, Emory University, GA, USA · Institute for Medical Engineering and Science, Massachusetts Institute of Technology, MA, USA +3

  25. SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

    Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation LearningMultimodal Contrastive Learning

    OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

  26. PhyMo: A Physical-Field Modality for Multimodal AI4Physics

    Sep 23, 2026Henan Sun, Haitao Hu, Jin Liu +4AI for ScienceCross-Modal Representation Learning

    The Hong Kong University of Science and Technology (Guangzhou) · Huawei Noah’s Ark Lab · Tencent HY +1

  27. Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

    Sep 19, 2026Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie TahonAudio Representation LearningCross-Modal Representation Learning

    Department of computer sciences of University of Yaounde I, Yaounde, 812, Cameroun · Department of computer sciences of University of Yaounde I and IRD, UMMISCO, Bondy, France, F-93143 · LIUM, Le Mans Universit´e, Av. Olivier Messiaen, 72085 Le Mans, France

  28. RPA: Residual Patch-Token Adapter for Image Retrieval from EEG and MEG

    Sep 19, 2026Yuhui Jin, Yonghao Song, Bingchuan LiuCross-Subject EEG DecodingCross-Modal Representation Learning

    Vortek Lab Inc. · Department of Computer Science and Technology, Tsinghua University · Biology and Biological Engineering, California Institute of Technology

  29. MarsRecon: Self-Supervised and Multimodal Surface Representations for Mars

    Sep 17, 2026Akshay Naik, Marius F. R. Juston, Jay MahajanGeospatial Representation LearningSelf-Supervised Visual Representation Learning

    Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA

  30. MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction

    Sep 17, 2026Akshit Sharma, Prashant W. PatilHarmful Meme DetectionMultimodal Classification

    Indian Institute of Technology Guwahati