Cross-Modal Representation Learning

Follow this topic Ask about this topic

The field studies how models can represent different kinds of data so that related observations can be matched, combined, or used to transfer knowledge. Common approaches include CLIP-style contrastive alignment, masked cross-modal reconstruction and predictive learning, teacher–student distillation, and methods that separate shared from modality-specific features; anchors and optimal transport can help when paired examples are scarce. These representations support image-text retrieval, cross-sensor recognition, and transfer across domains or missing-modality settings. A central tension is that shared spaces can improve transfer but discard useful modality-specific structure, and apparent semantic alignment does not always imply fine-grained correspondence; the best choice between alignment and prediction depends on data quality and cross-modal relationships.

Latest papers

210 papers22 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

SDF-Net: Structure-Aware Disentangled Feature Learning for Opticall-SAR Ship Re-identification
Mar 13, 2026Furui Chen, Han Wang, Yuhan Sun +5Cross-Modal Representation Learning Cross-Modal Retrieval
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing 100094, China · Key Laboratory of Space Utilization, Chinese Academy of Sciences, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China +1
Multimodal Graph Representation Learning with Dynamic Information Pathways
Mar 10, 2026Xiaobin Hong, Mingkai Lin, Xiaoli Wang +2Multimodal Graph Learning Multimodal Embedding
State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities
Mar 10, 2026Jindi Bao, Jianjun Qian, Mengkai Yan +1Multimodal Sentiment Analysis Missing-Modality Learning
Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China
RLPR: Radar-to-LiDAR Place Recognition via Two-Stage Asymmetric Cross-Modal Alignment for Autonomous Driving
Mar 9, 2026Zhangshuo Qi, Jingyi Xu, Luqi Cheng +2Autonomous Driving Cross-Modal Representation Learning
Beijing Institute of Technology, Beijing, 100081, China. · Shanghai Jiaotong University, Shanghai, 200240, China.
Aligning What EEG Can See: Structural Representations for Brain-Vision Matching
Mar 7, 2026Jingyi Tang, Shuai Jiang, Fei Su +1EEG-Based Visual Decoding Cross-Modal Alignment
Beijing University of Posts and Telecommunications, Beijing, 100876, China
Scaling Dense Event-Stream Pretraining from Visual Foundation Models
Mar 4, 2026Zhiwen Chen, Junhui Hou, Zhiyu Zhu +2Event-Based Vision Representation Learning
City University of Hong Kong · Xidian University
VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
Mar 2, 2026Athanasios Efthymiou, Stevan Rudinac, Monika Kackovic +2KG Embedding Multimodal KGs
University of Amsterdam, Amsterdam, The Netherlands
MIRAGE: Knowledge Graph-Guided Cross-Cohort MRI Synthesis for Alzheimer's Disease Prediction
Mar 2, 2026Guanchen Wu, Zhe Huang, Yuzhang Xie +6Alzheimer's Disease Classification Learning with Missing Data
Emory University, Atlanta, USA · Cornell University, New York, USA
From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification
Feb 28, 2026Vasiliy Kudryavtsev, Kirill Borodin, German Berezin +3Animal Re-Identification Multimodal Fusion
Faculty of IT, Technical University of Communication and Informatics, Moscow, Russia, 111024 · AI lab, Avito, Moscow, Russia, 125196
STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification
Feb 28, 2026Xingguo Xu, Zhanyu Liu, Weixiang Zhou +5Cross-Modal Representation Learning
School of Mathematical Sciences, Dalian University of Technology, China · College of Computer Science and Technology, Zhejiang University, China · School of Future Technology, Dalian University of Technology, China +1
A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Feb 27, 2026Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson +5Cross-Modal Representation Learning Cross-Modal Alignment
Google DeepMind · University College London
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
Towards Multimodal Domain Generalization with Few Labels
Feb 26, 2026Hongzhao Li, Hao Dong, Hualei Wan +3Multimodal Domain Generalization Multimodal Learning
Zhengzhou University · ETH Zürich · MBZUAI
E-MMKGR: A Unified Multimodal Knowledge Graph Framework for E-commerce Applications
Feb 24, 2026Jiwoo Kang, Yeon-Chang LeeMultimodal KGs Multimodal Recommendation
UNIST Ulsan, Korea
CrossLLM-Mamba: Multimodal State Space Fusion of LLMs for RNA Interaction Prediction
Feb 23, 2026Rabeya Tus Sadia, Qiang Ye, Qiang ChengCross-Modal Representation Learning State Space Models
Department of Computer Science, University of Kentucky, Lexington, KY, USA · Department of Mathematics, University of Kentucky, Lexington, KY, USA · Institute for Biomedical Informatics, University of Kentucky, Lexington, KY, USA
Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery
Feb 23, 2026Wei He, Xianghan Meng, Zhiyuan Huang +3Generalized Category Discovery Cross-Modal Representation Learning
School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Intellifusion Inc., Shenzhen, P.R. China
CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning
Feb 23, 2026Chunlei Meng, Guanhong Huang, Rong Fu +3Cross-Modal Representation Learning Multimodal Learning
Fudan University · Shantou University · University of Macau +1
CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension
Feb 22, 2026Lihao Liu, Yan Wang, Biao Yang +10Multimodal Large Language Models Cross-Modal Retrieval
Tsinghua University · Kuaishou Technology
MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment
Feb 22, 2026Duc Duy Nguyen, Tat-Jun Chin, Minh HoaiCross-Modal Representation Learning Cross-Modal Alignment
Australian Institute for Machine Learning, Adelaide University, Adelaide, SA 5000, Australia
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Feb 19, 2026Lorenzo Caselli, Marco Mistretta, Simone Magistri +1Cross-Modal Learning Cross-Modal Representation Learning
University of Florence, Media Integration and Communication Center (MICC), Italy
Semantic Purification for Conditional Representation Learning
Feb 5, 2026Jiaquan Wang, Yan Lyu, Chen Li +1Disentangled Representation Learning Representation Learning
School of Computer Science and Engineering, Southeast University, China · Biomedicine Discovery Institute, Department of Biochemistry and Molecular Biology, Monash University, Australia
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
Feb 2, 2026Xinshun Wang, Peiming Li, Ziyi Wang +5Cross-Modal Representation Learning Unified Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School · Sony R&D Center 3Nanyang Technological University
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Jan 31, 2026Yaohua Liu, Rong Fu, Amir H. Gandomi +2Representation Learning Cross-Modal Representation Learning
Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3
PPGFlowECG: Latent Rectified Flow with Cross-Modal Encoding for PPG-Guided ECG Generation and Cardiovascular Disease Detection
Sep 24, 2025Xiaocheng Fang, Jiarui Jin, Haoyu Wang +8Electrocardiography Conditional Generation
National Institute of Health Data Science, Peking University, China · School of Intelligence Science and Technology, Peking University, China · University of Chinese Academy of Sciences, China +1
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?
Aug 1, 2025Satiyabooshan Murugaboopathy, Connor T. Jerzak, Adel DaoudCross-Modal Representation Learning Multimodal Learning
AI and Global Development Lab, aidevlab.org · Institute for Analytical Sociology, Linköping University, Sweden · Institute of Computer Science, University Leipzig, Germany +1
Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging
Jun 19, 2025Jiawen Yang, Shuhao Chen, Shengtao Zhang +2Unsupervised Domain Adaptation Cross-Modal Alignment
Southern University of Science and Technology · SZ DJI Technology Co., Ltd
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
A quantitative analysis of semantic information in deep representations of text and images
May 21, 2025Santiago Acevedo, Andrea Mascaretti, Riccardo Rende +3Representational Similarity Analysis Cross-Modal Representation Learning
Scuola Internazionale Superiore di Studi Avanzati (SISSA) · Universitat Pompeu Fabra (UPF) · Catalan Institute of Research and Advanced Studies (ICREA) and Universitat Pompeu Fabra (UPF)
On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
Apr 14, 2025Yichao Cai, Yuhang Liu, Erdun Gao +4Cross-Modal Representation Learning Cross-Modal Alignment
Australian Institute for Machine Learning, The University of Adelaide, SA 5000, Australia

Mar 13, 2026J/K move · Enter open · S save

SDF-Net: Structure-Aware Disentangled Feature Learning for Opticall-SAR Ship Re-identification

Furui Chen, Han Wang, Yuhan Sun, Jianing You+4

Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing 100094, China · Key Laboratory of Space Utilization, Chinese Academy of Sciences, Beijing 100094, China · University of Chinese Academy of Sciences, Beijing 100049, China+1

Cross-Modal Representation Learning Cross-Modal Retrieval Disentangled Representation Learning Synthetic Aperture Radar

Open paper AskPDF arXiv

Cross-modal ship re-identification (ReID) between optical and synthetic aperture radar (SAR) imagery is fundamentally challenged by the severe radiometric discrepancy between passive optical imaging and coherent active radar sensing. While existing approaches primarily rely on statistical distribution alignment or semantic matching, they often overlook a critical physical prior: ships are rigid objects whose geometric structures remain stable across sensing modalities, whereas texture appearance is highly modality-dependent. In this work, we propose SDF-Net, a Structure-Aware Disentangled Feature Learning Network that systematically incorporates geometric consistency into optical--SAR ship ReID. Built upon a ViT backbone, SDF-Net introduces a structure consistency constraint that extracts scale-invariant gradient energy statistics from intermediate layers to robustly anchor representations against radiometric variations. At the terminal stage, SDF-Net disentangles the learned representations into modality-invariant identity features and modality-specific characteristics. These decoupled cues are then integrated through a parameter-free additive residual fusion, effectively enhancing discriminative power. Extensive experiments on the HOSS-ReID dataset demonstrate that SDF-Net consistently outperforms existing state-of-the-art methods. The code and trained models are publicly available at https://github.com/cfrfree/SDF-Net.