Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Jun 22, 2026Davide Caffagni, Alberto Compagnoni, Federico Melis +5Cross-Modal Alignment Multimodal Large Language Models
University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI
Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction
Jun 20, 2026Zixuan Liu, Kaijie Yu, Chun He +5Cross-Modal Alignment Phylogenetic Inference
Zhejiang Key Lab of Accessible Perception and Intelligent Systems, Zhejiang University, Hangzhou 310027, China · State Key Laboratory of Rice Biology and Breeding & Ministry of Agricultural and Rural Affairs Key Laboratory of Molecular Biology of Crop Pathogens and Insects, Institute of Insect Sciences, Zhejiang University, Hangzhou 310058, China · Rural Development Academy, Zhejiang University, Hangzhou 310058, China +1
Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
Jun 19, 2026Yuxun Qu, Minyu Zhou, Yongqiang Tang +2Cross-Modal Alignment Multimodal Classification
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 101408, China
FUSE: Frequency-domain Unification and Spectral Energy Alignment for Multi-modal Object Re-Identification
Jun 18, 2026Xuanhao Qi, Tom H. Luan, Yukang Zhang +4Cross-Modal Representation Learning Multimodal Robustness
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · School of Informatics, Xiamen University, Xiamen, China · National University of Singapore, Singapore
CDER-SME: A Cross-Device Event-RGB Micro-Expression Dataset under Multi-Level Stress Induction
Jun 16, 2026Jingting Li, Hui Sha, Su-Jing WangCross-Modal Alignment Multimodal Emotion Recognition
State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China · Department of Psychology, University of the Chinese Academy of Sciences, Beijing, 100049, China · School of Computer Science, Jiangsu University of Science and Technology, Jiangsu, 212028, China
GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving
Jun 15, 2026Ziying Song, Caiyan Jia, Lin Liu +3Diffusion Model Alignment End-to-End Autonomous Driving
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science and Technology, Beijing Jiaotong University · School of Artificial Intelligence (School of Software), Yanshan University · University of Macau +2
Propagating Structural Guidance: Synthesizing Fluorescein Angiography from Fundus Images and Sparse OCT Scans
Jun 15, 2026Tengfei Ma, Ruiqi Wu, Chenran Zhang +6Medical Imaging Cross-Modal Alignment
School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artiﬁcial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Nanjing, China · Tianyuan Honors School, Nanjing Medical University, China +2
Text-Driven Fusion for Infrared and Visible Images: Achieving Image Scene Adaptation on Hyperbolic Space
Jun 13, 2026Huan Kang, Hui Li, Tianyang Xu +3Multi-Source Image Fusion Infrared-Visible Image Fusion
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, UK
Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling
Jun 13, 2026Zhemin Zhang, Weijie Chen, David Le +7Event Time Prediction Multimodal Robustness
Arizona State University · Mayo Clinic
Trimodal Glioma Representation Alignment via Volumetric Contrastive Learning
Jun 12, 2026Denise Marini, Eleonora Grassucci, Danilo ComminielloCancer Survival Prediction Cross-Modal Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs
Jun 12, 2026Sirui Zhang, Xu Wang, Zhengyu Wu +2Cross-Modal Alignment Multimodal Graph Learning
A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition
Jun 11, 2026Haoran Zhang, Haokun Zhang, Pengyu Liu +3Micro-Gesture Recognition Unsupervised Domain Adaptation
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT), Hefei, China · School of Computer Science, University of Auckland(UOA), Auckland, New Zealand
Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement
Jun 11, 2026Tingyu Li, Le Zhou, Siyuan Li +5Cross-Modal Alignment Multimodal Reasoning
Shanghai Artificial Intelligence Laboratory · Shanghai Jiaotong University · Zhejiang University +1
SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs
Jun 11, 2026Zhengyu Wu, Xu Wang, Hongchao Qin +4Spectral Graph Embedding Graph Representation Learning
DIMOS: Disentangling Instance-level Moving Object Segmentation
Jun 11, 2026Hongxiang Huang, Hongwei Ren, Xiaopeng Lin +3Event-Based Vision Disentangled Representation Learning
The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Latent World Recovery for Multimodal Learning with Missing Modalities
Jun 10, 2026Hui Wang, Tianyu Ren, Joseph Butler +3Missing-Modality Learning Cross-Modal Alignment
Queen’s University Belfast, Belfast, United Kingdom
Tac-DINO: Learning Vision-Tactile Features with Patch Alignment
Jun 10, 2026Hong Li, Yankang Dong, Yue Xu +8Representation Learning Tactile Sensing
Shanghai Jiao Tong University · Ant Group · Shanghai Innovation Institute
On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning
Jun 10, 2026Zihan Zhang, Jie Hong, Siyuan Fan +2Zero-Shot Learning Cross-Modal Alignment
School of Software Engineering, Southeast University, Nanjing, 210096, China · Faculty of Engineering, The University of Hong Kong, Pok Fu Lam, 999077, Hong Kong SAR · Beijing Institute of Technology, Beijing, 100811, China +2
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Jun 9, 2026Ilay Kamai, Hugues Van Assel, Aviv Regev +2Cross-Modal Representation Learning Cross-Modal Alignment
Technion · Genentech · Brown University
Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data
Jun 9, 2026Anik GhoshPrototype-Based Learning Contrastive Learning
Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
Jun 9, 2026Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2Cross-Modal Knowledge Distillation Cross-Modal Alignment
School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam
ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphs
Jun 9, 2026Xianlin Zeng, Fan Xia, Xiangyu ChenGraph Representation Learning Cross-Modal Alignment
Postdoctoral Research Station at China RongTong Academy of Sciences Group Corporation Limited, Beijing, China · Duke University, Durham, NC, USA.
DeRA-MOS: Optimizing Text-to-Music Evaluation via Decoupled Listwise Ranking and Modality Alignment
Jun 8, 2026Chien-Chun Wang, Hung-Shin Lee, Hsin-Min Wang +1Text-to-Music Generation Music Generation Evaluation
E.SUN Financial Holding Co., Ltd., Taipei 10546, Taiwan · United Link Co., Ltd., Taipei 11493, Taiwan · Institute of Information Science, Academia Sinica, Taipei 11529, Taiwan +1
Cranio-Diff: Diffusion-based Cross-domain Craniofacial Reconstruction with 2D X-ray Skull Guidance and Structural Identity Constraints
Jun 8, 2026Ravi Shankar Prasad, Naresh Gurjar, Shashank Baghel +2Cross-Modal Alignment Conditional Diffusion Models
School of Computing and Electrical Engineering Indian Institute of Technology Mandi Himachal Pradesh, India · University Teaching Department, CSVTU Bhilai, India
A Unifying Framework for Concept-Based Representational Similarity
Jun 8, 2026Grégoire Dhimoïla, Agustin Martin Picard, Victor Boutin +2Representational Similarity Analysis Cross-Modal Alignment
Brown University · ENS Paris Saclay · DEEL - IRT Saint Exup´ery +2
Explicit Representation Alignment for Multimodal Sentiment Analysis
Jun 8, 2026Baode Wang, Ziming Wang, Huacan Wang +2Cross-Modal Alignment Sentiment Analysis
AgentAlpha
IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval
Jun 6, 2026Jiale Huang, Zixu Li, Zhiwei Chen +3Cross-Modal Alignment Cross-Modal Retrieval
Shandong University · Jinan, Shandong, China · Qilu University of Technology +1
DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning
Jun 6, 2026Hangui Lin, Yan Shu, Zhengyang Liang +8Cross-Modal Alignment VLM Reasoning
BAAI · University of Trento · Singapore Management University +1
Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification
Jun 3, 2026Feixiang Zhou, Jianyang Xie, Zhuangzhi Gao +10Missing-Modality Learning Cross-Modal Alignment
School of Eye and Vision Sciences, University of Liverpool, U.K. · School of Computer Science, University of Exeter, U.K. · School of Computer Science and Engineering, South China University of Technology, China +4
UniCA: Bi-directional Cross-Attention with Positive Similarity Loss for Robust Multi-Modal Retrieval
Jun 3, 2026Yini Huang, Wenlong ZhangMultimodal IR Cross-Modal Alignment
The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Southern Medical University, Guangdong, China

Jun 22, 2026J/K move · Enter open · S save

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto+4

University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI

Cross-Modal Alignment Multimodal Large Language Models Multimodal Foundation Models Multimodal Contrastive Learning VLM Hallucination Mitigation

Open paper AskPDF arXiv

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.