Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM Evaluation Modality Imbalance
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations
Jun 3, 2026Zixu Li, Yupeng Hu, Zhiwei Chen +3Cross-Modal Representation Learning Cross-Modal Alignment
School of Software, Shandong University, Jinan, 250100, China · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518000, China · Department of Data Science, City University of Hong Kong, Hong Kong 523808, China +1
DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction
Jun 2, 2026Koki Nagano, Hongyu Liu, Seonwook Park +9Cross-Modal Alignment Human Motion Generation
1NVIDIA · 2HKUST
Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning
Jun 2, 2026Yang Liu, Wentao Feng, Shu-Dong Huang +2Cross-Modal Alignment Cross-Modal Retrieval
College of Computer Science, Sichuan University, China · School of Computer Sci
Mixed-Modality Dual Face-Hair Retrieval
Jun 2, 2026Quoc-Anh Bui-Huynh, Mai-Tuyen Lam, Dai-Anh-Tuan Nguyen +1Cross-Modal Alignment Cross-Modal Retrieval
Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam
Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection
Jun 1, 2026David J. Lerch, Livien Majer, Zeyun Zhong +3Multimodal Robustness Cross-Modal Alignment
Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
NeuroAlign: Hierarchical Multimodal Fusion of Dynamic and Structural Neuroimaging for MCI Analysis
May 31, 2026Xiongri Shen, Zhenxi Song, Jiaqi wang +13Feature Attribution Functional Connectivity
Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518055, China · School of Intelligence Science and Engineering, College of Artificial Intelligence, Harbin Institute of Technology, Shenzhen, 518055, China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China +5
V-LynX: Token Interface Alignment for Video+X LLMs
May 30, 2026Jungin Park, Jiyoung Lee, Kwanghoon SohnCross-Modal Alignment Multimodal QA
Yonsei University, Seoul, South Korea · Ewha Womans University, Seoul, South Korea
Spatial Transcriptomics-Guided Alignment Enhances Molecular Profiling in Pathology Foundation Model
May 29, 2026Fengtao Zhou, Yingxue Xu, Zhengyu Zhang +20Spatial Transcriptomics Cross-Modal Alignment
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Department of Pathology, Nanfang Hospital, Southern Medical University, Guangzhou, China · Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, China +16
HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning
May 29, 2026Md Aminur Hossain, Ayush V. Patel, Sanjay K. Singh +1Representation Learning Cross-Modal Representation Learning
Space Applications Centre, Indian Space Research Organisation, Ahmedabad, India · Centre of Studies in Resources Engineering, Indian Institute of Technology Bombay, Mumbai, India
DiffCrossGait: Trajectory-Level Alignment for 2D-3D Cross-Modal Gait Recognition via Latent Diffusion
May 29, 2026Zhiyang Lu, Ming ChengDiffusion Model Alignment Latent Diffusion Models
1Fujian Key Laboratory of Urban Intelligent Sensing and Com · 2Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Min
Text-guided Feature Disentanglement for Cross-modal Gait Recognition
May 29, 2026Zhiyang Lu, Ming ChengDisentangled Representation Learning Multimodal Robustness
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University. · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Semantic Motion Anchors: Bridging Motion and Meaning in Co-Speech Gestures
May 28, 2026Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman +5Cross-Modal Alignment Human Motion Generation
1Saarland University · University of British Columbia · 4Vector Institute +2
Genetically Aligned Patient Representations Improve Hematological Diagnosis
May 28, 2026Muhammed Furkan Dasdelen, Fatih Ozlugedik, Ilaria Looser +3Computational Pathology Healthcare
Institute of AI for Health, Helmholtz Munich, Germany · International School of Medicine, Istanbul Medipol University, Türkiye · Munich Leukemia Laboratory, Germany +4
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal Learning Cross-Modal Alignment
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
SCENT: Aligning Mass Spectra with Molecular Structure for Olfactory Perception
May 26, 2026Ziqi Zhang, Eunyeong Jin, Miguel Vasco +6Cross-Modal Alignment Molecular Property Prediction
1Dept. of Intelligent Systems, KTH Royal Institute of Technology, Stockholm, Sweden · 2Atmospheric Chemistry Dept., Max Planck Institute for Chemistry, Mainz, Germany · 3Dept. of Information Technology, Uppsala University, Uppsala, Sweden +1
Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspective
May 26, 2026Xiang Fang, Zeyu Xiong, Wanlong Fang +7Temporal Video Grounding Cooperative Game Theory
Hubei Key Laboratory of Distributed System Security, Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology · University of Central Florida · Zhejiang Gongshang University +3
Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
May 25, 2026Tianling Liu, Lequan Yu, Tong Han +1Missing-Modality Learning Multimodal Robustness
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2
CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning
May 25, 2026Sriram MandalikaCross-Modal Learning VLM Adaptation
Hasso Plattner Institute Potsdam, Germany
MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models
May 23, 2026Jiaxiang Liu, Jiawei Du, Xupeng Chen +4Neural Decoding Cross-Modal Alignment
Guangdong Institute of Intelligence Science and Technology Hengqin, China · Agency for Science, Technology and Research Singapore · New York University Brooklyn, US +2
MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
May 23, 2026Zexuan Chen, Sichao Liu, Runhao Lu +4EEG-Based Visual Decoding Cross-Modal Alignment
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2
D2-V2X: Depth-Driven Cooperative V2X Reasoning for Autonomous Driving
May 22, 2026Kevin Richard, Alphin Varghese, Colin Pham +2VLMs for Autonomous Driving Cross-Modal Alignment
University of North Carolina at Charlotte
Representation Alignment Rests on Linear Structure
May 22, 2026Kiril Bangachev, Guy Bresler, Yury PolyanskiyLinear Representation Hypothesis Cross-Modal Alignment
Department of Electrical Engineering and Computer Science · Massachusetts Institute of Technology
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?
May 22, 2026Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra +1Cross-Modal Learning Cross-Modal Alignment
University of Amsterdam Amsterdam, The Netherlands
Multimodal Distribution Matching for Vision-Language Dataset Distillation
May 22, 2026Jongoh Jeong, Hoyong Kwon, Minseok Kim +1Cross-Modal Learning Cross-Modal Alignment
Visual Intelligence Lab., KAIST
STAMBRIDGE: Spectral-Temporal Amplitude-aware Mid-Feature Bridge for EEG Visual Decoding
May 22, 2026Jiahe Meng, Weiming Zeng, Yueyang Li +5Cross-Modal Knowledge Distillation Representation Learning
Lab of Digital Image and Intelligent Computation, Shanghai Maritime University, Shanghai 201306, China · Department of Language Science and Technology, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China · Department of Neurology, Affiliated Lianyungang Hospital of Xuzhou Medical University, Lianyungang 222002, China +1
STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval
May 20, 2026Miaoge Li, Dongsheng Wang, Zening Sun +3Cross-Modal Learning Cross-Modal Alignment
Department of COMP/LSGI, The Hong Kong Polytechnic University, Hong Kong SAR · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong SAR
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection
May 20, 2026Liming Hou, Yueping Peng, Hexiang Hao +6Infrared Object Detection Multispectral Object Detection
Engineering University of PAP, Xi’an 710086, China · Unit Command Department, Officers College of PAP, Chengdu 610213, China
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
May 20, 2026Andong Hua, Colton Bishop, Igor Mordatch +5Multimodal Large Language Models Modality Gap in VLMs
University of California, Santa Barbara · Google DeepMind

Jun 3, 2026J/K move · Enter open · S save

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

Alessandro Gambetti, Qiwei Han, Cláudia Soares, Hong Shen

NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics+2

VLM Evaluation Modality Imbalance Cross-Modal Alignment Medical VLMs Modality Gap in VLMs Vision-Language Alignment

Open paper AskPDF arXiv

Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited. Existing representation alignment metrics are symmetric, collapsing both modalities into a single score and hiding which modality drives cross-modal degradation. We introduce the Spectral Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance. We embed SAS within a benchmarking framework evaluating 15 VLMs across natural and medical image-text datasets alongside 6 alignment metrics and bidirectional retrieval. Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment. Code is available at this URL: https://github.com/iamalegambetti/medical-vlms-assessment.