Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding
Sep 4, 2026Kanglei Zhou, Chunyan Lan, Dongyang Li +2EEG-Based Visual Decoding Contrastive Learning
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China · Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing, China
Latent-Aligned Reasoning for Multimodal Recommendation
Sep 4, 2026Jiarui Jin, Anyang JiMultimodal Recommendation VLM Reasoning
Xiaohongshu Inc. · Nanjing University
CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception
Sep 3, 2026Weize Li, Yang Li, Quan Yuan +3Causal Representation Learning Cross-Modal Alignment
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection
Sep 3, 2026Yue Zhao, Hua Yu, Yukun Zhao +6Missing-Modality Learning Multimodal Robustness
School of Electronic Engineering, Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China · College of Computing and Data Science, Nanyang Technological University, Singapore · School of Electronics and Communication Engineering, Sun Yat-sen University, Guangzhou 510275, China +3
When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP
Sep 1, 2026Shota Sato, Hajime Kiyama, Tosho Hirasawa +1Cross-Modal Alignment Zero-Shot Image Classification
Hitotsubashi University
Differentially Private Paired Table-Image Multimodal Synthesis
Sep 1, 2026Kai Chen, Josephine Lamp, Somesh Jha +1Synthetic Data Generation Cross-Modal Alignment
University of Virginia · Dexcom · University of Wisconsin-Madison
Unmasking Face Embeddings: Reading, Rendering and Naming with Foundation Models
Aug 31, 2026Fizza Rubab, Yiying Tong, Arun RossCross-Modal Alignment Face Recognition
Michigan State University
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
Aug 31, 2026Merna Bibars, Bolaji Omofojoye, Allan I. Levey +3Missing-Modality Learning Depression Detection
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA · Department of Systems and Biomedical Engineering, Cairo University, Giza, Egypt · Department of Biomedical Informatics, Emory University, Atlanta, GA, USA +2
GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
Aug 28, 2026Jingpu Yang, Debin Tang, Yilin Sun +4Cross-Modal Alignment Multimodal Pretraining
Beihang University, Beijing, China. · Zhongguancun Academy, Beijing, China. · Northeastern University, Shenyang, China.
LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning
Aug 27, 2026Muyao Yuan, Muyan Jiao, Jiangyong Ying +5Multimodal Instruction Tuning Cross-Modal Alignment
MOEKLINNS, and School of Computer Science and Technology Xi’an Jiaotong University Xi’an, China · E-surfing Vision Technology Co., Ltd China Telecom Hangzhou, China · China Telecom Xi’an, China
PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology
Aug 26, 2026Sheethal Bhat, Mahfuzur Rahman Chowdhury, Paula Andrea Perez-Toro +4Multimodal Learning Missing-Modality Learning
Pattern Recognition Lab, Friedrich-Alexander-Universität, Erlangen-Nürnberg, Germany. · Department of Neurology, Klinikum Nürnberg, Paracelsus Medical University, Nürnberg, Germany. · Department of Radiology, LMU University Hospital, LMU Medizin, Ludwig-Maximilians-Universität München, Munich, Germany. +1
Asymmetric Cross-Modal Fine-Grained Visual Categorization: ACF-Net and the BirdPro Benchmark
Aug 26, 2026Bohan Deng, Shuo Ye, Zitong YuAudio-Visual Understanding Audio-Visual Representation Learning
Great Bay University
LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learning
Aug 25, 2026Xunkai Li, Zekai Chen, Zhengyu Wu +6Multimodal Graph Learning Multimodal Fusion
Beijing Institute of Technology, Beijing, China · Ant Group, Hangzhou, China
Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models
Aug 25, 2026Yingshu Li, Yunyi Liu, Zhanyu Wang +4Radiology Report Generation Radiology VLMs
University of Sydney Sydney, Australia · University of Wollongong Wollongong, Australia · University of Adelaide Adelaide, Australia
Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders
Aug 19, 2026Kaizhen Tan, Xin Xu, Siru Tao +4World Models Cross-Modal Alignment
New York University, New York, NY, USA · Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models
Aug 18, 2026Zongyang Qiu, Yihan Wu, Kaixuan Fan +2Unified Multimodal Models Multimodal Generation
The Hong Kong University of Science and Technology (Guangzhou) · Columbia University · MMLab, CUHK +1
XSA-MAD: Cross-modal Semantic Alignment for Morphing Attack Detection
Aug 14, 2026Jie Jin, Mahiro Tokumasu, Yu Makino +2Image Forgery Detection Face Presentation Attack Detection
Graduate School of Science and Technology Shizuoka University · Shizuoka University · RIKEN AIP
GALA: Generation-Aware Cross-Modal Alignment for Text-to-Time-Series Synthesis
Aug 13, 2026Haochen Zhang, Gengwei Zhang, Laura Yao +2Time Series Generation Conditional Generative Modeling
UNITES Lab, University of North Carolina at Chapel Hill
Warping Earth Observations for better ice labeling in the Marginal Marginal Ice Zone
Aug 12, 2026Tom Kelly, Martin S. J. RogersRemote Sensing Image Segmentation Cross-Modal Alignment
British Antarctic Survey, Cambridge
Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics
Aug 11, 2026Ruochen Liu, Wei LouSpatial Gene Expression Prediction Gene Expression Prediction from Histopathology Images
Faculty of Science and Engineering University of Liverpool Liverpool, United Kingdom · College of Mathematical Medicine Zhejiang Normal University Jinhua, China
Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration
Aug 11, 2026Qi Ming, Yuyang Wang, Mingjing Zhao +7Multispectral Object Detection Cross-Modal Alignment
Beijing University of Technology, China · Central South University, China · Beijing Electronics Science & Technology Institute +4
UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment
Aug 10, 2026Zijian Gu, Weikai Lin, Shuang Zhou +2Medical Diagnosis Cross-Modal Alignment
University of Central Florida · Orlando, FL, United States · University of Rochester +5
Warp-free Cross-view Geo-localization via Feature-space Consensus Mining
Aug 10, 2026Zhuo Song, Lian Xu, Runqing Jiang +4Multi-View Consistency Cross-View Geo-Localization
Sun Yat-sen University, Shenzhen, China · The University of Western Australia, Perth, Australia
GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views
Aug 10, 2026Jiahui Cui, Yan Zhao, Kan Wei +4Vision-Language Models Fine-Grained Image Retrieval
Aerospace Information Research Institute, Chinese Academy of Sciences · Key Laboratory of Target Cognition and Application Technology (TCAT), Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation
Aug 9, 2026Qiang Hu, Yuxuan Luo, Yingjie Guo +4Radiology Report Generation Medical Report Generation
Huazhong University of Science and Technology
MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval
Aug 8, 2026Fulong Liu, Liang Xu, Chengqun Yang +3Cross-Modal Alignment Information Retrieval
Shanghai Jiao Tong University · Beijing Zhongguancun Academy
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Aug 7, 2026Ivan Majic, Zexian Huang, Franziska Hübl +5Cross-Modal Alignment Multimodal Large Language Models
Graz University of Technology, Austria · University of Vienna, Austria · University of Liverpool, UK
Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification
Aug 7, 2026Yujian Zhao, Yukang Zhao, Hankun Liu +4Cross-Modal Alignment Multimodal Contrastive Learning
School of Artificial Intelligence, Beihang University · School of Humanities and Social Sciences, Beihang University · School of Computer Science and Engineering, Beihang University +1
UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction
Aug 5, 2026Bo Kong, Liruiz Jia, Yi Liang +5Relation Extraction Cross-Modal Alignment
Xinjiang University · Urumqi, China · Chongqing Jiaotong University +1
Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport
Aug 4, 2026Yixuan Florence Wu, Yilun Zhu, Naichen ShiCross-Modal Alignment Multimodal Embedding
Department of Engineering Sciences and Applied Mathematics Northwestern University Evanston, IL 60208, USA · Deparment of Electrical Engineering and Computer Science University of Michigan Ann Arbor, MI 48109, USA · Department of Industrial Engineering and Management Sciences Department of Mechanical Engineering Northwestern University Evanston, IL 60208, USA

Sep 4, 2026J/K move · Enter open · S save

ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding

Kanglei Zhou, Chunyan Lan, Dongyang Li, Jun Zhu+1

Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China · Department of Computer Science and Technology, Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint Center for ML, Tsinghua University, Beijing, China

EEG-Based Visual Decoding Contrastive Learning Cross-Modal Alignment Cross-Subject EEG Decoding

Open paper AskPDF arXiv

Electroencephalogram (EEG) visual decoding aims to recover visual semantics from non-invasive neural time-series signals, for which robust alignment between noisy neural responses and stable semantic representations is key to achieving high-performance decoding. Despite recent advances in contrastive learning, robust EEG decoding remains challenging because existing methods rely on fixed visual or textual anchors whose semantic relations may become misaligned with EEG representations that vary across trials, subjects, and learning stages. Our empirical evidence shows that this instability appears across both standard EEG decoding protocols and more challenging robustness settings, including strict cross-subject transfer and realistic personalized continual adaptation. We provide a formal analysis showing that fixed semantic supervision can bias optimization when EEG-specific relations evolve, and that structure-agnostic perturbations may distort semantically important EEG components. To address these issues, we propose Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment. ProCA progressively refines class-level contrastive supervision from frozen vision-language priors to EEG-aware semantic relations, and introduces structure-consistent interpolation to constrain feature mixing according to channel-wise and temporal importance. Across subject-dependent, subject-independent, strict cross-subject transfer, and continual adaptation settings, ProCA achieves average relative Top-1/Top-5 gains of 7.4%/3.9%, 10.0%/4.6%, 28.1%/17.8%, and 16.8%/11.6%, respectively.