Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

GMAIL: Generative Modality Alignment for generated Image Learning
Feb 17, 2026Shentong Mo, Sukmin YunCross-Modal Alignment VLM Adaptation
Department of Machine Learning, CMU, USA · Department of Machine Learning, MBZUAI, UAE · Department of Artificial Intelligence, Hanyang University ERICA, South Korea
Quantifying the Gap between Understanding and Generation within Unified Multimodal Models
Feb 2, 2026Chenlong Wang, Yuhang Chen, Zhihan Hu +4Unified Multimodal Models Cross-Modal Alignment
Independent Researcher · University of Maryland · University of Waterloo +1
The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence
Jan 27, 2026Yichao Cai, Zhen Zhang, Yuhang Liu +1Cross-Modal Learning Contrastive Learning
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
Multimodal rumor detection enhanced by external evidence and forgery features
Jan 21, 2026Han Li, Hua SunFake News Detection Cross-Modal Alignment
Information Engineering School of Dalian Ocean University,Dalian Liaoning, China
M2I2HA: Multi-modal Object Detection Based on Intra- and Inter-Modal Hypergraph Attention
Jan 21, 2026Xiaofan Yang, Yubin Liu, Wei Pan +5Cross-Modal Alignment Hypergraph Neural Networks
AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detection
Jan 19, 2026Shiming Wang, Holger Caesar, Liangliang Nan +1Autonomous Driving Perception 3D Object Detection
Intelligent Vehicles Group, Delft University of Technology · 3D Geoinformation Group, Delft University of Technology
GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models
Jan 12, 2026Zhankai Ye, Bofan Li, Yukai Jin +5Motion Representation Learning Cross-Modal Alignment
Florida State University · Texas Tech University · William & Mary University
MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
Dec 15, 2025Yannan He, Garvita Tiwari, Xiaohan Zhang +4Latent Diffusion Models Text-to-Motion Generation
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +2
Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis
Nov 22, 2025Wenjing Liu, Qin Ren, Wen Zhang +2Cancer Survival Prediction Survival Analysis
Stony Brook University, Stony Brook, NY, USA · Johns Hopkins University, Baltimore, MD, USA · Brookhaven National Laboratory, Upton, NY, USA
FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection
Nov 10, 2025Yulin Chen, Zeyuan Wang, Tianyuan Yu +2VLM Robustness Adversarial Attacks on VLMs
Laboratory for Big Data and Decision, National University of Defense Technology, Changsha 410073, China
NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment
Nov 10, 2025Wenjiang Zhang, Sifeng Wang, Yuwei Su +3EEG-Based Visual Decoding Cross-Subject EEG Decoding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China · Queen Mary School Hainan, Beijing University of Posts and Telecommunications, Hainan 570100, China
SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment
Nov 3, 2025Xinyu Mao, Junsi Li, Haoji Zhang +2Cross-Modal Alignment Cross-Modal Feature Matching
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China.
Uncovering Grounding IDs: How External Cues Shape Multi-Modal Binding
Sep 28, 2025Amirmohammad Izadi, Hosein Hasani, Fatemeh Askari +4Multimodal Grounding Cross-Modal Alignment
The Platonic Universe: Do Foundation Models See the Same Sky?
Sep 23, 2025UniverseTBD, :, Trinidad Borrell +11Cross-Modal Alignment Multimodal Model Evaluation
Independent Researcher · AstroAI · Harvard-Smithsonian CfA +4
Reconstruction Alignment Improves Unified Multimodal Models
Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal Models Cross-Modal Alignment
1UC Berkeley · University of Washington · 3Duke University
FairSSL: Fair Multimodal Self-Supervised Learning
Aug 22, 2025Jiaee Cheong, Abtin Mogharabin, Paul Liang +2Cross-Modal Learning Representation Learning
Harvard University · University of Cambridge · Middle East Technical University +1
EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation
Aug 22, 2025Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng +2Cross-Modal Alignment Multimodal Embedding
College of Computing and Data Science, Nanyang Technological University, Singapore
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding
Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal Learning VLM Adaptation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1
Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging
Jun 19, 2025Jiawen Yang, Shuhao Chen, Shengtao Zhang +2Unsupervised Domain Adaptation Cross-Modal Alignment
Southern University of Science and Technology · SZ DJI Technology Co., Ltd
Multimodal Representation Alignment for Cross-modal Information Retrieval
Jun 10, 2025Fan Xu, Luis A. LeivaCross-Modal Learning Multimodal IR
aDepartment of Computer Science, University of Luxembourg, 6, avenue de la Fonte, Esch-sur-Alzette, L-4364, Luxembourg
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
May 26, 2025Sanghyuk Chun, Olga RussakovskyCross-Modal Alignment Multimodal Model Evaluation
1Princeton University
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
May 15, 2025Chunyu Ye, Yunhao Zhang, Jingyuan Sun +3Brain-Computer Interfaces Neural Decoding
1State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Department of Computer Science, The University of Manchester +1
ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Dataset
May 15, 2025Chengsen Wang, Qi Qi, Zhongwen Rao +2Cross-Modal Learning Multivariate Time Series Forecasting
Beijing University of Posts and Telecommunications · Huawei Noah’s Ark Lab
Purify then Guide: Rethinking Domain Generalization for Multimodal Face Anti-Spoofing
May 14, 2025Yingjie Ma, Xun Lin, Zitong Yu +7Cross-Modal Alignment Domain Generalization
College of Computer Science and Software Engineering, Shenzhen University · School of Computing and Information Technology, Great Bay University · The Chinese University of Hong Kong +5
On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
Apr 14, 2025Yichao Cai, Yuhang Liu, Erdun Gao +4Cross-Modal Representation Learning Cross-Modal Alignment
Australian Institute for Machine Learning, The University of Adelaide, SA 5000, Australia

Feb 17, 2026J/K move · Enter open · S save

GMAIL: Generative Modality Alignment for generated Image Learning

Shentong Mo, Sukmin Yun

Department of Machine Learning, CMU, USA · Department of Machine Learning, MBZUAI, UAE · Department of Artificial Intelligence, Hanyang University ERICA, South Korea

Cross-Modal Alignment VLM Adaptation Vision-Language Alignment

Open paper AskPDF arXiv

Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.