Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

CrossWeave: Cross-Modal Scene and Instance Retrieval from Sparse 2D-3D Alignment
Oct 29, 2026Aadith Warrier, Gnana Prakash Punnavajhala, Siddharth Tourani +3Cross-Modal Retrieval Cross-Modal Alignment
IIIT Hyderabad · University of Heidelberg · MBZUAI +1
Proving and Improving the Platonic Representation Hypothesis with JEPAs
Oct 29, 2026Randall Balestriero, Yann LeCunPlatonic Representation Hypothesis Cross-Modal Alignment
Brown University · AMI Labs · New York University
Uncertainty-Guided Cross-Modal Alignment for Diffusion-Based IoT Intrusion Detection
Oct 29, 2026Mahdi Arab Loodaricheh, Anita Raja, Mohammad Hossein ManshaeiIoT Intrusion Detection Cross-Modal Alignment
Department of Computer Science, Hunter College and The Graduate Center, CUNY, New York, NY, USA
BASILISC: Bayesian Alignment of Siamese-Integrated Latents for Inferring Single-cell Correspondence
Oct 28, 2026Sepideh Saran, Mahsa Ghanbari, Uwe OhlerCross-Modal Alignment Multi-Omics Data Integration
The Berlin Institute for Medical Systems Biology, Max Delbrück Center for Molecular Medicine, Germany · Department of Electrical Engineering and Computer Science, Technical University of Berlin, Germany · Department of Biology, Humboldt University of Berlin, Germany +1
Separability Is Not Alignability: Representations for Spectrum-to-Molecule Retrieval
Oct 27, 2026Karol Musielinski, Fanurs C. E. Teh, Takahiko Koyama +1Molecular Representation Learning Mass Spectrometry
Auton Lab, Carnegie Mellon University · University of Warsaw · Keio University
Correspondence Is Not Transfer: A Pairing-Controlled Study of Clinical Audio-Metadata Alignment
Oct 26, 2026Liu He, Xi Wang, Jiahong Yuan +1Cross-Modal Alignment Audio Understanding
University of Science and Technology of China, China · Independent Researcher · University of Cambridge, UK
Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
GeoLEON-CLIP: Graph-Anchored Multimodal Contrastive Learning for Geographic Region Representation
Oct 23, 2026Radoslaw Malek, Szymon Soltysiak, Aleksandra Kawala-Sterniuk +1Geospatial Representation Learning Multimodal Contrastive Learning
Wrocław University of Science and Technology, Department of Artificial Intelligence, Wrocław, Poland
Hierarchical Fine-Grained Cross-Modal Alignment via Dynamic Granularity Discovery and Optimal Transport
Oct 22, 2026Pengfeng Song, Shaozhong Zhang, Pengfei ShaoImage-Text Retrieval Cross-Modal Alignment
College of Information and Intelligent Engineering,Zhejiang Wanli University,Ningbo,315100,Zhejiang,China.
SenseAlign: Understanding Vision and Touch through Language at Test Time
Oct 20, 2026Chuyang Ye, Qinting Jiang, Jingyan JiangMultimodal Understanding Cross-Modal Alignment
New York University · Tsinghua Shenzhen International Graduate School, Tsinghua University · Shenzhen Technology University
ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding
Oct 20, 2026Kieren Yu, Chang Huang, Ziyang Liu +1EEG-Based Visual Decoding Cross-Modal Alignment
Preserving the Manifold: Frozen Expert Alignment for Label-Efficient SAR Scene Understanding
Oct 18, 2026Eli A. Underwood, I. Taylor McKechnie, Leslie M. CollinsRemote Sensing VLMs Cross-Modal Alignment
Duke University
CIIR - Class-Indexed Interface Representation Learning
Oct 16, 2026Doo Yon Kim, Christian WallravenCross-Modal Representation Learning Cross-Modal Representation Transfer
ODSS-Net: Orthogonally Decoupled Shared-Specific Representations for Multi-Modal Object Re-Identification
Oct 16, 2026Zhifeng Ma, Yuhong Feng, Chongyi Huang +2Multimodal Disentangled Representation Learning Multimodal Fusion
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · Tencent, Shenzhen, China
MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal Learning Multimodal Foundation Models
University of California, Riverside · Amazon
FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation
Oct 14, 2026Hang Ren, Peidong Wang, Yichen Gao +4Emotional Support Conversation Multi-Objective Language Model Alignment
School of Computer Science and Engineering, Northeastern University, China
Revisiting Triplet Contrastive Learning for 3D Shape Understanding
Oct 14, 2026Chaoyang Zhu, Zhe Liao, Jiayi Zhang +33D Representation Learning Cross-Modal Alignment
Xiamen University · Shanghai AI Lab
Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations
Oct 13, 2026Junkai Liu, Haofan Wu, Nay Aung +3Medical Image Synthesis Video Diffusion Models
School of Engineering, University of Birmingham · William Harvey Research Institute, Queen Mary University London · Division of Cardiology, Johns Hopkins University School of Medicine
To Align or Not To Align: Check Your COMPASS Before You Train
Oct 13, 2026Fabian Gröger, Petar Damjanović, Maria BrbićLanguage Model Merging Cross-Modal Alignment
EPFL · University of Basel · HSLU
Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation
Oct 13, 2026Hu Cao, Yifeng Cheng, Mu He +3Multimodal Semantic Segmentation Multimodal Fusion
School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany.
Omni-Modal LLMs Know More Than They Say
Oct 12, 2026Na Min An, Taekyung Kim, Jaemyung Yu +2Multimodal Large Language Models Cross-Modal Alignment
KAIST AI · NAVER AI Lab
Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment
Oct 12, 2026Xizhe Zhang, Jingfeng Zhang, Meng-Fen ChiangCross-Modal Alignment Multimodal KGs
School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan
Rethinking Human Intent-to-CAD: Parametric CAD Model Generation via Cooperative Multi-Task Alignment and Spatial-Aware Reinforcement Learning
Oct 11, 2026Qingwang Zhang, Jiahao Li, Xiangdong ZhouParametric CAD Modeling Text-to-CAD Generation
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
TartanMatch: Towards Universal Dense Correspondence Across Modalities
Oct 10, 2026Hyeokjoon Kwon, Jiting Cai, Ruogu Li +7Cross-Modal Feature Matching Unified Multimodal Models
The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Kritan Bhandari is with Fisk University, Nashville, TN, USA
ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment
Oct 9, 2026Alex Junho LeeCross-Modal Alignment Latent Space Alignment
School of Mechanical Systems Engineering, Sookmyung Women’s University, Seoul 04310, South Korea
Representation Alignment from a Few Paired Data via Distilled Optimal Transport
Oct 8, 2026Julien Kordes, Reuben DorentCross-Modal Alignment Cross-Modal Representation Learning
Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, 75013 Paris, France
PXtal: Learning to Align Powder X-Ray Diffraction and Crystal Structures under Information Asymmetry across Modalities
Oct 7, 2026Zhuoran Yang, Christopher M. Collins, Bei Peng +3Cross-Modal Alignment Multimodal Learning
School of Computer Science and Informatics, University of Liverpool, Liverpool, UK · Leverhulme Research Centre for Functional Materials Design, Materials Innovation Factory, University of Liverpool, Liverpool, UK · School of Computer Science, University of Sheffield, Sheffield, UK +1
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Oct 7, 2026Arshia Hemmat, Amirhossein Vahidi, Amitis Shidani +4Compositional T2I Generation T2I Diffusion Models
Wellcome Sanger Institute · Cambridge Stem Cell Institute, University of Cambridge · Cambridge Centre for AI in Medicine, University of Cambridge +6
Unpaired Canonical Correlation Analysis
Oct 7, 2026Nir Ben-Ari, Ronen Talmon, Uri ShahamCanonical Correlation Analysis Multi-View Learning
Department of Computer Science and AI Bar-Ilan University Ramat-Gan, Israel · Electrical and Computer Engineering Technion Haifa, Israel
Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data
Oct 7, 2026Dominik Schnaus, Thomas Dagès, Daniel Cremers +2Cross-Modal Alignment Cross-Modal Representation Transfer
TU Munich · MCML · Ulm University +2

Oct 29, 2026J/K move · Enter open · S save

CrossWeave: Cross-Modal Scene and Instance Retrieval from Sparse 2D-3D Alignment

Aadith Warrier, Gnana Prakash Punnavajhala, Siddharth Tourani, Muhammad Haris Khan+2

IIIT Hyderabad · University of Heidelberg · MBZUAI+1

Cross-Modal Retrieval Cross-Modal Alignment

Open paper AskPDF arXiv

Cross-modal scene and instance retrieval aims to retrieve a corresponding 3D environment or object given query 2D images, serving as a foundational mechanism for spatial reasoning in robotics and AR/VR. Existing methods achieve this by learning alignment only at the level of whole scenes or objects, which provides no supervision for the fine-grained local structure within them. In this work, we show that supervision on a set of sparse 2D-3D correspondences, without scene- or instance-level contrastive objectives, is sufficient to train a single model for both scene- and instance-level retrieval. Specifically, we propose

\textbf{CrossWeave}

, a novel framework for cross-modal retrieval that learns sparse patch-point alignment, which alone outperforms existing methods. An optimal transport-based aggregator over the aligned features gives further gains by clustering structurally redundant regions, freeing descriptor capacity for semantically distinct local features. CrossWeave achieves state-of-the-art performance on both scene- and instance-level retrieval on ScanNet and 3RScan with a

\textit{single model}

, whereas existing methods require separate models for each task. Our results suggest that whole-scene encoding is unnecessary for cross-modal retrieval and that sparse local correspondence supervision is a more effective and flexible alternative.