Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution
Sep 24, 2026Jiahong Liu, Ming Shen, Xiaohao Liu +4Cross-Modal Alignment Hyperbolic Representation Learning
The Chinese University of Hong Kong · National University of Singapore · Yale University +1
M $^2$ PFN: End-to-End Disentangled Alignment for Generalizable Multimodal In-Context Learning in Alzheimer's Disease
Sep 23, 2026Lujia Zhong, Shuo Huang, Jianwei Zhang +2Multimodal Disentangled Representation Learning Multimodal ICL
Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA 90033, USA · Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA 90089, USA · Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA 90089, USA
S2A:Semantic-to-Spatial Alignment for Alignment-Free RGB-T Salient Object Detection
Sep 23, 2026Qiangqiang Zhou, Yang Luo, Yong Chen +1Cross-Modal Alignment Salient Object Detection
Jiangxi Normal University
Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification
Sep 22, 2026Akshit Sharma, Prashant W. PatilMultimodal Robustness Cross-Modal Alignment
CVPR Lab, MFSDSAI Indian Institute of Technology Guwahati
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Missing-Modality Learning Cross-Modal Alignment
Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1
Incentive Noise and Structural Prior Infusion for Multi-modal Object Re-Identification
Sep 21, 2026Weixiang Zhou, Yuhao Wang, Xingguo Xu +4Multimodal Robustness Cross-Modal Alignment
Dalian University of Technology, China · New York University, USA · Nanjing University of Science and Technology, China +1
Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities
Sep 21, 2026Chenming Shang, Yujin Tang, Jun Jie Ou Yang +3Cross-Modal Alignment Transfer Learning
Department of Computer Science, DARTMOUTH COLLEGE
Adaptive Cortically Constrained EEG-Vision Alignment for Zero-Shot Brain-to-Image Retrieval
Sep 21, 2026Ye Wang, Haokun Ren, Wei Wu +4Electroencephalography EEG-Based Visual Decoding
School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · School of Medicine, Shanghai Jiaotong University, Shanghai, China · National Center for Applied Mathematics in Chongqing, Chongqing Normal University, Chongqing, China +1
Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment
Sep 20, 2026Chenxiao Li, Yunhe Feng, Dongfang Liu +3Cross-Modal Alignment Uncertainty-Aware Multimodal Fusion
University of North Texas · Purdue University · Meta Inc
Low resource cross-modal alignment using HGNN to enhance speech representation
Sep 19, 2026Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia YontaCross-Modal Alignment Speech Processing
Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.
SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment
Sep 18, 2026Zhibo Zhang, Qijie Wang, Zengqiang YanMedical Image Segmentation Reasoning Segmentation
School of Electronic Information and Communications, Huazhong University of Science and Technology
Beyond Random Couplings: Contrastive Noise Alignment in Generative Flows
Sep 16, 2026Lennart Wittke, Vinicius AzevedoFlow Matching Cross-Modal Alignment
ETH Zürich · Disney Research | Studios
Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery
Sep 16, 2026Qi Ming, Xiaxin Yuan, Jiahuan Zhou +6Object Detection Infrared Object Detection
Beijing University of Technology · University of Science and Technology of China · Peking University +5
Hub-Spectral Activation of Latent Multimodal Knowledge
Sep 15, 2026Ying Guo, Haidong Chen, Linrui Xu +7Cross-Modal Representation Learning Cross-Modal Alignment
Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, North China University of Technology, Beijing 100144, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China · NExT++ Research Centre, National University of Singapore, Singapore +2
Bridging the Perceptual Gap: Residual-Enhanced Downscaling and Manifold-Aware Perception Alignment Adaptation for NR-IQA
Sep 15, 2026Yu Li, Zhengran Shen, Yachun Mi +2VLM Adaptation Cross-Modal Alignment
Harbin Institute of Technology, Harbin, China.
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Sep 15, 2026Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8Cross-Modal Representation Learning Cross-Modal Alignment
Meta AI
A Language-Guided Multimodal Foundation Model for Zero-Shot and Multi-Task Brain Signal Analysis
Sep 14, 2026Mingzhi Chen, Yiyu Gui, Guibo Luo +1Cross-Modal Alignment Multimodal Foundation Models
New Cornerstone Science Laboratory, Guangdong Provincial Key Laboratory of In-Memory Computing Chips, School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Center for Brain Inspired Intelligence, Chinese Institute for Brain Research (CIBR), Beijing, China · New Cornerstone Science Laboratory, Beijing Advanced Innovation Center for Integrated Circuits, School of Integrated Circuits, Peking University, Beijing, China
Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval
Sep 14, 2026Anindya Nag, Ambuj Mehrish, Sebastiano VasconMultimodal IR Cross-Modal Alignment
Department of Environmental Science, Informatics and Statistics Ca’ Foscari University of Venice, Italy
Weakly Supervised Spatial Grounding for Discriminative Attention-Based Ultrasound-Histopathology Alignment in Prostate Cancer Grading
Sep 14, 2026Obed Korshie Dzikunu, Emma Willis, Mohammad Mahdi Abootorabi +7Computational Pathology Cross-Modal Knowledge Distillation
Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, Canada · Vector Institute, Toronto, Canada · School of Computing, Queen’s University, Kingston, Canada +3
Long-Tail Adaptive Flow Matching with Explicit Conditional Consistency Guidance for Precise Multimodal Face Synthesis
Sep 14, 2026Yushe Cao, Xuechao Zou, Xing Xi +3Multimodal Generation Diffusion Model Guidance
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China +1
One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling
Sep 13, 2026Geigh Zollicoffer, Minh Vu, Rajiv Ranasinghe +1Physical Consistency in Video Generation World Model Learning
1Theoretical Division, Los Alamos National Laboratory · 2Computing and Artificial Intelligence Division, Los Alamos National Laboratory · 3Earth and Environmental Sciences Division, Los Alamos National Laboratory
SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity
Sep 12, 2026Md Ashik Khan, Md Nahid SiddiqueMultimodal Anomaly Detection Cross-Modal Alignment
Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India · Knight Foundation School of Computing and Information Sciences Florida International University Miami, FL, USA
CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation
Sep 11, 2026Jinting Wang, Chenxing Li, Dong Yu +1Cross-Modal Alignment Hierarchical Representation Learning
HKUST(GZ) · Tencent
Multimodal Floorplan Encoding: Learning Dense Modality-Invariant Representations
Sep 11, 2026Xavier Anadón, Rémi Pautrat, Rui WangLatent Space Alignment Cross-Modal Representation Learning
University of Zaragoza, Spain · Microsoft, Switzerland
Inferring Dislocation Microstructures from X-ray Diffraction via Cross-Modal Contrastive Learning
Sep 11, 2026Benjamin Udofia, Nicolas Bertin, Markus StrickerCross-Modal Alignment Materials Science
Interdisciplinary Centre for Advanced Materials Simulation Ruhr University Bochum Bochum, Germany · Lawrence Livermore National Laboratory Livermore Carlifornia, United States of America
MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment
Sep 10, 2026Guangyu Chen, Qiwei Liang, Shaolong Zhu +7Mobile Manipulation Cross-Modal Alignment
1THU · 2Xspark AI · 3HKUST (GZ) +1
Prototype Matters: Modality-unified Prototype Self-distillation for Unsupervised Visible-infrared Person Re-identification
Sep 10, 2026Menglin Wang, Xiaojin GongContrastive Learning Cross-Modal Alignment
Nanjing Normal University Nanjing, China · Zhejiang University Hangzhou, China
Supervised Cross-Modal Feature Alignment for Zero-Wearable Freezing of Gait Detection in Parkinsonism
Sep 8, 2026Aryan Singh, Chandan BiswasCross-Modal Knowledge Distillation Cross-Modal Alignment
NeuroAI Fusion Labs Kolkata, India
Cross-modal learning for SAR target recognition using optical vision foundation models
Sep 7, 2026Lucas Hirsch, James R. Hopgood, Javid Khan +2Cross-Modal Alignment Cross-Modal Representation Transfer
School of Engineering, University of Edinburgh, United Kingdom · Leonardo UK, United Kingdom · School of Engineering and Physical Sciences, Heriot-Watt University, United Kingdom
GLASS: Graph-Language Alignment with Spherical Scoring for Transferable Graph-Level Anomaly Detection
Sep 4, 2026Xudong Wang, Chris Ding, Tongxin Li +1Few-Shot Anomaly Detection Graph-Based Anomaly Detection
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

Sep 24, 2026J/K move · Enter open · S save

Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution

Jiahong Liu, Ming Shen, Xiaohao Liu, Rex Ying+3

The Chinese University of Hong Kong · National University of Singapore · Yale University+1

Cross-Modal Alignment Hyperbolic Representation Learning Multimodal Learning

Open paper AskPDF arXiv

Existing continual-learning methods protect parameters, replayed examples, or Euclidean feature subspaces. When applied to hyperbolic multimodal models, they do not explicitly preserve the Lorentz geometry that jointly encodes within-modality similarity, cross-modal correspondence, and semantic hierarchy; sequential updates can therefore retain task scores while still distorting previously learned relations. We address this gap with Hyperbolic Multimodal Continual Learning (HMCL). We show that preserving the old multimodal geometry amounts to restricting all modalities to one shared hyperbolic isometry, which induces a family of admissible first-order parameter changes. We formulate a joint closest-admissible (CA) correction that retains the shared rotation best matching the candidate modal updates; its minimal-rotation (MR) special case fixes this rotation to zero. Both variants correct the displacement realized by AdamW, and task anchoring bounds within-task accumulation while preserving learning freedom. Across a unified 16-task classification-retrieval stream with three hyperbolic backbones, HMCL improves final performance and backward transfer over sequential fine-tuning and four continual-learning baselines; HMCL-CA gives the highest Overall score on every backbone. A modality-extended stream confirms the retrieval gains. Representation analyses find 81.2 to 95.5 percent less radial, angular, cross-modal, and paired-distance drift; ImageNet-WordNet results show better semantic ancestry and radial hierarchy.