Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Federated Learning Cross-Modal Alignment
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes
Jul 14, 2026Minh-Quan Le, Armand Comas, Alexandros Lattas +7Masked Diffusion Models Markov Models
Google · Stony Brook University · Google DeepMind
Text-Aided Multi-Modal Panoptic Symbol Spotting for CAD Floor Plan Drawings
Jul 14, 2026Yan Gong, Bohao Li, Bowen Du +1Panoptic Segmentation Cross-Modal Alignment
aCCSE Lab, Beihang University, Beijing, 100191, China · bSchool of Transportation Science and Engineering, Beihang University, Beijing, 100191, China · cThe Hong Kong Polytechnic University, Hong Kong, China
Towards Predictive, Aligned, and Scalable Robot Learning
Jul 13, 2026Peijun Tang, Shangjin Xie, Baifu Huang +6Latent Action Learning Cross-Modal Alignment
Astribot Team
On the modality gap and the contrastive loss in multi-modal representation learning
Jul 12, 2026Fabian Mager, Hiba Nassar, Lars Kai HansenCross-Modal Alignment Multimodal Contrastive Learning
Department of Applied Mathematics and Computer Science Technical University of Denmark
Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Jul 12, 2026Ryota Kimura, Sangheon Park, Natalia Polouliakh +1Controllable Music Generation Cross-Modal Alignment
Sony Computer Science Laboratories · Keio University · Georgia Institute of Technology
Local Multimodal Music Alignment from Global Supervision
Jul 10, 2026Irmak Bukey, Zachary Novack, Jongmin Jung +2Cross-Modal Alignment Cross-Modal Retrieval
Carnegie Mellon University · University of California San Diego · Neutune +1
VTaMo: Video-Text Alignment Model for Sign Language Translation
Jul 10, 2026Junyi Hu, Zhewen He, Haomian Huang +2Sign Language Translation Cross-Modal Alignment
New York University Abu Dhabi, UAE · ChatSign Technology
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Jul 9, 2026Le Yang, Anoop K. Chandran, Jona Östreicher +8Cross-Modal Alignment Multimodal Embedding
Institute for Advanced Simulations · Institute for Advanced Simulations (IAS-9), Forschungszentrum Jülich GmbH, 52425 Jülich, Germany · Jülich Supercomputing Centre, Forschungszentrum Jülich +15
Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and A Large-scale Benchmark
Jul 9, 2026Qishun Wang, Yapeng Li, Zhengzheng Tu +2Cross-Modal Alignment Video Object Detection
Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei 230601, China
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Jul 8, 2026Kanglei Zhou, Ruizhi Cai, Xinning Wang +4Multimodal Robustness Cross-Modal Alignment
Dept. of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · State Key Lab of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Dept. of Rheumatology and Immunology, Children’s Hospital, Capital Institute of Pediatrics, Beijing, China
Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking
Jul 7, 2026Seungheon Doh, Minhee Lee, Sangmoon Lee +2Cross-Modal Alignment Multimodal Embedding
Graduate School of Culture Technology, KAIST, South Korea · Gaudio Lab, Inc, South Korea
DiCE-CIR: Direct Composition Learning for Efficient Zero-Shot Composed Image Retrieval
Jul 6, 2026Gwang-Ho Na, Ho-Joong Kim, Seong-Whan LeeCross-Modal Alignment Compositional Representation Learning
Department of Artificial Intelligence, Korea University, Anam-dong, Seongbuk-ku, Seoul 02841, Korea
Latent Clarity: Bridging World-Model Kinematics to Semantic Manifolds for Video Anomaly Anticipation
Jul 3, 2026Abu Anas Ibn SamadWorld Models Cross-Modal Alignment
Sigmind.ai Dhaka 1215, Bangladesh
Alignment Is All You Need For X-to-4D Generation
Jul 2, 2026Qiaowei Miao, Kehan Li, Yawei Luo +1Cross-Modal Alignment Multimodal Generation
Zhejiang University, Hangzhou 310027, China
CALM: Interpretable Cross-Modal Alignment for Biomarker Discovery from Unpaired Data
Jul 2, 2026Jueqi Wang, Zachary Jacokes, John Darrell Van Horn +3Cross-Modal Alignment Scientific Discovery
Department of Electrical and Computer Engineering, Boston University · School of Data Science, University of Virginia · Department of Psychology, University of Virginia +2
MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment
Jul 1, 2026Zhenhang Li, Xin Zhou, Hao Deng +1EEG Decoding Cross-Modal Alignment
Binghamton University · Massachusetts General Hospital
Attribute-Prompted Kernel Hashing for Unsupervised Data-Efficient Cross-Modal Retrieval
Jul 1, 2026Runhao Li, Xiaoxu Ma, Zhenyu Weng +5Cross-Modal Alignment Cross-Modal Retrieval
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, China · College of Computer and Information Engineering, Henan Normal University, China +2
Unsupervised Data-Efficient Cross-Modal Retrieval with Global-Neighborhood Alignment Hashing
Jun 30, 2026Runhao Li, Xiaoxu Ma, Zhenyu Weng +5Cross-Modal Alignment Cross-Modal Retrieval
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, China · College of Computer and Information Engineering, Henan Normal University, China +2
Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
Jun 30, 2026Jisung Park, Seohyeon Kang, Daeun Yoo +8Feature Superposition Neural Representation Geometry
1KAIST · 2Konyang University · 4UCL Queen Square Institute of Neurology & The Francis Crick Institute +1
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
Jun 30, 2026Weize Quan, Zhengwei Wu, Kai Wang +1Point Cloud Completion Cross-Modal Alignment
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing, China. · GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France.
Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation
Jun 29, 2026Kun Li, Shengxi Gui, Francesco Nex +1Remote Sensing Image Segmentation Cross-Modal Alignment
University of Liverpool, UK · University of Michigan, USA · University of Twente, NL +1
Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
Jun 29, 2026Dominik Winter, Dominik Vonficht, Loïc Le Bescond +6Computational Pathology Cross-Modal Alignment
AstraZeneca Computational Pathology and Biomarkers, Munich, Germany · AstraZeneca, Early Oncology and Translation Medicine, Cambridge, UK
Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders
Jun 29, 2026Chungpa Lee, Jihoon Kwon, Kyle Min +1Cross-Modal Alignment Sparse Autoencoders
Yonsei University, Korea · Seoul National University, Korea · Oracle, USA
HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Grounding
Jun 27, 2026Bo MaSelective Prediction Cross-Modal Alignment
Auckland University of Technology
Dual-Learning based Penalized Multi-Align Clustering for Multi-View Incomplete and Disorderly Data
Jun 26, 2026Liang Zhao, Shubin Ma, Bo Xu +1Missing-Modality Learning Cross-Modal Alignment
Dalian University of Technology · Dalian, Liaoning, China · Hainan University +1
See & Sniff: Learning Visuo-Olfactory Representations
Jun 25, 2026Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu +2Representation Learning Cross-Modal Representation Learning
Korea Advanced Institute of Science and Technology, Korea · Hankuk University of Foreign Studies, Korea · Ulsan National Institute of Science and Technology, Korea
What Does the Brain See? Multiview Neural Representations to Demystify the Brain-Visual Alignment
Jun 24, 2026Salini Yadav, Taveena Lotey, Pravendra Singh +1Representation Learning EEG-Based Visual Decoding
Department of CSE IIT Roorkee, India · Department of CSE IIT (ISM) Dhanbad, India
SAC $^2$ -Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition
Jun 24, 2026Xuepeng Zheng, Tong Chen, Chaoping Gui +3Cross-Modal Alignment Multimodal Emotion Recognition
College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China · Faculty of psychology, Southwest University, Chongqing, 400715, China · Chongqing Key Laboratory of Generic Technology and System of Service Robots, Southwest University, Chongqing, 400715, China +1
Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection
Jun 23, 2026Tian Qiu, Jifeng Shen, Xin ZuoCross-Modal Attention Cross-Modal Alignment
School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, China · School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China

Jul 15, 2026J/K move · Enter open · S save

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen, Ting Shu

School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia

Multimodal Federated Learning Cross-Modal Alignment Medical VQA Medical Image Classification Multimodal Foundation Models Federated Learning+1

Open paper AskPDF arXiv

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM

^2

, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM

^2

equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable

O(1/\sqrt{T})

convergence and generalization guarantees. FM

^2

further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.