Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction
Apr 18, 2026Bowen Peng, Yongxiang Liu, Jie Zhou +4Cross-Modal Learning Masked Autoencoders
College of Electronic Science and Technology, National University of Defense Technology (NUDT), Changsha 410073, China. · Beijing Institute of Remote Sensing Information, Beijing 100192, China.
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Apr 18, 2026Zibo Shao, Baochen Xiong, Xiaoshan Yang +4Cross-Modal Learning Cross-Modal Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Pengcheng Laboratory, Shenzhen 518066, China · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing 100049, China +1
CrossFlowDG: Bridging the Modality Gap with Cross-modal Flow Matching for Domain Generalization
Apr 18, 2026Antonios Kritikos, Nikolaos Spanos, Athanasios VoulodimosCross-Modal Learning Cross-Modal Alignment
National Technical University of Athens
UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
Apr 17, 2026Hangke Sui, Yuqing Wang, Minh N DoContrastive Learning Cross-Modal Alignment
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC +1
Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinCross-Modal Learning Modality Imbalance
Dalhousie University, Halifax NS, Canada
Causal Bootstrapped Alignment for Unsupervised Video-Based Visible-Infrared Person Re-Identification
Apr 17, 2026Shuang Li, Jiaxu Leng, Changjiang Kuang +3Cross-Modal Alignment Unsupervised Representation Learning
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China
Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo
Apr 16, 2026Ninghui Xu, Fabio Tosi, Lihui Wang +5Event-Based Vision Cross-Modal Alignment
School of Instrument Science and Engineering, Southeast University, State Key Lab of Comprehensive PNT Network and Equipment Technology, Key Lab of Micro-Inertial Instrument and Advanced Navigation Technology, MOE · University of Bologna · Department of Computer Science and Engineering, University of Bologna +2
Geo2Sound: A Scalable Geo-Aligned Framework for Soundscape Generation from Satellite Imagery
Apr 16, 2026Kunlin Wu, Yanning Wang, Haofeng Tan +6Cross-Modal Alignment Audio Generation
The Hong Kong University of Science and Technology (Guangzhou), China · University of South Carolina, USA · University of Canterbury, New Zealand +2
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1
SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker
Apr 14, 2026Junbin Su, Ziteng Xue, Shihui Zhang +3Multimodal Visual Object Tracking Efficient Multimodal Inference
School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3
UNIGEOCLIP: Unified Geospatial Contrastive Learning
Apr 13, 2026Guillaume Astruc, Eduard Trulls, Jan Hosang +2Geospatial Representation Learning Multimodal Embedding
LASTIG, Univ Gustave Eiffel, IGN, ENSG, France · CNES, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, Institut Polytechnique de Paris, Marne-la-Vallée, France +1
Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering
Apr 13, 2026Nirmalendu Prakash, Narmeen Fatimah Oozeer, Xin Su +8Cross-Modal Retrieval Cross-Modal Alignment
Singapore University of Technology and Design · Martian · Thoughtworks +2
Efficient Training for Cross-lingual Speech Language Models
Apr 13, 2026Yan Zhou, Qingkai Fang, Yun Hong +1Speech Language Models Cross-Modal Alignment
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
EmergentBridge: Improving Zero-Shot Cross-Modal Transfer in Unified Multimodal Embedding Models
Apr 13, 2026Jincheng Xie, Xingchen Xiao, Runheng Liu +3Cross-Modal Learning Multimodal Embedding
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JD iCity, JD Technology, JD Intelligent Cities Research, Beijing, China
Omnimodal Dataset Distillation via High-order Proxy Alignment
Apr 12, 2026Yuxuan Gao, Xiaohao Liu, Xiaobo Xia +1Dataset Distillation Multimodal Dataset
University of Science and Technology of China · Xidian University · National University of Singapore +1
FashionMV: Product-Level Composed Image Retrieval with Multi-View Fashion Data
Apr 11, 2026Peng Yuan, Bingyin Mei, Hui ZhangComposed Image Retrieval Information Retrieval
Tsinghua University Beijing, China
MAGE: Modality-Agnostic Music Generation and Target-Source Extraction
Apr 10, 2026Muhammad Usama Saleem, Tejasvi Ravi, Tianyu Xu +4Controllable Music Generation Cross-Modal Alignment
Google · University of North Carolina at Charlotte
Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization
Apr 9, 2026Benjamin Léger, Kazem Meidani, Christian GagnéLatent Space Optimization Symbolic Regression
IID / Mila, Université Laval, Québec, Canada · Department of Mechanical Engineering, Carnegie Mellon University Canada-CIFAR AI Chair
Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
Apr 9, 2026Yuan Xie, Jiaqi Song, Guang Qiu +4Automatic Speech Recognition Cross-Modal Alignment
Advanced Intelligent Systems Group, NIO
Sheaf-Laplacian Obstruction and Projection Hardness for Cross-Modal Compatibility on a Modality-Independent Site
Apr 8, 2026Tibor SlobodaCross-Modal Alignment
UPAI, Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava, Bratislava, Slovakia. · aleph0 s. r. o., Bratislava, Slovakia.
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Apr 7, 2026Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9Domain Adaptation for ASR Cross-Modal Alignment
Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland +2
SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection
Apr 7, 2026Letian Bai, Chengyu Tao, Juan DuAnomaly Detection Multimodal Anomaly Detection
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · College of Mechanical and Vehicle Engineering, Hunan University, Changsha, 410082, China · The Hong Kong University of Science and Technology, Hong Kong SAR, 999077, China
Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs
Apr 7, 2026Hongcheng Liu, Yuhao Wang, Zhe Chen +5Cross-Modal Alignment Multimodal Large Language Models
Shanghai Jiao Tong University
The Indra Representation Hypothesis for Multimodal Alignment
Apr 6, 2026Jianglin Lu, Hailing Wang, Kuo Yang +3Representation Learning Cross-Modal Alignment
Department of Electrical and Computer Engineering, Northeastern University · Adobe Research · Khoury College of Computer Science, Northeastern University
Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning
Apr 5, 2026Donghuo Zeng, Hao Niu, Masato TayaCanonical Correlation Analysis Audio-Visual Representation Learning
KDDI Research, Inc., Saitama, Japan
User-Aware Conditional Generative Total Correlation Learning for Multi-Modal Recommendation
Apr 3, 2026Jing Du, Zesheng Ye, Congbo Ma +2Multimodal Recommendation Multimodal Fusion
The University of New South Wales, Sydney, Australia · University of Melbourne, Melbourne, Australia · New York University Abu Dhabi, Abu Dhabi, UAE
Exploring Motion-Language Alignment for Text-driven Motion Generation
Apr 3, 2026Ruxi Gu, Zilei Wang, Wei WangText-to-Motion Generation Human Motion Generation
Department of Automation, University of Science and Technology of China, Hefei, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
Apr 2, 2026Jiachun Jin, Zetong Zhou, Xiao Yang +4Unified Multimodal Models Multimodal Reasoning
Shanghai Jiao Tong University · Tsinghua University · UCSD
ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding
Apr 1, 2026Lala Shakti Swarup Ray, Mengxi Liu, Alcina Pinto +4Human Activity Recognition Cross-Modal Alignment
DFKI, Germany · RPTU
Camouflage-aware Image-Text Retrieval via Expert Collaboration
Apr 1, 2026Yao Jiang, Zhongkuan Mao, Xuan Wu +2Image-Text Retrieval Cross-Modal Retrieval
College of Computer Science, Sichuan University · National Key Lab of Fundamental Science on Synthetic Vision, Sichuan University

Apr 18, 2026J/K move · Enter open · S save

MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction

Bowen Peng, Yongxiang Liu, Jie Zhou, Xiaodong Chen+3

College of Electronic Science and Technology, National University of Defense Technology (NUDT), Changsha 410073, China. · Beijing Institute of Remote Sensing Information, Beijing 100192, China.

Cross-Modal Learning Masked Autoencoders Cross-Modal Alignment Multimodal Pretraining Synthetic Aperture Radar Multimodal Contrastive Learning

Open paper AskPDF arXiv

Joint optical and Synthetic Aperture Radar (SAR) pretraining seeks cross-modal gains over representations learned from either modality alone. This is challenging because the modalities share only partial information, and their correspondence varies across pairs with scene context, acquisition timing, sensor configuration, and spatial resolution. In particular, finer resolution exposes more detailed modality-specific optical appearance and SAR backscatter, further complicating cross-modal supervision. Existing methods encourage cross-modal agreement through representation alignment or cross-modal prediction. Under simplified models of partial, pair-dependent correspondence, we show why a fixed alignment relation can be insufficient and characterize prediction ambiguity arising from target variation unresolved by the source representation. These analyses motivate CoDe-MAE, a masked-autoencoding framework for high-resolution optical-SAR joint pretraining. Its Conditioned Contrastive Learning (CCL) aligns cross-attention-conditioned representations, while Cross-modal Degraded-Target Prediction (CDP) predicts transformed targets to balance conditional predictability and retained supervision. We curate OSPretrain-1M, a 1M-sample optical-SAR corpus, and use CoDe-MAE to pretrain MMATR-X, which achieves leading transfer performance across optical-only, SAR-only, and dual-modal tasks. Controlled experiments on remote-sensing optical-SAR, optical-DEM, and indoor RGB-depth pairs further support these design principles across heterogeneous modality pairs. Code: https://github.com/scenarri/MMATR-X.