Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

Precise and Simple Audio-to-Score Alignment
May 19, 2026Silvan Peter, Patricia Hu, Gerhard WidmerCross-Modal Alignment Music Information Retrieval
Institute of Computational Perception, Johannes Kepler University, Linz, Austria · LIT AI Lab, Linz Institute of Technology, Linz, Austria
MSAlign: Aligning Molecule and Mass Spectra Foundation Models for Metabolite Identification
May 19, 2026Paul Krzakala, Gabriel Melo, Camille Lançon +4Distribution Shift Cross-Modal Alignment
LTCI, Télécom Paris & CMAP, Ecole Polytechnique, Institut Polytechnique de Paris · LTCI, Télécom Paris, Institut Polytechnique de Paris · CEA, INRAE, MetaboHUB, Université Paris-Saclay +1
Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence
May 19, 2026Zebin He, Mingxin Yang, Shuhui Yang +43D Foundation Models Cross-Modal Alignment
1HKUST · 2Tencent Hunyuan
Semantic Generative Tuning for Unified Multimodal Models
May 18, 2026Songsong Yu, Yuxin Chen, Ying Shan +1Image Segmentation Unified Multimodal Models
Shanghai Jiao Tong University · Tencent ARCLab
S2Aligner: Pair-Efficient and Transferable Pre-Training for Sparse Text-Attributed Graphs
May 18, 2026Yuhan Wang, Haopeng Zhang, Yibo Ding +6Cross-Modal Alignment Text-Attributed Graph Learning
Beihang University, Beijing, China · Tianjin University, Tianjin, China
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
May 18, 2026Zeyu Chen, Jie Li, Kai HanCross-Modal Learning Multimodal Disentangled Representation Learning
Visual AI Lab, The University of Hong Kong
Brain-to-Image Retrieval and Reconstruction via Multimodal EEG Alignment
May 18, 2026Chi Kit Wong, Yan Liu, Haowen YanEEG-Based Visual Decoding Cross-Modal Alignment
LatentUMM: Dual Latent Alignment for Unified Multimodal Models
May 18, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +2Cross-Modal Learning Unified Multimodal Models
Carnegie Mellon University · William & Mary
Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
May 16, 2026Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque PrinceCross-Modal Alignment Multimodal Large Language Models
Qatar Computing Research Institute, HBKU, Qatar · York University, Canada
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
May 16, 2026Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3Missing-Modality Learning Multimodal Robustness
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data
May 16, 2026Shiwon Kim, Yu Rang ParkCross-Modal Learning Representation Learning
Yonsei University
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
May 16, 2026Wen-Hsin Tsai, Chia-Ming Lee, Yuk-Ying TungCross-Modal Learning Multimodal Robustness
Institute of Education, National Cheng Kung University · Institute of Intelligent System, National Yang Ming Chiao Tung University · Department of Computer Science, University at Albany, State University of New York
Cross-Domain Molecular Relational Learning: Leveraging Chemical Structure-Activity Analysis
May 16, 2026Peiliang Zhang, Jingling Yuan, Shiqing Wu +4Domain-Adversarial Learning Cross-Modal Representation Learning
Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things State Key Laboratory of Silicate Materials for Architectures Wuhan University of Technology Wuhan, Hubei, China · Yonsei University Seoul, Republic of Korea +3
Controlla: Learning Controllability via Graph-Constrained Latent Geometry
May 15, 2026Jamuna S. Murthy, Amin Karimi Monsefi, Rajiv RamnathDisentangled Representation Learning Multimodal Disentangled Representation Learning
Ramaiah Institute of Technology · The Ohio State University
SOLAR: Self-supervised Joint Learning for Symmetric Multimodal Retrieval
May 15, 2026Wenjie Yang, Hang Yu, Yuyu Guo +1Cross-Modal Learning Multimodal IR
Ant Group
Cross-Modal Registration Between 3D and 2D Fingerprints via Pose-Aware Unwrapping and Point-Cloud Fusion
May 15, 2026Xiongjun Guan, Jianjiang Feng, Jie ZhouBiometric Identification Cross-Modal Alignment
Department of Automation, Tsinghua University, Beijing 100084, China
GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
May 15, 2026Xu Wang, Xunkai Li, Yinlin Zhu +2Cross-Modal Learning Multimodal IR
School of Airspace Science and Engineering, Shandong University, WeiHai, China · Department of Computer Science, Beijing Institute of Technology, Beijing, China · School of Computer Science and Engineering, Sun Yat-sen University, GuangZhou, China
Semi-MedRef: Semi-Supervised Medical Referring Image Segmentation with Cross-Modal Alignment
May 15, 2026Yuchen Li, Zhen Zhao, Yi Liu +1Interactive Medical Image Segmentation Cross-Modal Alignment
The University of Sydney, Sydney, NSW 2006, Australia · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Changzhou University, Changzhou, China
Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment
May 14, 2026Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim +1Multimodal RAG Healthcare
Division of Intramural Research National Library of Medicine, National Institutes of Health Bethesda, MD, USA
Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke
May 14, 2026Liren Chen, Lidong Sun, Mingyan Huang +5Cross-Modal Alignment Clinical Outcome Prediction
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, PR China
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
May 14, 2026Tan Pan, Shuhao Mei, Yixuan Sun +8Representation Learning Multimodal Robustness
Fudan University, China · Shanghai Academy of AI for Science, China · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore +2
UniTriGen: Unified Triplet Generation of Aligned Visible-Infrared-Label for Few-Shot RGB-T Semantic Segmentation
May 14, 2026Ping Zhou, Haoyu Wang, Mengmeng Zheng +4Cross-Modal Alignment Multimodal Generation
School of Computer Science, Northwestern Polytechnical University · School of Computer Science & Technology, Xi’an University of Posts & Telecommunications · MMLab, The Chinese University of Hong Kong
Neural Visual Decoding via Cognitive guided Adaptive Blurring and Information Constrained Alignment
May 14, 2026Fan Yin, Chuhang Zheng, Peiliang Gong +2EEG-Based Visual Decoding Cross-Modal Alignment
Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · Department of Electrical and Information Engineering, Tianjin University, Tianjin, China
Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance
May 14, 2026Tianfang Zhu, Ning An, Rui Wang +4Vision-Based Tactile Sensing Visual-to-Tactile Prediction
Institute for AI Industry Research, Tsinghua University, Beijing, China. · MoE Key Laboratory for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China. · School of New Media Art and Design, Beihang University, Beijing, China. +1
PRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities
May 13, 2026Guangqian Yang, Ye Du, Wenlong Hou +2Missing-Modality Learning Alzheimer's Disease
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo,2026 Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerCross-Modal Learning Cross-Modal Alignment
The University of Texas at Dallas
Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention
May 12, 2026Hamza Ahmed Durrani, Rafay Suleman DurraniVision-Language Models Cross-Modal Alignment
Sejong University, Computer Science Engineering · Technische Universität Ilmenau, Computer Engineering
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiMultimodal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemCross-Modal Alignment Determinantal Point Process
School of Computing, Macquarie University, Sydney, Australia
TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
May 12, 2026Seongah Kim, Dinh Phu Tran, Hyeontaek Hwang +3Audio-Visual Understanding Audio-Visual Representation Learning
AI2 Lab, KAIST

May 19, 2026J/K move · Enter open · S save

Precise and Simple Audio-to-Score Alignment

Silvan Peter, Patricia Hu, Gerhard Widmer

Institute of Computational Perception, Johannes Kepler University, Linz, Austria · LIT AI Lab, Linz Institute of Technology, Linz, Austria

Cross-Modal Alignment Music Information Retrieval Sequence Alignment

Open paper AskPDF arXiv

Audio-to-score alignment is a long-standing challenge in music information retrieval and arguably the most widely applicable alignment task for music research. Alignment algorithms match two versions of a piece of music, and for this to work these versions need to be in comparable formats. Audio-to-audio alignment matches audio features; when matching audio files to scores, they must either synthesize the score or derive audio-like features by means of piano rolls or similar feature sequences. Symbolic alignment, by contrast, matches symbolically encoded notes; in an audio-to-score scenario these would be obtained by a transcription of the audio file. In this article, we present an algorithm that bridges audio-like and symbol-level features directly. Sequential audio features encoding onset and spectral activation are matched to score positions by a bespoke dynamic programming-based matching algorithm derived from symbolic alignment methods. The resulting method is both precise - surpassing widely used audio-to-audio approaches based on synthesized scores -, and remains flexible in its digital signal processing components, i.e., the method is adaptable to diverse timbral characteristics without requiring a separate transcription model. Furthermore it inherits some of the symbolic alignment runtime advantages with an algorithmic complexity that is at worst linear in the length of the (typically short) symbolic score and (typically long) audio feature sequence. In the following sections, we provide a detailed algorithm description and evaluate its alignment quality on a large-scale dataset of solo piano recordings.