Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment
Aug 4, 2026Jing Dai, Qibin Zhang, Weiwei Zhou +4Cancer Survival Prediction Survival Analysis
Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany +2
Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping
Aug 4, 2026Jingwei Zhao, Gus Xia, Ziyu Wang +1Controllable Music Generation Cross-Modal Alignment
1Songscription · School of Computing, National University of Singapore · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) +1
Syntax Meets Semantics: Understanding Scientific Formulae
Aug 3, 2026Yuni Susanti, Moritz SchubotzSemantic Correspondence Cross-Modal Representation Learning
FIZ Karlsruhe Berlin, Germany
CalibBEV: LiDAR-Camera Calibration via BEV Alignment
Aug 3, 2026Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace +3Cross-Modal Alignment Camera-LiDAR Fusion
University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA. +1
FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision
Aug 2, 2026Tongyan Wang, Zhengyuan Li, Muhan Lin +5Cross-Modal Alignment
1Purdue University · University of Illinois Urbana-Champaign
CORTIVA: Candidate-Score Fusion of Complementary Visual Teachers for EEG- and MEG-to-Image Retrieval
Aug 2, 2026Junhan Wang, Kani ChenEEG-Based Visual Decoding Cross-Modal Alignment
The Hong Kong University of Science and Technology, Hong Kong SAR, China
GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System
Jul 31, 2026Jiping Liu, Zhongmin Zhang, Zisen Sang +7Cross-Modal Alignment Multimodal Pretraining
Rajax Network Technology (Taobao Shangou of Alibaba) · Shanghai & Beijing, China · Central South University +1
Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation
Jul 31, 2026Weixiang Zhou, Xingguo Xu, Yuhao Wang +4Cross-Modal Representation Learning Multimodal Robustness
School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China · Department of Radiology and Biomedical Imaging, University of California, San Francisco, 94107, USA +1
Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization
Jul 31, 2026Tyler Ashoff, Jordan RoduKG Entity Alignment AI Alignment
Department of Statistics, University of Virginia. · Department of Statistics, University of Virginia
VCP-DCN: Beyond Visual Concealed Property via Depth Collaborative Network for Camouflaged Object Detection
Jul 30, 2026Songsong Duan, Xi Yang, Nannan WangCross-Modal Alignment Multimodal Fusion
the State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China
Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering
Jul 29, 2026Yu Wang, Sharon LiUnified Multimodal Models Cross-Modal Alignment
Department of Computer Sciences, University of Wisconsin-Madison
EchoBridge: Long-Tail-Aware ECG-Echocardiography Text Alignment for Echocardiography-Derived Cardiac Findings
Jul 27, 2026Xiaocheng Fang, Jieyi Cai, Guangkun Nie +9Long-Tailed Classification Cross-Modal Alignment
School of Intelligence Science and Technology, Peking University Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University · University of the Chinese Academy of Sciences Beijing, China +2
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Jul 27, 2026Minchong Chen, Xiaoyun Yuan, Minyu Cao +4Diffusion Models for Image Restoration Multi-Source Image Fusion
MoE Key Lab of Artificial Intelligence, Institute of AI, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Tsinghua University, Beijing, China · Institute of Modern Optics, Nankai University, Tianjin, China
Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance
Jul 26, 2026Weixiang Zhou, Jiabei Zuo, Yuhao Wang +3Cross-Modal Alignment Multimodal Fusion
School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China +1
CODA: Cascaded Online Discontinuity-Aware Alignment for Real-Time Image-Based Score Following
Jul 24, 2026Yining Yang, Ruogu Chen, Jie HanCross-Modal Alignment Music Information Retrieval
Don Wright Faculty of Music, Western University, Canada · Department of Electrical and Computer Engineering, University of Alberta, Canada
UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging
Jul 23, 2026Rafsan Jany, Shadab Tanjeed Ahmad, Ahsan Bulbul +3Cross-Modal Knowledge Distillation Cross-Modality Medical Image Segmentation
Korea Institute of Oriental Medicine, Daejeon, South Korea · Islamic University of Technology, Gazipur, Bangladesh
Distribution-Alignment Bridge for Uncertainty-Aware Text-to-Video Retrieval
Jul 23, 2026Kyeongmo Chae, Jihoon Lee, Sangtae AhnRepresentation Learning Cross-Modal Alignment
School of Electronic and Electrical Engineering, Kyungpook National University, Daegu, South Korea
Diverse-Intent Multi-Turn Fashion Image Retrieval
Jul 22, 2026Mingqiang Tang, Haokun Wen, Meng Liu +3Multimodal IR Cross-Modal Alignment
1Southern University of Science and Technology, Shenzhen, China · 2Harbin Institute of Technology (Shenzhen), Shenzhen, China · 3Shandong University, Jinan, China +1
Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training
Jul 22, 2026Qiwei Ma, Bin Deng, Junjie Zhu +5Visual Representation Learning Self-Supervised Pre-Training
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, Hunan, China · Intelligent Game and Decision Lab, Beijing, 100166, Beijing, China · Yuelushan Center for Industrial Innovation, Changsha, 410082, Hunan, China
Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction
Jul 22, 2026Min-Jae Kim, Jun-Yeong Moon, Mujeen Sung +1Cross-Modal Alignment Multimodal Learning
1Korea University, Seoul, Republic of Korea · 2Kyung Hee University, Yongin, Republic of Korea
RMS@CC-MMD 2026: Multimodal Misogyny Detection via Geometric Interaction and Multi-View Consensus
Jul 20, 2026Md. Ajwad HossainCross-Modal Alignment Hate Speech Detection
Department of Electronics and Telecommunication Engineering Chittagong University of Engineering & Technology (CUET) Chattogram, Bangladesh
OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment
Jul 20, 2026Wenxiao Fan, Hang Yin, Kan Li3D Spatial Reasoning Cross-Modal Alignment
School of Computer Science, Beijing Institute of Technology
FSDBN: Foreground-Aware EEG-Visual Alignment via Dynamic Brain Networks
Jul 20, 2026Yiheng Liu, Chuhang Zheng, Peiliang Gong +3EEG-Based Visual Decoding Cross-Modal Alignment
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · The International Joint Institute of Tianjin University, Fuzhou, Tianjin University, Tianjin, China
STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMs
Jul 19, 2026Ye Wang, Hongjun Wang, Hao Fang +7Unified Multimodal Models Cross-Modal Alignment
Jilin University · Tencent YouTu Lab
MIME: Multimodal Interactive Motion Encoder
Jul 18, 2026Addison Zucek, Prerit Gupta, Kamila Kuatova +1Text-to-Motion Generation Cross-Modal Alignment
Purdue University
Knowledge-Guided Cross-Modal Fusion for Adult-to-Pediatric ECG Transfer via Label-Conditioned Contrastive Alignment
Jul 17, 2026Xinran Liu, Yuwen Li, Hongxiang Gao +4Electrocardiography Cross-Modal Alignment
School of Instrument Science and Engineering, Southeast University, Nanjing, 210096, China · Nanjing Medical University, Nanjing, 211166, China · Zhengzhou University, Zhengzhou, 450001, China
Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Jul 17, 2026Xunkai Li, Guohao Fu, Yuming Ai +4Multimodal Federated Learning Cross-Modal Alignment
Beijing Institute of Technology, Beijing, 100811, China
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Jul 16, 2026Sarthak Jain, Qiran Hu, Zhen Zhu +1Cross-Modal Representation Learning Cross-Modal Alignment
University of Illinois Urbana-Champaign · Google DeepMind
LATTICE: Graph Self-Supervised Learning for Multimodal Spatial Omics Integration
Jul 15, 2026Jagan Mohan Reddy Dwarampudi, Veena Kochat, Suresh Satpati +2Spatial Transcriptomics Graph SSL
University of Houston · MD Anderson Cancer Center

Aug 4, 2026J/K move · Enter open · S save

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment

Jing Dai, Qibin Zhang, Weiwei Zhou, Mingde Xu+3

Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany+2

Cancer Survival Prediction Survival Analysis Cross-Modal Alignment Oncology Multimodal Survival Analysis Multimodal Fusion

Open paper AskPDF arXiv

Multimodal learning has significantly advanced survival prediction by integrating pathology images with genomic data. However, clinical information, despite its critical role in reflecting a patient' s overall health, remains underutilized due to its discrete, sparse, and low-dimensional nature. Furthermore, the inherent heterogeneity across these modalities pose significant challenges in modeling cross-modal interactions. In this paper, we propose CIGTSurv, a Clinical Information Guided Tri-modal framework for Survival prediction. Specifically, we first design a holistic text template and use pretrained foundation models to transform clinical tabular data into high-dimensional tokenized embeddings. Using clinical information as an anchor, we then introduce a dual-level interaction mechanism: 1) a local prototype association (LPA) module based on cross-attention to explicitly learn token-level correspondences between different modalities, and 2) a global feature alignment (GFA) loss based on Maximum Mean Discrepancy (MMD) to implicitly enhance cross-modal distribution consistency. Extensive experiments on five TCGA cancer cohorts demonstrate that CIGTSurv achieves state-of-the-art (SOTA) survival prediction performance. Our source code is publicly available at https://github.com/Daijing-ai/CIGT-Surv.git.