Cross-Modal Alignment

Follow this topic Ask about this topic

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition
May 12, 2026Sanghyeon Lee, Jinwoo Kim, Jong Taek LeeSkeleton-Based Action Recognition Human Activity Recognition
School of Computer Science and Engineering Kyungpook National University, Daegu, South Korea
CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation
May 12, 2026Daohan Su, Hao Liu, Xunkai Li +6Efficient Multimodal Inference Cross-Modal Alignment
Fashion130K: An E-commerce Fashion Dataset for Outfit Generation with Unified Multi-modal Condition
May 11, 2026Yu He, Ting Zhu, Yichun Liu +6Conditional Image Generation Cross-Modal Alignment
HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchies
May 10, 2026Zihan Ma, Tian Xia, Kexin Wang +3Neural Representation Geometry Cross-Modal Alignment
School of Electronic Information (School of Artificial Intelligence), the Xi’an Key Laboratory of Radiomics and Intelligent Perception, Northwest University, Xi’an 710127, China
The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?
May 10, 2026Zhaoyang Zhang, Run Shao, Dongyue Wu +4Neural Representation Geometry Cross-Modal Alignment
1Central South University · 4Ant Group · 2Huazhong University of Science and Technology +1
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
May 8, 2026Mingyu Liu, Sihan Huang, Yijia Fan +4Cross-Modal Learning VLM Adaptation
1Sun Yat-sen University · 2Guangdong Province Key Laboratory of Information Security Technology, China · 3Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
Anisotropic Modality Align
May 8, 2026Xiaomin Yu, Yijiang Li, Yuhui Zhang +8Cross-Modal Learning Cross-Modal Alignment
HKUST(GZ) · NUS · Stanford +2
SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models
May 8, 2026Jiesong Lian, Zixiang Zhou, Ruizhe Zhong +6Diffusion Model Alignment Video Diffusion Models
1Huazhong University of Science and Technology · 2Tencent Hunyuan · 3Shanghai Jiao Tong University +1
Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer
May 8, 2026Gajendra Katuwal, Advait Koparkar, Salar Abbaspourazad +2Cross-Modal Learning Representation Learning
Apple
UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition
May 8, 2026Shuai Zhang, Zhecheng Shi, Zhuxiao Li +4Disentangled Representation Learning Multimodal Semantic Segmentation
1HKUST(GZ) · 2HKUST
TRAJGANR: Trajectory-Centric Urban Multimodal Learning via Geospatially Aligned Neural Representations
May 7, 2026Maria Despoina Siampou, Gengchen Mai, Ni Lao +4Geospatial Foundation Models Cross-Modal Alignment
Google Research, Mountain View, CA · Dept. of Computer Science, University of Southern California, Los Angeles, CA · SEAI Lab, Dept. of Geography and the Environment, The University of Texas at Austin, Austin, TX +1
Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval
May 7, 2026Jun Li, Peifeng Lai, Xuhang Lou +5Cross-Modal Learning Evidential Deep Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China
MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
May 7, 2026Shichao Kan, Xuyang Zhang, Haojie Zhang +7Cross-Modal Alignment Vision-Language Grounding
School of Computer Science and Engineering, Central South University, Changsha, 410083, China · School of Computer Science and Technology and the State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University, Beijing, 100044, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 101408, China +1
Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection
May 7, 2026Letian Bai, Xuanming Cao, Juan Du +1Prompt Learning Cross-Modal Alignment
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · College of Mechanical and Vehicle Engineering, Hunan University
Unifying Scientific Communication: Fine-Grained Correspondence Across Scientific Media
May 7, 2026Megha Mariam K. M, Vineeth N. Balasubramanian, C. V. JawaharCross-Modal Alignment Scientific Communication
IIIT Hyderabad · Microsoft Research India & IIT Hyderabad
Direct Product Flow Matching: Decoupling Radial and Angular Dynamics for Few-Shot Adaptation
May 6, 2026Hongxu Chen, Yanghao Wang, Bowei Zhu +6Cross-Modal Learning Flow Matching
HKUST · USTC · Huawei Research
Multi-Level Bidirectional Biomimetic Learning for EEG-Based Visual Decoding
May 6, 2026Jingtao Liu, Peiliang Gong, Chuhang Zheng +2Visual Representation Learning EEG-Based Visual Decoding
Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · Department of Electrical and Information Engineering, Tianjin University, Tianjin, China
Interpreting V1 Population Activity via Image-Neural Latent Representation Alignment
May 5, 2026Xin Wang, Zhuangzhi Gao, Hongyi Qin +3Contrastive Learning Neural Processes
School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Department of Primary Care and Mental Health, University of Liverpool, Liverpool, United Kingdom · Liverpool Centre for Cardiovascular Science, University of Liverpool, Liverpool, United Kingdom +2
Identity-Consistent Multi-Pose Generation of Contactless Fingerprints
May 5, 2026Zhiyu Pan, Xiongjun Guan, Jianjiang Feng +1Synthetic Data Augmentation Biometric Identification
Department of Automation, Tsinghua University, Beijing 100084, China
MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding
May 5, 2026Ran Ran, Jiwei Wei, Shuchang Zhou +5Temporal Video Grounding Cross-Modal Alignment
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Cyberspace Security, Hainan University, Haikou, China
ViewSAM: Learning View-aware Cross-modal Semantics for Weakly Supervised Cross-view Referring Multi-Object Tracking
May 4, 2026Jiawei Ge, Xintian Zhang, Jiuxin Cao +9Cross-Modal Alignment Weakly Supervised Learning
Southeast University · Universität der Bundeswehr München · Queen Mary, University of London +2
StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation
May 4, 2026Jintao Guo, Lin Wang, Shumeng Li +5Source-Free Domain Adaptation Neural Decoding
National Key Laboratory for Novel Software Technology and the Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 · School of Intelligence Science and Technology, Nanjing University, Nanjing 215163, China +2
Ultrasound Vision-Language Alignment via Contrastive Learning
May 4, 2026Zhuoyang Lyu, Yiyang Zhang, Tongxin Wang +1Cross-Modal Alignment Ultrasound Imaging
Department of Biostatistics Harvard T. H. Chan School of Public Health Boston, MA, USA
Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridges
May 3, 2026Eitan Kosman, Gabriele Serussi, Chaim BaskinCross-Modal Learning Cross-Modal Alignment
Ben-Gurion University of the Negev, Israel.
Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
May 3, 2026Junyuan Xiao, Dingkang Liang, Xin Zhou +9World Models Cross-Modal Alignment
Tsinghua University, China · Huazhong University of Science and Technology, China · CSU, China +4
MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation
May 2, 2026Ke Qiu, Yawen Qin, Tianzhi Jia +3Cross-Modal Alignment Human Motion Generation
Malou Tech Inc · South-Central Minzu University · Beijing Jiaotong University +2
Multimodal Data Curation Through Ranked Retrieval
May 1, 2026Pratyush Muthukumar, Harshil Kotamreddy, Sarah Amiraslani +4Cross-Modal Learning Cross-Modal Alignment
NVIDIA
EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure
May 1, 2026Zihao Ding, Beining Wu, Jun HuangCross-Modal Learning Multimodal Federated Learning
Department of Electrical Engineering and Computer Science South Dakota State University Brookings, SD 57007, USA
Label-Conditioned Cross-Modal Fusion for Adult-to-Pediatric ECG Transfer via Curriculum-Gated Contrastive Alignment
May 1, 2026Xinran Liu, Yuwen Li, Hongxiang Gao +4Electrocardiography Cross-Modal Alignment
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, Nanjing, China · Nanjing Medical University, Nanjing 211166, Nanjing, China · Zhengzhou University, Zhengzhou 450001, Zhengzhou, China
CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detection
May 1, 2026Hang Wang, Chao Shen, Chenhao Lin +3Cross-Modal Alignment Deepfake Detection
Xi’an Jiaotong University, Xi’an, China · The Hong Kong Polytechnic University, Hong Kong, China · Guangdong OPPO Mobile Communications Co., Ltd. +1

May 12, 2026J/K move · Enter open · S save

PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition

Sanghyeon Lee, Jinwoo Kim, Jong Taek Lee

School of Computer Science and Engineering Kyungpook National University, Daegu, South Korea

Skeleton-Based Action Recognition Human Activity Recognition Cross-Modal Alignment Video Action Recognition Zero-Shot Action Recognition

Open paper AskPDF arXiv

Zero-shot skeleton-based action recognition (ZSSAR) is typically treated as a skeleton-text alignment problem: encode joint-coordinate sequences, align them with language, and classify unseen actions. We argue that this alignment is often too late. Skeletons are not complete action observations, but compressed outputs of human pose estimation (HPE); by the time alignment begins, human-object interactions and pose-relative visual cues may no longer be explicit. We call this upstream semantic loss. To address it, we propose PoseBridge, an HPE-aware ZSSAR framework that bridges intermediate HPE representations to skeleton-text alignment. Rather than adding an RGB action branch or object detector, PoseBridge extracts pose-anchored semantic cues from the same HPE process that produces skeletons, then transfers them through skeleton-conditioned bridging and semantic prototype adaptation. Across NTU-RGB+D 60/120, PKU-MMD, and Kinetics-200/400, PoseBridge improves ZSSAR performance under the evaluated protocols. On the Kinetics-200/400 PURLS benchmark, which contains in-the-wild videos with diverse scenes and action contexts, PoseBridge shows the clearest separation, improving the strongest compared baseline by 13.3-17.4 points across all eight splits. Our code will be publicly released.