Cross-Modal Alignment

Ask about this topic

Overview

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

CardsList

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

  1. The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

    Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal AlignmentModality Gap in VLMs

    University of Electronic Science and Technology of China · University of Bristol

  2. Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method

    Mar 29, 2026Xiaoran Xu, Xiaoshan Yang, Jiangang Yang +3Open-Vocabulary Object DetectionVLM Robustness

    the MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · the School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · the Institute of Microelectronics, University of Chinese Academy of Sciences, Beijing, China

  3. On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

    Mar 28, 2026Krisanu SarkarFunctional MapsCross-Modal Alignment

    Indian Institute of Technology Bombay, Mumbai, India

  4. LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

    Mar 28, 2026Ruosi Wang, Fangwei Zuo, Lei Li +1Skeleton-Based Action RecognitionCross-Modal Alignment

    Northwestern Polytechnical University, No.1 Dongxiang Road, Chang’an District , Xi’an, 710129, Shaanxi, China · Shandong University of Finance and Economics, No.40 Shungeng Road, Licheng District, Jinan, Jinan, Shandong, China

  5. SemConFlow: Semantic Grounding of Holistic Co-Speech Gesture Generation with Contrastive Flow-Matching

    Mar 27, 2026Lanmiao Liu, Esam Ghaleb, Aslı Özyürek +1Multimodal GroundingFlow Matching

    Max Planck Institute for Psycholinguistics · Donders Institute for Brain Cognition and Behaviour · Utrecht University

  6. Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

    Mar 27, 2026Zizhao Chen, Ping Wei, Ziyang Ren +2Multimodal FusionFake News Detection

    State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

  7. R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning

    Mar 26, 2026Zirui Zhang, Haoyu Dong, Kexin Pei +1Multimodal ReasoningCross-Modal Learning

    Rutgers University · Columbia University · University of Chicago

  8. BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

    Mar 25, 2026Risa Shinoda, Kaede Shiohara, Nakamasa Inoue +3Cross-Modal AlignmentMultimodal Embedding

    The University of Osaka · The University of Tokyo · Institute of Science Tokyo +1

  9. HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment

    Mar 24, 2026Sangmin Jo, Wootaek Jeong, Da-Woon Heo +2Cross-Modal AlignmentEEG-Based Visual Decoding

    Department of Artificial Intelligence, Korea University, Seoul, South Korea

  10. UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

    Mar 23, 2026Ziyi Wang, Xinshun Wang, Shuang Chen +2Cross-Modal LearningUnified Multimodal Models

    State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China · Donghua University, China · School of Automation Science and Engineering, South China University of Technology

  11. Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interaction

    Mar 23, 2026Shiyan Hu, Jianxin Jin, Yang Shu +3Multimodal Anomaly DetectionTime-Series Anomaly Detection

    East China Normal University

  12. Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning

    Mar 21, 2026Xu Zhang, Jin Yuan, BinHong Yang +5Video CaptioningVideo Object Segmentation

    the College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China. · the School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, P.R. China. · CRRC Zhuzhou Institute Company Ltd., Zhuzhou, Hunan 412001, China. +2

  13. Cross-modal Fuzzy Alignment Network for Text-Aerial Person Retrieval and A Large-scale Benchmark

    Mar 21, 2026Yifei Deng, Chenglong Li, Yuyang Zhang +2Image-Text RetrievalCross-Modal Alignment

    State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology · School of Computer Science and Technology, Anhui University · The University of Hong Kong +1

  14. Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervision

    Mar 20, 2026Jiyeong Kim, Yerim So, Hyesong Choi +2Unified Multimodal ModelsMultimodal Generation

    Ewha Womans University, South Korea · Soongsil University, South Korea

  15. A Novel FACS-Aligned Anatomical Text Description Paradigm for Fine-Grained Facial Behavior Synthesis

    Mar 19, 2026Jiahe Wang, Cong Liang, Xuandong Huang +5Facial Expression SynthesisT2I Generation

    University of Science and Technology of China, Hefei, China

  16. AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba

    Mar 19, 2026Yan Li, Yifei Xing, Xiangyuan Lan +3Multimodal Sentiment AnalysisMultimodal Fusion

    Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China

  17. Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

    Mar 18, 2026Dongqiang Gou, Xuming He3D Affordance Grounding3D Visual Grounding

    ShanghaiTech University

  18. Complementarity-Preserving Generative Theory for Multimodal ECG Synthesis: A Quantum-Inspired Approach

    Mar 16, 2026Timothy Oladunni, Farouk Ganiyu-Adewumi, Clyde Baidoo +1Multimodal GenerationElectrocardiography

    Department of Computer Science, Morgan State University, Baltimore, MD, USA

  19. SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis

    Mar 14, 2026Ehud Gordon, Meir Yossef Levi, Guy GilboaCanonical Correlation AnalysisMultimodal Model Interpretability

    Viterbi Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel

  20. Anchored Alignment: Preventing Positional Collapse in Multimodal Recommender Systems

    Mar 13, 2026Yonghun Jeong, David Yoon Suk Kang, Yeon-Chang LeeMultimodal RecommendationCross-Modal Alignment

    UNIST, Ulsan, Korea · Chungbuk National University, Cheongju, Korea

  21. CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignment

    Mar 13, 2026Kaifan Zhang, Lihuo He, Junjie Ke +4EEG-Based Visual DecodingUncertainty-Aware Multimodal Fusion

    Visual Information Processing Laboratory, School of Electronic Engineering, Xidian University, Xi’an 710071, China. · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China

  22. Text-Phase Synergy Network with Dual Priors for Unsupervised Cross-Domain Image Retrieval

    Mar 13, 2026Jing Yang, Hui Xue, Shipeng Zhu +1Unsupervised Representation LearningInvariant Representation Learning

    School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

  23. Toward Unified Multimodal Representation Learning for Autonomous Driving

    Mar 9, 2026Ximeng Tao, Dimitar Filev, Gaurav PandeyMultimodal PretrainingMultimodal Contrastive Learning

    J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA

  24. Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding

    Mar 8, 2026Shumeng Li, Jintao Guo, Jian Zhang +3fMRI-Based Visual DecodingNeural Decoding

    State Key Laboratory of Novel Software Technology, Nanjing University · Institute of Brain-Machine Interface, Nanjing University

  25. Aligning What EEG Can See: Structural Representations for Brain-Vision Matching

    Mar 7, 2026Jingyi Tang, Shuai Jiang, Fei Su +1EEG-Based Visual DecodingCross-Modal Alignment

    Beijing University of Posts and Telecommunications, Beijing, 100876, China

  26. ProtAlign: Contrastive learning paradigm for Sequence and structure alignment

    Mar 6, 2026Aditya Ranganath, Hasin Us Sami, Kowshik Thopalli +2Cross-Modal AlignmentProtein Representation Learning

    Center for Applied Scientific Computing, Lawrence Livermore National Laboratory

  27. Fusion4CA: Boosting 3D Object Detection via Comprehensive Image Exploitation

    Mar 5, 2026Kang Luo, Xin Chen, Yangyi Xiao +13D Object DetectionCamera-LiDAR Fusion

    IRMV Lab, the Department of Automation, Shanghai Jiao Tong University.

  28. Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment

    Mar 5, 2026Xiang Ma, Lexin Fang, Litian Xu +1Cross-Modal AlignmentVision-Language Alignment

    Shandong University, Jinan, China · The University of Exeter, Exeter, United Kingdom

  29. Augmenting representations with scientific papers

    Mar 4, 2026Nicolò Oreste Pinciroli Vago, Rocco Di Tella, Carolina Cuesta-Lázaro +3Cross-Modal AlignmentMultimodal Fusion

    Department of Electronics, Information and Bioengineering, Politecnico di Milano, Via Giuseppe Ponzio, 34, Milan, 20133, MI, Italy · Osservatorio Astronomico di Roma, INAF, Via Frascati 33, Monte Porzio Catone, 00078, RM, Italy · AstroAI, Center for Astrophysics | Harvard & Smithsonian, 60 Garden Street, Cambridge, 02138, MA, United States of America +1

  30. Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

    Mar 4, 2026Haoyang Chen, Jing Zhang, Hebaixu Wang +7Remote Sensing Image UnderstandingImage-to-Image Translation

    National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University, and Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. 100094 · School of Electronic Information, Wuhan University, Wuhan, China +2