Cross-Modal Alignment

Ask about this topic

Overview

Cross-modal alignment studies how to connect information from different data types, such as images and text, audio and video, or brain signals and visual features, so models can retrieve, interpret, or generate across them. Common approaches use contrastive learning and shared embedding spaces, alongside optimal transport, prototypes, geometric mappings, and local or temporal correspondence; some methods rely on weakly paired or unpaired data. Work is evaluated on tasks such as image-text retrieval, grounding, zero-shot transfer, and multimodal generation, including in medical and scientific settings. A central tension is how to align shared meaning without erasing useful modality-specific information, since stronger alignment scores do not always translate into better downstream performance or consistent behavior across tasks.

Latest papers

CardsList

446 papers47 papers in the last four weeks, up 4% on the four weeks before. 0.3% of all new papers.

  1. DQE-CIR: Distinctive Query Embeddings through Learnable Attribute Weights and Target Relative Negative Sampling in Composed Image Retrieval

    Mar 4, 2026Geon Park, Ji-Hoon Park, Seong-Whan LeeComposed Image RetrievalFine-Grained Image Retrieval

    Department of Artificial Intelligence, Korea University, Seoul, 02841, Republic of Korea

  2. MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction

    Mar 4, 2026Seunghoi Kim, Chen Jin, Henry F. J. Tregidgo +2Cross-Modal AlignmentMultimodal Pretraining

    University College London, UK · Centre for AI, DS&AI, AstraZeneca, UK

  3. ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

    Mar 3, 2026Hanpeng Liu, Yaqian Li, Zidan Wang +6Cross-Modal LearningCross-Modal Alignment

    School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences

  4. OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

    Mar 2, 2026Hymalai Bello, Lala Ray, Joanna Sorysz +2Human Activity RecognitionVideo Action Recognition

    DFKI Kaiserslautern · RPTU Kaiserslautern-Landau · Korea University

  5. Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining

    Mar 2, 2026Yuxuan Li, Yuming Chen, Yunheng Li +3Remote Sensing Image UnderstandingMultimodal Pretraining

    PCA Lab & VCIP, CS, Nankai University. · NKIARI, Shenzhen Futian.

  6. Align-cDAE: Alzheimer's Disease Progression Modeling with Attention-Aligned Conditional Diffusion Auto-Encoder

    Mar 2, 2026Ayantika Das, Keerthi Ram, Mohanasankar SivaprakasamDiffusion AutoencoderConditional Diffusion Models

    Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India · Sudha Gopalakrishnan Brain Centre, Indian Institute of Technology Madras, Chennai, India

  7. Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning

    Mar 2, 2026Zhongjian Zhang, Xiao Wang, Mengmei Zhang +2Graph ReconstructionLLMs for Graph Learning

    Beijing University of Posts and Telecommunications, Beijing, China · Beihang University, Beijing, China · China Telecom Bestpay, Beijing, China

  8. Neural Functional Alignment Space: Brain-Referenced Representation of Artificial Neural Networks

    Feb 28, 2026Ruiyu Yan, Hanqi Jiang, Yi Pan +4Neural Representation GeometryCross-Modal Alignment

    Tandon School of Engineering, New York University, Brooklyn, NY 11201, USA · School of Computing, University of Georgia, Athens, GA 30602, USA · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA +1

  9. Linking Modality Isolation in Heterogeneous Collaborative Perception

    Feb 28, 2026Changxing Liu, Zichen Chao, Siheng ChenCross-Modal AlignmentHeterogeneous Collaborative Perception

    Shanghai Jiao Tong University · Nanjing University of Science and Technology

  10. SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoning

    Feb 28, 2026Yi Zhang, Youya Xia, Yong Wang +73D Spatial Reasoning3D VLMs

    Foundation Model Department, Huawei · Central Media Technology Institute, Huawei

  11. A Mixed Diet Makes DINO An Omnivorous Vision Encoder

    Feb 27, 2026Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson +5Cross-Modal Representation LearningCross-Modal Alignment

    Google DeepMind · University College London

  12. CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models

    Feb 27, 2026Hanseul Choi, Jinyeong Park, Seongwon Jin +2EEG Foundation ModelsMultimodal Pretraining

    Department of Computer Science and Engineering, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea · Department of Artificial Intelligence, Inha University, Michuhol-gu, 22212, Incheon, Republic of Korea · Center for Brain-Machine Interface, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea

  13. Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

    Feb 27, 2026Abhishek Dalvi, Vasant HonavarCross-Modal AlignmentVision-Language Alignment

    Artificial Intelligence Research Laboratory, The Pennsylvania State University

  14. PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation

    Feb 26, 2026Yiqing Wang, Chunming He, Ziyun Yang +5Cross-Modal AlignmentMultimodal Pretraining

    Department of Biomedical Engineering, Duke University 101 Science Drive, Durham, NC 27705 USA · Department of Ophthalmology and Visual Sciences, University of Michigan 1000 Wall Street, Ann Arbor, MI, USA

  15. Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception

    Feb 26, 2026Yiding Sun, Jihua Zhu, Haozhe Cheng +4Transfer LearningPoint Cloud Learning

    the School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710048, China · the School of Electrical and Information Engineering, Hunan University, Changsha, 410082, China +1

  16. PSQE: A Theoretical-Practical Approach to Pseudo Seed Quality Enhancement for Unsupervised Multimodal Entity Alignment

    Feb 26, 2026Yunpeng Hong, Chenyang Bu, Jie Zhang +3Cross-Modal AlignmentPseudo-Labeling

    Key Laboratory of Knowledge Engineering with Big Data (the Ministry of Education of China), Hefei University of Technology, Hefei, Anhui, CN · Department of Data Science, College of William and Mary, Williamsburg, VA, USA · College of Computer and Information Science, Southwest University, Chongqing, CN

  17. Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptation

    Feb 24, 2026Lin Li, Ziqi Jiang, Gefan Ye +5VLM AdaptationCross-Modal Alignment

    The Hong Kong University of Science and Technology (HKUST) · Zhejiang University

  18. Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

    Feb 23, 2026Wei He, Xianghan Meng, Zhiyuan Huang +3Generalized Category DiscoveryCross-Modal Representation Learning

    School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Intellifusion Inc., Shenzhen, P.R. China

  19. CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning

    Feb 23, 2026Chunlei Meng, Guanhong Huang, Rong Fu +3Cross-Modal Representation LearningMultimodal Learning

    Fudan University · Shantou University · University of Macau +1

  20. A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data

    Feb 23, 2026Joseph BinghamReferring Expression ComprehensionMultimodal Grounding

    Faculty of Biology, Technion University, Haifa, Israel.

  21. Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

    Feb 22, 2026Pratham Yashwante, Rose YuCross-Modal AlignmentMultimodal Contrastive Learning

    Department of Computer Science and Engineering, University of California San Diego, USA.

  22. CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion

    Feb 22, 2026Sijie Mai, Shiqin HanMultimodal FusionCross-Modal Alignment

    South China Normal University

  23. MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment

    Feb 22, 2026Duc Duy Nguyen, Tat-Jun Chin, Minh HoaiCross-Modal Representation LearningCross-Modal Alignment

    Australian Institute for Machine Learning, Adelaide University, Adelaide, SA 5000, Australia

  24. Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations

    Feb 21, 2026Xiaoyu Dong, Jiahuan Li, Ziteng Cui +1Image Super-ResolutionCross-Modal Alignment

    The University of Tokyo, Japan · RIKEN AIP, Japan

  25. Synthesizing Multimodal Geometry Datasets from Scratch and Enabling Visual Alignment via Plotting Code

    Feb 21, 2026Haobo Lin, Tianyi Bai, Chen Chen +4Multimodal ReasoningSynthetic Data Generation

    HKUST · JLU · USTC +1

  26. Canonicalizing Multimodal Contrastive Representation Learning

    Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal EmbeddingRepresentation Alignment

    Aalto University, YaiYai Ltd

  27. SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

    Feb 19, 2026Lorenzo Caselli, Marco Mistretta, Simone Magistri +1Cross-Modal LearningCross-Modal Representation Learning

    University of Florence, Media Integration and Communication Center (MICC), Italy

  28. VETime: Vision Enhanced Zero-Shot Time Series Anomaly Detection

    Feb 18, 2026Yingyuan Yang, Tian Lan, Yifei Gao +6Time-Series Anomaly DetectionZero-Shot Anomaly Detection

    Department of Industrial Engineering, Tsinghua University, Beijing, China · Lab, Huawei Technologies Ltd, Shenzhen, China · Datadog AI Research

  29. MRC-GAT: A Meta-Relational Copula-Based Graph Attention Network for Interpretable Multimodal Alzheimer's Disease Diagnosis

    Feb 17, 2026Fatemeh Khalvandi, Saadat Izadi, Abdolah ChalechaleAlzheimer's Disease ClassificationGraph Attention Networks

    Computer Engineering Department, Razi University, Kermanshah, Iran

  30. An Industrial Dataset for Scene Acquisitions and Functional Schematics Alignment

    Feb 17, 2026Flavien Armangeon, Thibaud Ehret, Enric Meinhardt-Llopis +4Cross-Modal AlignmentDigital Twins

    Universit´e Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, 91190, Gif-sur-Yvette, France · EDF R&D (Electricit´e de France), France · AMIAD, Pˆole Recherche, France +1