Multimodal Contrastive Learning

Ask about this topic

Overview

Multimodal contrastive learning trains encoders to bring related examples from different modalities closer in an embedding space and separate unrelated ones, supporting retrieval, classification, and transfer. Common approaches include CLIP-style dual encoders with contrastive losses, fine-grained local or hierarchical alignment, and objectives that combine contrastive learning with reconstruction or captioning. Work increasingly addresses noisy or incomplete pairs, false negatives, long-tail data, and the need to preserve modality-specific information alongside shared semantics. A central tension is that stronger alignment can improve cross-modal matching but may also erase useful differences between modalities or reduce robustness and downstream transfer.

Latest papers

CardsList

246 papers28 papers in the last four weeks, up 40% on the four weeks before. 0.2% of all new papers.

  1. LoCA: Enabling Multi-Modality in Geospatial Location Encoders

    Oct 23, 2026Panagiotis Koromilas, Eleanna Vali, Diogenis Antonopoulos +7Geospatial Representation LearningMultimodal Contrastive Learning

    University of Athens · The Cyprus Institute · Archimedes, Athena Research Center +7

  2. Closing the Modality Gap via Simplex-Constrained Representations

    Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMsCross-Modal Alignment

    Mila – Québec AI Institute · ServiceNow Research · Université Laval +1

  3. GeoLEON-CLIP: Graph-Anchored Multimodal Contrastive Learning for Geographic Region Representation

    Oct 23, 2026Radoslaw Malek, Szymon Soltysiak, Aleksandra Kawala-Sterniuk +1Geospatial Representation LearningMultimodal Contrastive Learning

    Wrocław University of Science and Technology, Department of Artificial Intelligence, Wrocław, Poland

  4. UNIVERS: Learning UNIversal Voxel-wise Embedding via Randomized Synthesis for generalizable one-shot landmark localization

    Oct 21, 2026Derong Yu, Qingsong Zhao, Yanbing Yue +3Anatomical Landmark DetectionMedical Image Representation Learning

    Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China. · Department of Cardiology, Shanghai General Hospital (originally named Shanghai First People’s Hospital), Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China.

  5. ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

    Oct 20, 2026Kieren Yu, Chang Huang, Ziyang Liu +1EEG-Based Visual DecodingCross-Modal Alignment

  6. Preserving the Manifold: Frozen Expert Alignment for Label-Efficient SAR Scene Understanding

    Oct 18, 2026Eli A. Underwood, I. Taylor McKechnie, Leslie M. CollinsRemote Sensing VLMsCross-Modal Alignment

    Duke University

  7. CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering

    Oct 15, 2026Yinghua Xu, Pengyang Xie, Yanxiang Zhou +4ClusteringMultimodal Embedding

    College of Control Science and Engineering, Zhejiang University · Alibaba International Digital Commerce

  8. Semantic Alignment of EEG and Text: A Contrastive Learning Framework for Decoding Imagined Speech

    Oct 14, 2026Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1EEG DecodingEEG-to-Text Decoding

    Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy

  9. ACR-Net: Mitigating Semantic Dominance via Contrastive Acoustic-Semantic Decoupling

    Oct 14, 2026Mengke Zhang, Yanda Shao, Tianhe Wu +1Speech Emotion RecognitionAudio-Language Model Evaluation

    Beijing University of Posts and Telecommunications, China

  10. Revisiting Triplet Contrastive Learning for 3D Shape Understanding

    Oct 14, 2026Chaoyang Zhu, Zhe Liao, Jiayi Zhang +33D Representation LearningCross-Modal Alignment

    Xiamen University · Shanghai AI Lab

  11. MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

    Oct 13, 2026Yanran Ruan, Zhixiang Wei, Shiyao Hong +5Fluorescence MicroscopyMicroscopy Image Analysis

    University of Science and Technology of China

  12. Multimodal Contrastive Representation Learning Based Voice Conversion for Electrolaryngeal Speech Enhancement

    Oct 13, 2026Ding Ma, Jinyi Mi, Jiajun He +3Voice ConversionSpeech Enhancement

    Graduate School of Informatics, Nagoya University, Nagoya, Japan · Alibaba Token Foundry, Alibaba Group, Hangzhou, China · Headquarters, TARVO, Inc., Nagoya, Japan +2

  13. Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

    Oct 12, 2026Daniel Shao, Dongmin Bang, Luca L. Weishaupt +6Computational PathologyMedical VLMs

    MIT · MGB AI Institute · Harvard University +1

  14. Approximation in Contrastive Representation Learning

    Oct 11, 2026Yuanfan Li, Zihan Zhang, Yiming Ying +1Contrastive LearningMultimodal Contrastive Learning

    School of mathematics and statistics, University of Sydney

  15. What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

    Oct 10, 2026Duo Li, Haojun Pan, Zuhao Yang +4Multimodal EmbeddingMultimodal Contrastive Learning

    CCDS, Nanyang Technological University · Alibaba Group · Universität Hamburg +1

  16. Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

    Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive LearningMultimodal Learning

    Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

  17. Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder

    Oct 8, 2026Zidan Wang, Yaqian Li, Xiaokai Zhang +3VLM Post-TrainingVLM Adaptation

    Huazhong University of Science and Technology · Li Auto Inc.

  18. Dynamic Alignment and Calibration for Multimodal Learning

    Oct 6, 2026Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu +1Multimodal RobustnessCross-Modal Alignment

    School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China

  19. MS-ECG-FM: Towards a More Universal Electrocardiogram Foundation Model for Health Monitoring using Multi-source Contrastive Learning

    Oct 6, 2026Robert A. Lewis, I-Min Chiu, Kyle Verrier +7Electrocardiogram Foundation ModelsElectrocardiogram Classification

    Massachusetts Institute of Technology (work done while at Apple). · Apple, Inc. · Division of Cardiology, Johns Hopkins Medicine. +1

  20. fMRI-TAMCL: Text-Anchored Supervised Multimodal Contrastive Learning for fMRI-Based Brain Disorder Classification

    Oct 5, 2026Juliana Mantebea Danso, Enoch Opanin Gyamfi, Mylene C. Q. FariasNeuroimagingMultimodal Contrastive Learning

    Department of Computer Science, Texas State University, San Marcos, TX 78666 USA · University of Technology and Applied Sciences, Navrongo, Ghana

  21. Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology

    Oct 4, 2026Sigrid Vila-Bagaria, Mar Teixidó, Miquel Piñol +3Computational PathologyCross-Modal Retrieval

    Signal Theory and Communications Department (TSC) Universitat Politècnica de Catalunya - BarcelonaTech (UPC) · Cancer Biomarkers Research Group (GReBiC) - IRB Lleida

  22. Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

    Oct 1, 2026Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Sahal Shaji Mullappilly +2Cross-Modal Knowledge DistillationMultimodal Embedding

    Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)

  23. PLRS-IC: A Dual-Calibration Framework for Chest X-Ray Vision-Language Alignment

    Sep 30, 2026Qixing Zhao, Jinpeng LiRadiology VLMsVision-Language Alignment

    South China University of Technology

  24. MoTIF-X: A Multimodal Tokenized Framework for Interpretable and Extensible Molecular Representation Learning

    Sep 29, 2026Linqing Mo, Jiayu Zhou, Bin ChenMolecular Representation LearningCross-Modal Representation Learning

    Department of Computer Science and Engineering, Michigan State University, East Lansing, 48824, MI, USA. · School of Information, University of Michigan, Ann Arbor, 48109, MI, USA. · Department of Pediatrics and Human Development, Michigan State University, Grand Rapids, MI, 49503, USA. +2

  25. End-to-End Self-Supervised RGB-T Tracking without Modality Misleading

    Sep 29, 2026Shenglan Li, Rui Yao, Kunyang Sun +4Multimodal RobustnessVisual Object Tracking

    School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, China. · Mine Digitization Engineering Research Center of the Ministry of Education, China. · University of Auckland, Auckland, New Zealand. +1

  26. Language as the Interface: Foundation-Model Contrastive Learning Links Transcriptomes and Electrophysiology

    Sep 29, 2026Junbo Shen, Jinying Gao, Bo LeiCross-Modal AlignmentMultimodal Foundation Models

    Department of Computer Science and Engineering, The Chinese University of Hong Kong · State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China +1

  27. Structured Visual Target Learning For Cross-Subject eeg-to-image retrieval

    Sep 29, 2026Salini Yadav, Taveena Lotey, Mickaël Coustaty +2Cross-Subject EEG DecodingVisual Representation Learning

    Indian Institute of Technology Roorkee, India · University of La Rochelle, France · Indian Institute of Technology (ISM) Dhanbad, India

  28. SyncRA: Learning Temporal Correspondence in Omni-Modal Models

    Sep 28, 2026Zelong Xu, Yan Li, Wenhe Hu +1Audio-Visual Representation LearningAudio-Visual Synchronization

    University of Wisconsin–Madison · University of Alberta · Arizona State University

  29. Learning Multimodal Embeddings with Evidence-Aligned Readout

    Sep 27, 2026Zirong Chen, Fuda Ye, Enjun Du +8Multimodal Large Language ModelsMultimodal Embedding

    The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · Tsinghua University +3

  30. SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

    Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation LearningMultimodal Contrastive Learning

    OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA