Multimodal Contrastive Learning

Follow this topic Ask about this topic

Multimodal contrastive learning trains encoders to bring related examples from different modalities closer in an embedding space and separate unrelated ones, supporting retrieval, classification, and transfer. Common approaches include CLIP-style dual encoders with contrastive losses, fine-grained local or hierarchical alignment, and objectives that combine contrastive learning with reconstruction or captioning. Work increasingly addresses noisy or incomplete pairs, false negatives, long-tail data, and the need to preserve modality-specific information alongside shared semantics. A central tension is that stronger alignment can improve cross-modal matching but may also erase useful differences between modalities or reduce robustness and downstream transfer.

Latest papers

246 papers28 papers in the last four weeks, up 40% on the four weeks before. 0.2% of all new papers.

FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation
Mar 11, 2026Wenxuan Ma, Chaofan Zhang, Yinghao Cai +3Tactile Sensing for Robotic Manipulation Contact-Rich Robotic Manipulation
Institute of Automation, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence
Joint Imaging-ROI Representation Learning via Cross-View Contrastive Alignment for Brain Disorder Classification
Mar 10, 2026Wei Liang, Lifang HeBrain Disorder Classification Neuroimaging
Lehigh University, Bethlehem, PA, USA
WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition
Mar 10, 2026Shan Ning, Longtian Qiu, Jiaxuan Sun +1Vision-Language Alignment Multimodal Contrastive Learning
ShanghaiTech University, Shanghai, China · Lingang Laboratory, Shanghai, China · Shanghai Engineering Research Center of Intelligent Vision and Imaging
Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Framework
Mar 9, 2026Yutong Hu, Jinhui Chen, Chaoqiang Xu +6Cross-View Geo-Localization Multimodal Contrastive Learning
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, 430079, China · First Surveying and Mapping Institute of Hunan Province, Changsha, 421001, China
MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data
Mar 9, 2026Siarhei Sheludzko, Dhimitrios Duka, Bernt Schiele +2Multimodal Contrastive Learning
University of Bonn · MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen +1
Toward Unified Multimodal Representation Learning for Autonomous Driving
Mar 9, 2026Ximeng Tao, Dimitar Filev, Gaurav PandeyMultimodal Pretraining Multimodal Contrastive Learning
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
Augmenting representations with scientific papers
Mar 4, 2026Nicolò Oreste Pinciroli Vago, Rocco Di Tella, Carolina Cuesta-Lázaro +3Cross-Modal Alignment Multimodal Fusion
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Via Giuseppe Ponzio, 34, Milan, 20133, MI, Italy · Osservatorio Astronomico di Roma, INAF, Via Frascati 33, Monte Porzio Catone, 00078, RM, Italy · AstroAI, Center for Astrophysics | Harvard & Smithsonian, 60 Garden Street, Cambridge, 02138, MA, United States of America +1
ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
Mar 3, 2026Hanpeng Liu, Yaqian Li, Zidan Wang +6Cross-Modal Learning Cross-Modal Alignment
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Li Auto Inc. · Institute of AI for Industries, Chinese Academy of Sciences
Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
Mar 3, 2026Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati +9T2I Generation Visual Representation Learning
MIT Computer Science & Artificial Intelligence Laboratory · Meta AI
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Feb 27, 2026Hanseul Choi, Jinyeong Park, Seongwon Jin +2EEG Foundation Models Multimodal Pretraining
Department of Computer Science and Engineering, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea · Department of Artificial Intelligence, Inha University, Michuhol-gu, 22212, Incheon, Republic of Korea · Center for Brain-Machine Interface, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea
Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models
Feb 27, 2026Hiroshi SasakiContrastive Learning Multimodal Contrastive Learning
The Japan Research Institute, Limited
CG-DMER: Hybrid Contrastive-Generative Framework for Disentangled Multimodal ECG Representation Learning
Feb 24, 2026Ziwei Niu, Hao Sun, Shujun Bian +4Electrocardiogram Representation Learning Multimodal Disentangled Representation Learning
Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · College of information science and engineering, Ritsumeikan university, ibaraki Osaka, Japan +1
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
Feb 23, 2026Zanxi Ruan, Songqun Gao, Qiuyu Kong +2Vision-Language Alignment Cross-Modal Retrieval
University of Verona · Sapienza University of Rome · University of Trento +2
Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation
Feb 23, 2026Filip Wolf, Blaž Rolih, Luka Čehovin ZajcRemote Sensing Image Understanding Visual Representation Learning
University of Ljubljana, Faculty of Computer and Information Science, Slovenia
Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces
Feb 22, 2026Pratham Yashwante, Rose YuCross-Modal Alignment Multimodal Contrastive Learning
Department of Computer Science and Engineering, University of California San Diego, USA.
Re-purposing Multimodal Large Language Models for Audio-Text Retrieval
Feb 20, 2026Jilan Xu, Carl Thomé, Danijela Horak +2Multimodal Large Language Models Audio Understanding
Visual Geometry Group, University of Oxford · Epidemic Sound · School of Artificial Intelligence, Shanghai Jiao Tong University
Canonicalizing Multimodal Contrastive Representation Learning
Feb 19, 2026Sharut Gupta, Sanyam Kansal, Stefanie Jegelka +2Multimodal Embedding Representation Alignment
Aalto University, YaiYai Ltd
A Contrastive Variational AutoEncoder for NSCLC Survival Prediction with Missing Modalities
Feb 19, 2026Michele Zanitti, Vanja Miskovic, Francesco Trovò +5Cancer Survival Prediction Multimodal Survival Analysis
Department of Electronic Systems, Aalborg University, Copenhagen, Denmark · Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy · Department of Medical Oncology, Istituto Nazionale dei Tumori, Milan, Italy
BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning
Feb 19, 2026Siyuan Liang, Yongcheng Jing, Yingjie Wang +3Backdoor Attacks on VLMs Multimodal Contrastive Learning
College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computing, National University of Singapore, Singapore
Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection
Feb 18, 2026Rong Fu, Ziming Wang, Shuo Yin +3Multimodal Emotion Recognition Multimodal Sentiment Analysis
Independent Researcher
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead
Feb 4, 2026Jindong Li, Dario Zanca, Vincent Christlein +4Contrastive Learning Online Handwriting Recognition
Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Munich Center for Machine Learning, Munich, Germany · STABILO International GmbH, Heroldsberg, Germany +2
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Jan 31, 2026Yaohua Liu, Rong Fu, Amir H. Gandomi +2Representation Learning Cross-Modal Representation Learning
Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3
The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence
Jan 27, 2026Yichao Cai, Zhen Zhang, Yuhang Liu +1Cross-Modal Learning Contrastive Learning
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis
Nov 22, 2025Wenjing Liu, Qin Ren, Wen Zhang +2Cancer Survival Prediction Survival Analysis
Stony Brook University, Stony Brook, NY, USA · Johns Hopkins University, Baltimore, MD, USA · Brookhaven National Laboratory, Upton, NY, USA
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model Evaluation Audio Representation Learning
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1
Contrastive vision-language learning with paraphrasing and negation
Nov 20, 2025Kwun Ho Ngan, Saman Sadeghi Afgeh, Joe Townsend +1VLM Robustness VLM Reasoning
Fujitsu Research of Europe · United Kingdom · City St George’s, University of London

Mar 11, 2026J/K move · Enter open · S save

FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation

Wenxuan Ma, Chaofan Zhang, Yinghao Cai, Guocai Yao+2

Institute of Automation, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence

Tactile Sensing for Robotic Manipulation Contact-Rich Robotic Manipulation Visuo-Tactile Representation Learning Multimodal Contrastive Learning

Open paper AskPDF arXiv

Recent advancements in integrating tactile sensing into vision-language-action (VLA) models have demonstrated transformative potential for robotic perception. However, existing tactile representations predominantly rely on qualitative descriptors (e.g., texture), neglecting quantitative contact states such as force magnitude, contact geometry, and principal axis orientation, which are indispensable for fine-grained manipulation. To bridge this gap, we propose FG-CLTP, a fine-grained contrastive language tactile pretraining framework. We first introduce a novel dataset comprising over 100k tactile 3D point cloud-language pairs that explicitly capture multidimensional contact states from the sensor's perspective. We then implement a discretized numerical tokenization mechanism to achieve quantitative-semantic alignment, effectively injecting explicit physical metrics into the multimodal feature space. The proposed FG-CLTP model yields a 95.9% classification accuracy and reduces the regression error (MAE) by 52.6% compared to state-of-the-art methods. Furthermore, the integration of 3D point cloud representations establishes a sensor-agnostic foundation with a minimal sim-to-real gap of 3.5%. Building upon this fine-grained representation, we develop a 3D tactile-language-action (3D-TLA) architecture driven by a flow matching policy to enable multimodal reasoning and control. Extensive experiments demonstrate that our framework significantly outperforms strong baselines in contact-rich manipulation tasks, providing a robust and generalizable foundation for tactile-language-action models.