Multimodal Contrastive Learning

Follow this topic Ask about this topic

Momentum

16 papers in the last four weeks, up 45% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 153

All topics

Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection
Jun 1, 2026David J. Lerch, Livien Majer, Zeyun Zhong +3Multimodal Robustness Cross-Modal Alignment
Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning
Jun 1, 2026Dong Li, Lingling Zhang, Binghao Han +2Missing-Modality Learning Cross-Modal Attention
School of Information, Liaoning University, Shenyang, China · School of Computer Science and Engineering, Northeastern University, Shenyang, China
HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition
May 31, 2026Koby Aharonov, Oren Shrout, Ayellet TalContrastive Learning Open-Set Recognition
Technion – Israel Institute of Technology
A Triple-Modal Contrastive Learning Framework with Sequence, Graph, and 3D Features for Drug-Target Interaction Prediction
May 28, 2026Le Xu, Xi Zhang, Dan Luo +2Multimodal Contrastive Learning Molecular Representation Learning
School of Computer Science, Xiangtan University, Xiangtan 411105, China
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal Learning Cross-Modal Alignment
Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
May 28, 2026Yonggang Zhu, Liting Gao, Aidong Men +1Audio Understanding Audio-Language Models
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey, GU2 7XH Guildford, U.K.
TRACER: Persistent Regularization for Robust Multimodal Finetuning
May 28, 2026Hesam Asadollahzadeh, Feng Liu, Christopher Leckie +1VLM Robustness Neural Network Robustness
School of Computing and Information Systems (CIS), Faculty of Engineering and IT (FEIT), University of Melbourne, Australia.
Utility-Aware Multimodal Contrastive Learning for Product Image Generation
May 27, 2026Xiaohang Feng, Yiling XieImage Generation E-Commerce
City University of Hong Kong
Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals
May 26, 2026Phu X. Nguyen, Konstantinos Kontras, Wei Dai +5Representation Learning Cross-Modal Representation Learning
1KU Leuven · 3MIT · 4DFKI +1
A Signal-Language Foundation Model for Broad-Spectrum Cardiovascular Assessment from Routine Electrocardiography
May 25, 2026Ziqing Yu, Yuhui Tao, Jiayu Huo +19Electrocardiography Electrocardiogram Foundation Models
Department of Cardiology, Zhongshan Hospital of Fudan University, Shanghai Institute of Cardiovascular Diseases, National Clinical Research Centre for Interventional Medicine, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, 200032, China. · Shanghai Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention, Shanghai, 200032, China. +8
Checking Fact with Better Retrieval: Dynamic Contrastive Learning for Evidence Retrieval
May 24, 2026Zhongtian Hua, Yi Luo, Meijia Yu +1Automated Fact-Checking Cross-Modal Retrieval
Zhengzhou University, Zhengzhou, Henan, China · Henan University of Science and Technology, Luoyang, Henan, China
Cross-Modal Contrastive Learning of ECG and Angiography Representations for Severe Stenosis Classification
May 23, 2026Nikola Cenikj, Özgün Turgut, Alexander Müller +5Medical Imaging Coronary Angiography
Chair for AI in Healthcare and Medicine, Technical University of Munich and TUM University Hospital, Munich, Germany · Department of Internal Medicine, TUM University Hospital, Munich, Germany · Department of Computing, Imperial College London, UK
MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
May 23, 2026Zexuan Chen, Sichao Liu, Runhao Lu +4EEG-Based Visual Decoding Cross-Modal Alignment
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2
SLIP-RS: Structured-Attribute Language-Image Pre-Training for Remote Sensing Object Detection
May 22, 2026Chenxu Wang, Yuxuan Li, Yunheng Li +3Open-Vocabulary Object Detection Remote Sensing Image Understanding
1VCIP, CS, Nankai University · 2NKIARI, Shenzhen Futian · 3National University of Defense Technology
Multimodal LLMs under Pairwise Modalities
May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal Learning Representation Identifiability
Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
May 16, 2026Wen-Hsin Tsai, Chia-Ming Lee, Yuk-Ying TungCross-Modal Learning Multimodal Robustness
Institute of Education, National Cheng Kung University · Institute of Intelligent System, National Yang Ming Chiao Tung University · Department of Computer Science, University at Albany, State University of New York
Semi-MedRef: Semi-Supervised Medical Referring Image Segmentation with Cross-Modal Alignment
May 15, 2026Yuchen Li, Zhen Zhao, Yi Liu +1Interactive Medical Image Segmentation Cross-Modal Alignment
The University of Sydney, Sydney, NSW 2006, Australia · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Changzhou University, Changzhou, China
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
May 14, 2026Jakub Grzywaczewski, Dawid Płudowski, Przemysław BiecekRepresentation Geometry Representation Learning
Warsaw University of Technology · Centre for Credible Artificial Intelligence · University of Warsaw
Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke
May 14, 2026Liren Chen, Lidong Sun, Mingyan Huang +5Cross-Modal Alignment Clinical Outcome Prediction
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, PR China
CA-GCL: Cross-Anatomy Global-Local Contrastive Learning for Robust 3D Medical Image Understanding
May 13, 2026Hanwen Zhang, Yao Liu, Die Dai +4VLM Robustness Medical Image Anomaly Detection
University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerCross-Modal Learning Cross-Modal Alignment
The University of Texas at Dallas
Multitask Multimodal Fusion with Tabular Foundation Models for Peak and Durability Prediction of Pertussis Booster Response
May 13, 2026Divya SitaniMulti-Task Learning Clinical Prediction
Berlin, Germany
Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation
May 11, 2026Yujia Cai, Boxuan Li, Chenghao Xu +1Multimodal IR VLM Adaptation
School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi, China · School of Electronic Engineering, Xidian University, Xi’an, Shaanxi, China
CoLVR: Enhancing Exploratory Latent Visual Reasoning via Contrastive Optimization
May 9, 2026Ziyang Ding, Linjian Meng, Yiming Wu +3Visual Reasoning Latent Visual Reasoning
Shandong University · Shanghai AI Laboratory · Nanjing University +1
DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
May 7, 2026Shuyang Jiang, Nan Yu, Yiming Zhang +2Contrastive Learning Vision-Language Alignment
University of California, Los Angeles · Aimaikj · HFIPS, Chinese Academy of Sciences +2
DART: A Vision-Language Foundation Model for Comprehensive Rope Condition Monitoring
May 6, 2026Anju Rani, Daniel Ortiz-Arroyo, Petar DurdevicJoint-Embedding Predictive Architecture Large Vision-Language Models
Department of Energy Aalborg University Esbjerg, Denmark 6700
Multi-Level Bidirectional Biomimetic Learning for EEG-Based Visual Decoding
May 6, 2026Jingtao Liu, Peiliang Gong, Chuhang Zheng +2Visual Representation Learning EEG-Based Visual Decoding
Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · Department of Electrical and Information Engineering, Tianjin University, Tianjin, China
Ultrasound Vision-Language Alignment via Contrastive Learning
May 4, 2026Zhuoyang Lyu, Yiyang Zhang, Tongxin Wang +1Cross-Modal Alignment Ultrasound Imaging
Department of Biostatistics Harvard T. H. Chan School of Public Health Boston, MA, USA
EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure
May 1, 2026Zihao Ding, Beining Wu, Jun HuangCross-Modal Learning Multimodal Federated Learning
Department of Electrical Engineering and Computer Science South Dakota State University Brookings, SD 57007, USA
Label-Conditioned Cross-Modal Fusion for Adult-to-Pediatric ECG Transfer via Curriculum-Gated Contrastive Alignment
May 1, 2026Xinran Liu, Yuwen Li, Hongxiang Gao +4Electrocardiography Cross-Modal Alignment
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, Nanjing, China · Nanjing Medical University, Nanjing 211166, Nanjing, China · Zhengzhou University, Zhengzhou 450001, Zhengzhou, China

Jun 1, 2026, cs.CVJ/K move · Enter open · S save

Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection

David J. Lerch, Livien Majer, Zeyun Zhong, Manuel Martin+2

Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

Multimodal Robustness Cross-Modal Alignment Multimodal Contrastive Learning Driver Monitoring Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Robust self-supervised learning of multi-modal video representations is critical for real-world applications such as driver distraction detection, where multiple sensors provide complementary but noisy signals. Conventional contrastive objectives, such as InfoNCE, assume all negatives are equally informative and all positives are reliable. However, this assumption is frequently violated in multi-modal data due to viewpoint changes, occlusions, or semantic overlap across modalities. In this work, we propose a novel framework for multi-modal global alignment that addresses these challenges by jointly modeling faulty negatives and unreliable or faulty positives. We introduce soft targets derived from cycle-consistency scores to relax the hard-negative assumption, and a weighting mechanism based on similarity distributions to mitigate the impact of noisy or faulty positives. Our approach extends traditional pairwise alignment to a principled global multi-modal setting, aggregating alignment information across all modality pairs. We evaluate our method on the Drive&Act dataset, demonstrating that it consistently outperforms both pairwise and existing global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Cross-view ablation studies further show strong generalization to unseen camera perspectives, highlighting the robustness of our representations. Overall, our framework provides a scalable and effective solution for self-supervised global multi-modal representation learning, enabling reliable driver distraction detection and pioneering in real-world multi-modal video understanding. Our code will be published on GitHub.

Momentum

Related topics

Latest papers 153