Multimodal Pretraining

Follow this topic Ask about this topic

Multimodal pretraining studies how models can learn reusable representations from combinations of data such as text, images, video, audio, medical signals, and sensor measurements. Common approaches include contrastive alignment in the style of CLIP, masked reconstruction, joint-embedding prediction, and generative objectives such as next-token prediction. The aim is to transfer across tasks and settings, including retrieval, reasoning, forecasting, and generation, often with fewer task-specific labels. A central tension is how to align modalities enough to share useful information while retaining their distinct structure, especially when data are noisy, incomplete, or unevenly represented.

Latest papers

182 papers17 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.

VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection
Mar 18, 2026Chupeng Liu, Jiyong Rao, Shangquan Sun +23D Object Detection Monocular 3D Object Detection
The University of Sydney · Tongji University · Nanyang Technological University
Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling
Mar 17, 2026Hongjian Zou, Yue Ge, Qi Ding +2Multimodal Large Language Models Multimodal Learning
vivo AI Lab, Shenzhen, China · Wuhan University, Wuhan, China
ExpressMind: A Multimodal Pretrained Large Language Model for Expressway Operation
Mar 17, 2026Zihe Wang, Yihuan Wang, Haiyang Yu. Zhiyong Cui +4Intelligent Transportation Systems Multimodal Large Language Models
Beihang University Beijing, China · Shandong Hi-speed Group Co., Ltd Jinan, China · Institute of automation, Chinese Academy of Sciences Beijing, China
GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models
Mar 17, 2026Jiaxin Zhang, Junjun Jiang, Haijie Li +3Multimodal Large Language Models Multimodal Pretraining
Harbin Institute of Technology, China · School of Electronic and Computer Engineering, Peking University, China · Huawei, China
Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training
Mar 17, 2026Peng Sun, Jun Xie, Tao LinUnified Multimodal Models Multimodal Pretraining
Zhejiang University · Westlake University · Shanghai Innovation Institute
Overcoming the Modality Gap in Context-Aided Forecasting
Mar 12, 2026Vincent Zhihao Zheng, Étienne Marcotte, Arjun Ashok +4Synthetic Data Augmentation Multimodal Pretraining
ServiceNow AI Research, Montréal, Canada · Department of Civil Engineering, McGill University, Montréal, Canada · DIRO, Université de Montréal, Montréal, Canada +1
Learning Transferable Sensor Models via Language-Informed Pretraining
Mar 12, 2026Yuliang Chen, Arvind Pillai, Yu Yvonne Wu +6Multimodal Pretraining Language Modeling
Dartmouth College
MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals
Mar 9, 2026Junyu Shen, Zhendong She, Chenghanyu Zhang +13Multimodal Large Language Models Multimodal Pretraining
Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications +6
Toward Unified Multimodal Representation Learning for Autonomous Driving
Mar 9, 2026Ximeng Tao, Dimitar Filev, Gaurav PandeyMultimodal Pretraining Multimodal Contrastive Learning
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
Mar 7, 2026Lance Legel, Qin Huang, Brandon Voelker +9World Model Learning Geospatial Representation Learning
Ecological Intelligence Lab Ecodash.ai · School of Complex Adaptive Systems Arizona State University · Geosensing Systems Engineering & Sciences Lab University of Houston +6
Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning
Mar 7, 2026Paul Julius Kühn, Cedric Spengler, Michael Weinmann +23D Shape Retrieval Cross-Modal Retrieval
Fraunhofer IGD, 64283 Darmstadt, Germany · Delft University of Technology, 2628 CD Delft, Netherlands
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
Mar 6, 2026Yingtai Li, Shuai Ming, Qiuli Wang +13Medical Imaging Foundation Models Radiology VLMs
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC, Hefei, Anhui, 230001, China +7
Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages
Mar 6, 2026Shaharukh Khan, Ali Faraz, Abhinav Ravi +6Vision-Language Pretraining Multimodal Pretraining
Krutrim AI, Bangalore, India
MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction
Mar 4, 2026Seunghoi Kim, Chen Jin, Henry F. J. Tregidgo +2Cross-Modal Alignment Multimodal Pretraining
University College London, UK · Centre for AI, DS&AI, AstraZeneca, UK
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Mar 3, 2026Shengbang Tong, David Fan, John Nguyen +18Multimodal Pretraining Unified Multimodal Models
FAIR, Meta · New York University
Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining
Mar 2, 2026Yuxuan Li, Yuming Chen, Yunheng Li +3Remote Sensing Image Understanding Multimodal Pretraining
PCA Lab & VCIP, CS, Nankai University. · NKIARI, Shenzhen Futian.
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
Mar 2, 2026Jiahan Chen, Da Li, Hengran Zhang +6Multimodal Embedding Multimodal Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China
FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning
Mar 1, 2026Xingcan Hu, Wei Wang, Li XiaoBrain Network Analysis Functional Connectivity
University of Science and Technology of China
MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules
Mar 1, 2026Idelfonso B. R. Nogueira, Carine M. Rebello, Mumin Enis Leblebici +1Molecular Property Prediction Multimodal Foundation Models
Department of Chemical Engineering, Norwegian University of Science and Technology (NTNU), Trondheim, 7034, Norway · Faculty of Industrial Engineering, KU Leuven, Diepenbeek, Belgium · University of Surrey, Guildford, United Kingdom
MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine
Mar 1, 2026Kai Zhang, Zhengqing Yuan, Cheng Peng +10Healthcare Multimodal Pretraining
Department of Computer Science and Engineering, Lehigh University · Department of Computer Science and Engineering, University of Notre Dame · Department of Health Outcomes & Biomedical Informatics, University of Florida +3
CAMEL-CLIP: Channel-aware Multimodal Electroencephalography-text Alignment for Generalizable Brain Foundation Models
Feb 27, 2026Hanseul Choi, Jinyeong Park, Seongwon Jin +2EEG Foundation Models Multimodal Pretraining
Department of Computer Science and Engineering, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea · Department of Artificial Intelligence, Inha University, Michuhol-gu, 22212, Incheon, Republic of Korea · Center for Brain-Machine Interface, Incheon National University, Yeonsu-gu, 22012, Incheon, Republic of Korea
SleepLM: Natural-Language Intelligence for Human Sleep
Feb 27, 2026Zongzhe Xu, Zitao Shuai, Eideen Mozaffari +3Multimodal Pretraining Cross-Modal Retrieval
University of California, Los Angeles
PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation
Feb 26, 2026Yiqing Wang, Chunming He, Ziyun Yang +5Cross-Modal Alignment Multimodal Pretraining
Department of Biomedical Engineering, Duke University 101 Science Drive, Durham, NC 27705 USA · Department of Ophthalmology and Visual Sciences, University of Michigan 1000 Wall Street, Ann Arbor, MI, USA
Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
Feb 26, 2026Hanning Guo, Hanwen Bi, Farah Abdellatif +4Functional Magnetic Resonance Imaging Multimodal Foundation Models
INM-4, Forschungszentrum Jülich, Germany · Department of Computer Science 3 - Software Engineering, RWTH Aachen University, Germany · INM-7, Forschungszentrum Jülich, Germany +6
The Design Space of Tri-Modal Masked Diffusion Models
Feb 25, 2026Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21Diffusion Models Multimodal Diffusion Models
Apple · University of Cambridge · MIT +1
Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation
Feb 23, 2026Filip Wolf, Blaž Rolih, Luka Čehovin ZajcRemote Sensing Image Understanding Visual Representation Learning
University of Ljubljana, Faculty of Computer and Information Science, Slovenia
Xray-Visual Models: Scaling Vision models on Industry Scale Data
Feb 18, 2026Shlok Mishra, Tsung-Yu Lin, Linda Wang +24Vision Foundation Models Multimodal Pretraining
Meta-AI · MIT (work done while at Meta AI) · University of Maryland (work done while at Meta AI)
TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics
Feb 16, 2026Tobia Boschi, Andrea Loreti, Nicola C. Amorisco +13Multimodal Foundation Models Multimodal Pretraining
IBM Research Dublin, Ireland · UK Atomic Energy Authority Abingdon, UK · STFC Hartree Centre Daresbury, UK
Query as Anchor: Scenario-Adaptive User Representation via Large Language Model
Feb 16, 2026Jiahao Yuan, Yike Xu, Jinyong Wen +9User Modeling Representation Learning
Ant Group · East China Normal University · Zhejiang University
PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning
Nov 21, 2025Melika Shirian, Kianoosh Vadaei, Kian Majlessi +3Multimodal SSL Self-Supervised Learning
dept. Computer Engineering University of Isfahan Isfahan, Iran · dept. Computer Engineering University of Texas at Dallas Isfahan, Iran · dept. Artificial Intelligence University of Isfahan Isfahan, Iran

Mar 18, 2026J/K move · Enter open · S save

VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection

Chupeng Liu, Jiyong Rao, Shangquan Sun, Runkai Zhao+1

The University of Sydney · Tongji University · Nanyang Technological University

3D Object Detection Monocular 3D Object Detection Visual Prompt Learning Multimodal Pretraining

Open paper AskPDF arXiv

Monocular 3D object detection typically relies on pseudo-labeling techniques to reduce dependency on real-world annotations. Recent advances demonstrate that deterministic linguistic cues can serve as effective auxiliary weak supervision signals, providing complementary semantic context. However, hand-crafted textual descriptions struggle to capture the inherent visual diversity of individuals across scenes, limiting the model's ability to learn scene-aware representations. To address this challenge, we propose Visual-referred Probabilistic Prompt Learning (VirPro), an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks. Specifically, we generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB). Subsequently, we introduce Multi-Gaussian Prompt Modeling (MGPM), which incorporates scene-based visual features into the corresponding textual embeddings, allowing the text prompts to express visual uncertainties. Then, from the fused vision-language embeddings, we decode a prompt-targeted Gaussian, from which we derive a unified object-level prompt embedding for each instance. RoI-level contrastive matching is employed to enforce modality alignment, bringing embeddings of co-occurring objects within the same scene closer in the latent space, thus enhancing semantic coherence. Extensive experiments on the KITTI benchmark demonstrate that integrating our pretraining paradigm consistently yields substantial performance gains, achieving up to a 4.8% average precision improvement than the baseline. Code is available at https://github.com/AustinLCP/VirPro.