Multimodal Pretraining

Ask about this topic

Overview

Multimodal pretraining studies how models can learn reusable representations from combinations of data such as text, images, video, audio, medical signals, and sensor measurements. Common approaches include contrastive alignment in the style of CLIP, masked reconstruction, joint-embedding prediction, and generative objectives such as next-token prediction. The aim is to transfer across tasks and settings, including retrieval, reasoning, forecasting, and generation, often with fewer task-specific labels. A central tension is how to align modalities enough to share useful information while retaining their distinct structure, especially when data are noisy, incomplete, or unevenly represented.

Latest papers

CardsList

182 papers17 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.

  1. EmbeddingGemma 2: Unified and Efficient Multimodal Representations

    Oct 27, 2026Henrique Schechter Vera, Sahil Dua, Jovan Andonov +70Multimodal EmbeddingUnified Multimodal Models

  2. MedVTok: A General-Purpose Medical Visual Tokenizer

    Oct 26, 2026Chenglong Ma, Yuanfeng Ji, Junzhi Ning +18Image TokenizationVisual Tokenization

    Fudan University, China · Shanghai Innovation Institute, China · Stanford University School of Medicine, United States +4

  3. OlmoEarth v1.3: Embeddings

    Oct 23, 2026Gabriel Tseng, Favyen Bastani, Patrick Alan Johnson +13Geospatial Representation LearningMultimodal Embedding

    Allen Institute for AI

  4. Text-to-Image Generative Modeling Without Language Pretraining

    Oct 19, 2026Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas +5T2I GenerationAutoregressive Image Generation

    Stanford University · Radical Numerics

  5. SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention

    Oct 19, 2026Saurav Raj Pandey, Harlin LeeMultimodal PretrainingCross-Modal Representation Learning

    Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA · School of Data and Information Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

  6. MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

    Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal LearningMultimodal Foundation Models

    University of California, Riverside · Amazon

  7. More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

    Oct 14, 2026Haoyuan Wu, Aoqi Wu, Jiani Hu +1Multimodal PretrainingCross-Modal Learning

    The Chinese University of Hong Kong · Foundation Model Department, Tencent

  8. MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

    Oct 13, 2026Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo +3Remote SensingGeospatial Foundation Models

    Image and Signal Processing (ISP) Group, Universitat de València, Valencia, Spain · Asterisk Labs, London, United Kingdom

  9. Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

    Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive LearningMultimodal Learning

    Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

  10. HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignment

    Oct 10, 2026Jiaqi Wang, Wei Chen, Xiaoyu Li +7Representation LearningPhotoplethysmography

    Southeast University · OPPO · Shenzhen Technology University +1

  11. Scaling Laws for Multimodal Data Mixtures

    Oct 10, 2026Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5Multimodal PretrainingAI Scaling Laws

    MILA - Quebec AI Institute, McGill University · EPFL · ETH +1

  12. Mid-Training Language Models on Raw Video

    Oct 8, 2026Jaedong Hwang, Xiaoqian Shen, Ernie Chang +9Multimodal Large Language ModelsVideo-Language Models

    Meta AI · Massachusetts Institute of Technology · Work done at Meta +2

  13. SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

    Oct 7, 2026Jeonghyeok Do, Munchurl KimRemote Sensing Image UnderstandingMultimodal Pretraining

    Korea Advanced Institute of Science and Technology (KAIST)

  14. Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

    Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal ModelsMultimodal Pretraining

    Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics

  15. How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

    Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal PretrainingMultimodal Large Language Models

    CASIA · UCAS · Foundation Model Department, Tencent

  16. Advancing Video-Text Pretraining with Multi-View Captions

    Sep 28, 2026Fida M. Thoker, Renaud Vandeghen, Karen Sanchez +2Multimodal Pretraining

    King Abdullah University of Science and Technology (KAUST) · University of Li`ege

  17. InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

    Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal PretrainingMultimodal QA

    The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute

  18. Clinical Trajectory Alignment for Medical Vision-Language Pre-training

    Sep 28, 2026Huimin Yan, Xian Yang, Zhi Wang +1Multimodal PretrainingLongitudinal Medical Image Analysis

    Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK

  19. WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation

    Sep 28, 2026Chuan Qin, Shaoting Zhu, Siyuan Luo +4Humanoid Loco-ManipulationMultimodal Pretraining

    IIIS, Tsinghua University, Beijing, China. · Xiong’an Institute of Artificial Intelligence, Xiong’an, China. · The University of Melbourne, Melbourne, Australia.

  20. PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment

    Sep 27, 2026Xiaoda Wang, Minxiao Wang, Maxwell A Xu +10Multimodal PretrainingPhotoplethysmography

    Emory University · University of California, Los Angeles · Google +3

  21. Phenomenon-Graph JEPA: Label-Efficient Representation Learning for Contactless Cardiorespiratory Sensing

    Sep 26, 2026Constantino Álvarez Casado, Nhi Nguyen, Mohammad Rakibur Rahman +4Self-Supervised Time Series Representation LearningJoint-Embedding Predictive Architecture

    Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland · Candour Ltd, Oulu, Finland

  22. TimeBraid: Unifying Time Series and Language for Understanding and Forecasting

    Sep 24, 2026Xinyue Wang, Jiacheng Pang, Kun Zhou +7Unified Multimodal ModelsMultimodal Pretraining

    University of California San Diego · Aether AI · University of Southern California

  23. PhyMo: A Physical-Field Modality for Multimodal AI4Physics

    Sep 23, 2026Henan Sun, Haitao Hu, Jin Liu +4AI for ScienceCross-Modal Representation Learning

    The Hong Kong University of Science and Technology (Guangzhou) · Huawei Noah’s Ark Lab · Tencent HY +1

  24. MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction

    Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Missing-Modality LearningCross-Modal Alignment

    Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1

  25. Qwen3.8-Omni: Towards Native Omni-Modal Agents

    Sep 22, 2026Qwen TeamMultimodal PretrainingMultimodal Large Language Models

    Qwen Team

  26. AtomEgo: Exploring Ego-Robot Integration for Embodied Foundation Model Pretraining

    Sep 18, 2026Di Wu, Dongchen Zheng, Junhe Sheng +8Robot LearningRobot Foundation Models

    INFIFORCE · Tongji University · HKU +2

  27. FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

    Sep 15, 2026Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8Cross-Modal Representation LearningCross-Modal Alignment

    Meta AI

  28. Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning

    Sep 10, 2026Andrea Agostini, Simon Böhi, Moritz Vandenhirtz +7Masked AutoencodersHealthcare

    Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland

  29. Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

    Sep 10, 2026Christian Gapp, Elias Tappeiner, Martin Welk +8Multimodal LearningMultimodal Fusion

    Institute of Biomedical Image Analysis UMIT TIROL – Private University for Health Sciences and Health Technology, Eduard-Wallnöfer-Zentrum 1, 6060 Hall in Tirol, Austria · VASCage – Centre on Clinical Stroke Research, 6020 Innsbruck, Austria · Department of Radiology, Medical University of Innsbruck, 6020 Innsbruck, Austria +2

  30. Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

    Sep 8, 2026Siting Li, Zhengyang Wang, Simon Shaolei Du +2Image TokenizationUnified Multimodal Models

    University of Washington · Amazon FAR