Overview

Multimodal pretraining studies how models can learn reusable representations from combinations of data such as text, images, video, audio, medical signals, and sensor measurements. Common approaches include contrastive alignment in the style of CLIP, masked reconstruction, joint-embedding prediction, and generative objectives such as next-token prediction. The aim is to transfer across tasks and settings, including retrieval, reasoning, forecasting, and generation, often with fewer task-specific labels. A central tension is how to align modalities enough to share useful information while retaining their distinct structure, especially when data are noisy, incomplete, or unevenly represented.

Latest papers

CardsList

192 papers14 papers in the last four weeks, down 30% on the four weeks before. 0.1% of all new papers.

  1. Unbundling Multimodal Vision Encoding

    Nov 12, 2026Ellis Brown, Chris Hoang, Shengyi Qian +9Multimodal Foundation ModelsMultimodal Learning

  2. Model Distribution-Aware Multimodal Dataset Distillation

    Nov 12, 2026Jiajun Shen, Yonglin Wu, Ruonan Yu +1Dataset DistillationMultimodal Pretraining

  3. Relaxing Cross-Modal Correspondence for Multimodal Remote Sensing Pretraining

    Nov 10, 2026Xianping Ma, Yu Zhang, Mingzhe Jiang +5Multimodal PretrainingMultimodal Learning

  4. Leveraging Complementary Event Frame and Voxel Representations for RGB-Guided Multimodal Pre-training

    Nov 9, 2026Wentao Wu, Xiao Wang, Yi Wang +3Multimodal PretrainingMultimodal SSL

  5. FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics

    Nov 6, 2026Qiang Chen, Xiao Wang, Qingquan Yang +4Multimodal PretrainingMultimodal Contrastive Learning

  6. Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining

    Nov 4, 2026Daichi Hotta, Yohei Kobashi, Takeshi Kojima +2Language Model PretrainingRecurrent Transformers

  7. Zero-Gated Cross-Attention

    Oct 31, 2026Basavaraj R M, Manav VermaCross-AttentionGated Attention

  8. Brain-Behavior Modeling from Multimodal MRI using Opportunistic Pretraining and Common-Cohort Adaptation

    Oct 28, 2026Md. Tanvir Rahman, Taslima Ferdaus Shuva, Nabil Anan Orka +2Multimodal FusionMultimodal Pretraining

  9. MedVTok: A General-Purpose Medical Visual Tokenizer

    Oct 26, 2026Chenglong Ma, Yuanfeng Ji, Junzhi Ning +18Image TokenizationVisual Tokenization

  10. OlmoEarth v1.3: Embeddings

    Oct 23, 2026Gabriel Tseng, Favyen Bastani, Patrick Alan Johnson +13Geospatial Representation LearningMultimodal Embedding

  11. Text-to-Image Generative Modeling Without Language Pretraining

    Oct 19, 2026Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas +5T2I GenerationAutoregressive Image Generation

  12. SleepPairNet: Learning Multimodal Pediatric Sleep Representations with Random-Pair Cross-Attention

    Oct 19, 2026Saurav Raj Pandey, Harlin LeeMultimodal PretrainingCross-Modal Representation Learning

  13. MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

    Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal LearningMultimodal Foundation Models

  14. More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

    Oct 14, 2026Haoyuan Wu, Aoqi Wu, Jiani Hu +1Multimodal PretrainingCross-Modal Learning

  15. MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

    Oct 13, 2026Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo +3Remote SensingGeospatial Foundation Models

  16. Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

    Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive LearningMultimodal Learning

  17. HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignment

    Oct 10, 2026Jiaqi Wang, Wei Chen, Xiaoyu Li +7Representation LearningPhotoplethysmography

  18. Scaling Laws for Multimodal Data Mixtures

    Oct 10, 2026Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5Multimodal PretrainingAI Scaling Laws

  19. Mid-Training Language Models on Raw Video

    Oct 8, 2026Jaedong Hwang, Xiaoqian Shen, Ernie Chang +9Multimodal Large Language ModelsVideo-Language Models

  20. SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

    Oct 7, 2026Jeonghyeok Do, Munchurl KimRemote Sensing Image UnderstandingMultimodal Pretraining

  21. Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

    Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal ModelsMultimodal Pretraining

  22. Advancing Video-Text Pretraining with Multi-View Captions

    Sep 28, 2026Fida M. Thoker, Renaud Vandeghen, Karen Sanchez +2Multimodal Pretraining

  23. InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

    Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal PretrainingMultimodal QA

  24. Clinical Trajectory Alignment for Medical Vision-Language Pre-training

    Sep 28, 2026Huimin Yan, Xian Yang, Zhi Wang +1Multimodal PretrainingLongitudinal Medical Image Analysis

  25. WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation

    Sep 28, 2026Chuan Qin, Shaoting Zhu, Siyuan Luo +4Humanoid Loco-ManipulationMultimodal Pretraining

  26. PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment

    Sep 27, 2026Xiaoda Wang, Minxiao Wang, Maxwell A Xu +10Multimodal PretrainingPhotoplethysmography

  27. Phenomenon-Graph JEPA: Label-Efficient Representation Learning for Contactless Cardiorespiratory Sensing

    Sep 26, 2026Constantino Álvarez Casado, Nhi Nguyen, Mohammad Rakibur Rahman +4Self-Supervised Time Series Representation LearningJoint-Embedding Predictive Architecture