Multimodal Pretraining

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 121

All topics

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models
Oct 7, 2026Jeonghyeok Do, Munchurl KimRemote Sensing Image Understanding Multimodal Pretraining
Korea Advanced Institute of Science and Technology (KAIST)
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal Models Multimodal Pretraining
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
Advancing Video-Text Pretraining with Multi-View Captions
Sep 28, 2026Fida M. Thoker, Renaud Vandeghen, Karen Sanchez +2Multimodal Pretraining
King Abdullah University of Science and Technology (KAUST) · University of Li`ege
InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision
Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal Pretraining Multimodal QA
The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute
Clinical Trajectory Alignment for Medical Vision-Language Pre-training
Sep 28, 2026Huimin Yan, Xian Yang, Zhi Wang +1Multimodal Pretraining Longitudinal Medical Image Analysis
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK
WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation
Sep 28, 2026Chuan Qin, Shaoting Zhu, Siyuan Luo +4Humanoid Loco-Manipulation Multimodal Pretraining
IIIS, Tsinghua University, Beijing, China. · Xiong’an Institute of Artificial Intelligence, Xiong’an, China. · The University of Melbourne, Melbourne, Australia.
PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment
Sep 27, 2026Xiaoda Wang, Minxiao Wang, Maxwell A Xu +10Multimodal Pretraining Photoplethysmography
Emory University · University of California, Los Angeles · Google +3
TimeBraid: Unifying Time Series and Language for Understanding and Forecasting
Sep 24, 2026Xinyue Wang, Jiacheng Pang, Kun Zhou +7Unified Multimodal Models Multimodal Pretraining
University of California San Diego · Aether AI · University of Southern California
PhyMo: A Physical-Field Modality for Multimodal AI4Physics
Sep 23, 2026Henan Sun, Haitao Hu, Jin Liu +4AI for Science Cross-Modal Representation Learning
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Noah’s Ark Lab · Tencent HY +1
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Missing-Modality Learning Cross-Modal Alignment
Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1
Qwen3.8-Omni: Towards Native Omni-Modal Agents
Sep 22, 2026Qwen TeamMultimodal Pretraining Multimodal Large Language Models
Qwen Team
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Sep 15, 2026Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8Cross-Modal Representation Learning Cross-Modal Alignment
Meta AI
Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning
Sep 10, 2026Andrea Agostini, Simon Böhi, Moritz Vandenhirtz +7Masked Autoencoders Healthcare
Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Sep 8, 2026Siting Li, Zhengyang Wang, Simon Shaolei Du +2Image Tokenization Unified Multimodal Models
University of Washington · Amazon FAR
OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
Sep 7, 2026Yuran Wang, Siqiao Huang, Mingleyang Li +21World Model Learning World Models for Robotics
National University of Singapore · Tsinghua University · Peking University +4
MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation
Sep 4, 2026Mohanad AlbughdadiMasked Autoencoders Geospatial Representation Learning
European Centre for Medium-Range Weather Forecasts Bonn, Germany
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
Sep 3, 2026Xiangyang Miao, Kelu Yao, Yekai Huang +7Remote Sensing Image Understanding Multimodal Pretraining
School of Computer Science and Technology, Zhejiang University · Space-based Computing System Research Center, Zhejiang Lab
GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
Aug 28, 2026Jingpu Yang, Debin Tang, Yilin Sun +4Cross-Modal Alignment Multimodal Pretraining
Beihang University, Beijing, China. · Zhongguancun Academy, Beijing, China. · Northeastern University, Shenyang, China.
Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules
Aug 23, 2026Florian Rottach, Sebastian Schieferdecker, William Rudman +2Joint-Embedding Predictive Architecture Multimodal Pretraining
University of Tübingen · 2Boehringer Ingelheim · 3The University of Texas at Austin +1
Fusion Anything: A Generalized Multimodal Foundation Model
Aug 16, 2026Huizi Cui, Zongbo Han, Chenggong Ding +6Unified Multimodal Models Multimodal Pretraining
Tianjin University · Beijing University of Posts and Telecommunications
Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
Aug 13, 2026Zhi Qiao, Xintong Wu, Yichu He +1Medical Imaging Foundation Models 3D Medical Imaging
United Imaging Intelligence, Shanghai, China
Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
Aug 12, 2026Zhongbin Guo, Jiahao Xie, Dongling Xiao +5Multimodal Pretraining Multimodal Understanding
ByteDance
MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
Aug 11, 2026Changhao Xiang, Shangyu Xing, Zhen Wu +2Multimodal Grounding Multimodal Pretraining
National Key Laboratory for Novel Software Technology, Nanjing University, China
Unlocking the Power of Medical Tabular Data via Semantic-Aware Multimodal Pre-training
Aug 11, 2026Yingsheng Liu, Haiming Li, Jingmin Zhu +6Cross-Modal Representation Learning Multimodal Pretraining
Faculty of Information Technology, Monash University, Melbourne, Australia · Monash University, Melbourne, Victoria, Australia · The University of Queensland, Brisbane, Queensland, Australia
SLED: Scalable Location Encoding via Distillation
Aug 6, 2026Kevin Lane, Zhongying Wang, Esther Rolf +1Geospatial Representation Learning Multimodal Pretraining
University of Colorado Boulder
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
Aug 6, 2026Omar Coser, Antonio Orvieto, Paolo Soda +1Time Series Classification Self-Supervised Time Series Representation Learning
Unit of Artificial Intelligence & Computer Systems, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Unit of Advanced Robotics and Human-Centered Technologies, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Max Planck Institute for Intelligent Systems +2
Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation
Aug 6, 2026Théo Danielou, Antoine Saporta, Léo Alberge +1Medical Imaging Foundation Models Masked Autoencoders
Raidium
SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment
Aug 6, 2026Yin-Loon Khor, Yi-Jie Wong, Jing Jie Tan +1VLM Adaptation Multimodal Pretraining
Universiti Malaya · Universiti Tunku Abdul Rahman · National University of Singapore
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

Jeonghyeok Do, Munchurl Kim

Korea Advanced Institute of Science and Technology (KAIST)

Remote Sensing Image Understanding Multimodal Pretraining Multimodal Foundation Models Cross-Modal Representation Learning Masked Image Modeling

Open paper AskPDF arXiv

Synthetic aperture radar (SAR) and electro-optical (EO) imagery provide complementary observations: SAR enables day-and-night, weather-resilient sensing, whereas EO provides rich appearance and fine-grained semantic cues. We introduce SAREO-FM, which avoids forcing a single token stream to serve two distinct roles: modality tokens preserve how each sensor observes the scene through masked reconstruction, while learnable semantic queries capture what the scene contains under guidance from a pretrained vision foundation model (VFM). By jointly encoding these queries with SAR and EO tokens, the queries acquire modality-grounded semantic context, while the modality-token outputs remain the explicit targets of masked reconstruction. This design assigns semantic and reconstruction supervision to separate token streams while preserving their interaction within the shared encoder. Pretrained on the million-scale SAR-1M corpus, SAREO-FM achieves strong unimodal transfer for both SAR-only and EO-only inputs, while delivering substantial gains from joint SAR--EO observations on tasks that benefit from complementary sensing.

Momentum

Related topics

Latest papers 121