Multimodal Pretraining

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 122

All topics

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs
Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language
Aug 5, 2026Xinran Feng, Yi Xie, Chao Zhang +4Cross-Modal Representation Learning Multimodal Pretraining
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1
ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment
Aug 3, 2026Nan Bi, Taoyue Wang, Lijun Yin +1Missing-Modality Learning Masked Autoencoders
School of Computing, Binghamton University, Binghamton, NY, USA · Department of Anesthesiology, SUNY Upstate Medical University, Syracuse, NY, USA
HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
Aug 3, 2026Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu +1Multi-Source Domain Adaptation Chest X-Ray Classification
ARTORG Center for Biomedical Engineering Research, University of Bern · Kaiko.AI, Zurich, Switzerland · Shanghai Jiao Tong University, Shanghai, China
Douyin Multimodal Embedding Model Technical Report
Aug 3, 2026Haonan Chen, Chu Li, Zhicheng Wang +4Multimodal IR Multimodal Pretraining
ByteDance Douyin Search Multimodal Team · Renmin University of China GSAI
GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System
Jul 31, 2026Jiping Liu, Zhongmin Zhang, Zisen Sang +7Cross-Modal Alignment Multimodal Pretraining
Rajax Network Technology (Taobao Shangou of Alibaba) · Shanghai & Beijing, China · Central South University +1
Data Pyramid for Embodied Manipulation: A Survey
Jul 27, 2026Yifan Ye, Yankai Fu, Yaoxu Lv +26Robotic Data Collection Robot Foundation Models
PKU · NTU · HKUST +8
Scaling GUI Agents with Visual State Transitions
Jul 27, 2026Xiangyan Liu, Kaixin Li, Haonan Wang +6World Models GUI Agents
NUS · UTS · University of Liverpool +1
mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction
Jul 25, 2026Cheng Guo, Qiming Cao, Shengkai Xu +4Synthetic Data Pretraining Sim-to-Real Transfer
University of North Carolina at Charlotte, Charlotte, NC, USA · Purdue University, West Lafayette, IN, USA
LunarFM: A Shared Multimodal Representation of the Moon's Surface
Jul 24, 2026Marc Girona-Mata, Jakob Gawlikowski, Sumit Goski +6Geospatial Representation Learning Multimodal Pretraining
German Aerospace Center (DLR) · European Space Resources Innovation Center (ESRIC) · Universidad de Antioquia
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
The Chinese University of Hong Kong · LLM Department, Tencent
Multimodal Pretraining for Generalizable EEG Representation Learning
Jul 23, 2026Targol Bakhtiarvand, Jugal Kalita, Adham AtyabiCross-Subject EEG Decoding Electroencephalography
Department of Computer Science, University of Colorado Colorado Springs, Colorado Springs, CO, USA
Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design
Jul 22, 2026Xiaoliang Shi, Zichen Wang, Runze Ma +2Multimodal Pretraining Protein Design
Shanghai Jiao Tong University
scMIR: a vision-language foundation model for single-cell light microscopy image representation
Jul 21, 2026Yifan Shang, Jiahui Tan, Xiangxiang Zeng +1Cross-Modal Representation Learning Visual Representation Learning
Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China
Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation
Jul 20, 2026Xiaohan Ye, Xu Chen, Zihan Gong +15Multimodal IR Multimodal Pretraining
Taobao & Tmall Group of Alibaba
Monkey King Bang: A Unified Scientific Multimodal Foundation Model
Jul 17, 2026Hesen Chen, Xinyu Su, Xiaomeng Yang +11AI for Science Unified Multimodal Models
SAIS Team, Shanghai Academy of AI for Science
Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Jul 17, 2026Xunkai Li, Guohao Fu, Yuming Ai +4Multimodal Federated Learning Cross-Modal Alignment
Beijing Institute of Technology, Beijing, 100811, China
Video = World + Event Stream
Jul 16, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +24Audio-Visual Understanding Streaming Video Understanding
Wan Team, Alibaba Group
Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging
Jul 16, 2026Sara Ketabi, Matthias W. Wagner, Cynthia Hawkins +3Contrastive Learning 3D Medical Imaging
University of Toronto · The Hospital for Sick Children · University Hospital Augsburg +1
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal Learning Cross-Modal Representation Learning
Swiss Federal Institute of Technology Lausanne (EPFL) · Apple
Empowering Long-form Omni-modal Understanding with Robust Audio Perception
Jul 11, 2026Kaiying Yan, Luoyi Sun, Xiao Zhou +1Audio-Visual Understanding Multimodal Pretraining
SAI, Shanghai Jiao Tong University, shanghai, China · Zhejiang University, Hangzhou, Zhejiang, china · Shanghai AI Lab, shanghai, China
A Foundation Model for Multimodal Event Sequences in Financial Applications
Jul 10, 2026Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4Financial Services Multimodal Pretraining
Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Jul 8, 2026Hyunjae Kim, Dain Kim, Pan Xiao +25Medical Imaging Foundation Models Training Data Curation
Yale University, New Haven, CT, USA · Korea University, Seoul, South Korea · Washington University in St. Louis, St. Louis, MO, USA +4
GaussFusion: Towards Multimodal 3D Gaussian Pretraining
Jul 7, 2026Zhixuan You, Jihua Zhu, Yiding Sun +53D Gaussian Splatting Cross-Modal Representation Learning
School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710100, Shaanxi, China · Wuhu HIT Robot Technology Research Institute Co., Ltd., Wuhu, Anhui, China
Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing
Jul 3, 2026Ayush Prasad, Swarnalee MazumderMissing-Modality Learning Masked Autoencoders
Space and Earth Observation Centre, Finnish Meteorological Institute, Finland · Data Management, German Climate Computing Centre, Germany
Rosetta: Composable Native Multimodal Pretraining
Jul 1, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Unified Multimodal Models Multimodal Pretraining
HKUST · Tencent Hunyuan
Orca: The World is in Your Mind
Jun 29, 2026Yihao Wang, Yuheng Ji, Mingyu Cao +54World Model Learning Multimodal Pretraining
Heterogeneous Tactile Transformer
Jun 29, 2026Jianxin Bi, Qiang Wang, Jayaram Reddy +4Representation Learning Multimodal Pretraining
National University of Singapore · Carnegie Mellon University · 3Smart Systems Institute, NUS
NIVA: A Multimodal Foundation Model for Actionable Earth System Intelligence
Jun 26, 2026Anisha Pal, Aodhan Sweeney, Kyle Heyblom +1Climate Science Multimodal Pretraining
Independent Researcher · Work done while at Planette AI · Planette AI, USA.
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Jun 25, 2026Xumin Yu, Zuyan Liu, Zhenyu Yang +5Visual Representation Learning Multimodal Pretraining
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1

Aug 5, 2026, cs.CVJ/K move · Enter open · S save

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Junlin Han, Shengbang Tong, David Fan, Minghao Chen+3

1FAIR · University of Oxford · 2Reality Labs

Unified Multimodal Models Multimodal Pretraining Transfer Learning Multimodal Fusion

Open paper AskPDF arXiv

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

Momentum

Related topics

Latest papers 122