Multimodal Pretraining

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 119

All topics

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal Models Multimodal Pretraining
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Sep 28, 2026Lin Chen, Bolin Ni, Qi Yang +6Multimodal Pretraining Multimodal Large Language Models
CASIA · UCAS · Foundation Model Department, Tencent
Advancing Video-Text Pretraining with Multi-View Captions
Sep 28, 2026Fida M. Thoker, Renaud Vandeghen, Karen Sanchez +2Multimodal Pretraining
King Abdullah University of Science and Technology (KAUST) · University of Li`ege
InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision
Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal Pretraining Multimodal QA
The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute
Clinical Trajectory Alignment for Medical Vision-Language Pre-training
Sep 28, 2026Huimin Yan, Xian Yang, Zhi Wang +1Multimodal Pretraining Longitudinal Medical Image Analysis
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, China · Alliance Manchester Business School, The University of Manchester, Manchester, UK
WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation
Sep 28, 2026Chuan Qin, Shaoting Zhu, Siyuan Luo +4Humanoid Loco-Manipulation Multimodal Pretraining
IIIS, Tsinghua University, Beijing, China. · Xiong’an Institute of Artificial Intelligence, Xiong’an, China. · The University of Melbourne, Melbourne, Australia.
PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment
Sep 27, 2026Xiaoda Wang, Minxiao Wang, Maxwell A Xu +10Multimodal Pretraining Photoplethysmography
Emory University · University of California, Los Angeles · Google +3
TimeBraid: Unifying Time Series and Language for Understanding and Forecasting
Sep 24, 2026Xinyue Wang, Jiacheng Pang, Kun Zhou +7Unified Multimodal Models Multimodal Pretraining
University of California San Diego · Aether AI · University of Southern California
PhyMo: A Physical-Field Modality for Multimodal AI4Physics
Sep 23, 2026Henan Sun, Haitao Hu, Jin Liu +4AI for Science Cross-Modal Representation Learning
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Noah’s Ark Lab · Tencent HY +1
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Missing-Modality Learning Cross-Modal Alignment
Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1
Qwen3.8-Omni: Towards Native Omni-Modal Agents
Sep 22, 2026Qwen TeamMultimodal Pretraining Multimodal Large Language Models
Qwen Team
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Sep 15, 2026Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8Cross-Modal Representation Learning Cross-Modal Alignment
Meta AI
Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning
Sep 14, 2026Andrea Agostini, Simon Böhi, Moritz Vandenhirtz +7Masked Autoencoders Healthcare
Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Sep 8, 2026Siting Li, Zhengyang Wang, Simon Shaolei Du +2Image Tokenization Unified Multimodal Models
University of Washington · Amazon FAR
OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
Sep 7, 2026Yuran Wang, Siqiao Huang, Mingleyang Li +21World Model Learning World Models for Robotics
National University of Singapore · Tsinghua University · Peking University +4
MEOX: Compact Multimodal Mixture-of-Experts for Earth Observation
Sep 4, 2026Mohanad AlbughdadiMasked Autoencoders Geospatial Representation Learning
European Centre for Medium-Range Weather Forecasts Bonn, Germany
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
Sep 3, 2026Xiangyang Miao, Kelu Yao, Yekai Huang +7Remote Sensing Image Understanding Multimodal Pretraining
School of Computer Science and Technology, Zhejiang University · Space-based Computing System Research Center, Zhejiang Lab
GAAT: Geometry-Aware Alignment Transformer for Multimodal UAV Perception
Aug 28, 2026Jingpu Yang, Debin Tang, Yilin Sun +4Cross-Modal Alignment Multimodal Pretraining
Beihang University, Beijing, China. · Zhongguancun Academy, Beijing, China. · Northeastern University, Shenyang, China.
Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules
Aug 23, 2026Florian Rottach, Sebastian Schieferdecker, William Rudman +2Joint-Embedding Predictive Architecture Multimodal Pretraining
University of Tübingen · 2Boehringer Ingelheim · 3The University of Texas at Austin +1
Fusion Anything: A Generalized Multimodal Foundation Model
Aug 16, 2026Huizi Cui, Zongbo Han, Chenggong Ding +6Unified Multimodal Models Multimodal Pretraining
Tianjin University · Beijing University of Posts and Telecommunications
Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
Aug 13, 2026Zhi Qiao, Xintong Wu, Yichu He +1Medical Imaging Foundation Models 3D Medical Imaging
United Imaging Intelligence, Shanghai, China
Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
Aug 12, 2026Zhongbin Guo, Jiahao Xie, Dongling Xiao +5Multimodal Pretraining Multimodal Understanding
ByteDance
MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
Aug 11, 2026Changhao Xiang, Shangyu Xing, Zhen Wu +2Multimodal Grounding Multimodal Pretraining
National Key Laboratory for Novel Software Technology, Nanjing University, China
Unlocking the Power of Medical Tabular Data via Semantic-Aware Multimodal Pre-training
Aug 11, 2026Yingsheng Liu, Haiming Li, Jingmin Zhu +6Cross-Modal Representation Learning Multimodal Pretraining
Faculty of Information Technology, Monash University, Melbourne, Australia · Monash University, Melbourne, Victoria, Australia · The University of Queensland, Brisbane, Queensland, Australia
SLED: Scalable Location Encoding via Distillation
Aug 6, 2026Kevin Lane, Zhongying Wang, Esther Rolf +1Geospatial Representation Learning Multimodal Pretraining
University of Colorado Boulder
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
Aug 6, 2026Omar Coser, Antonio Orvieto, Paolo Soda +1Time Series Classification Self-Supervised Time Series Representation Learning
Unit of Artificial Intelligence & Computer Systems, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Unit of Advanced Robotics and Human-Centered Technologies, Universit`a Campus Bio-Medico di Roma, Via ´Alvaro del Portillo, 21, Rome, 00128, Italy. · Max Planck Institute for Intelligent Systems +2
Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation
Aug 6, 2026Théo Danielou, Antoine Saporta, Léo Alberge +1Medical Imaging Foundation Models Masked Autoencoders
Raidium
SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment
Aug 6, 2026Yin-Loon Khor, Yi-Jie Wong, Jing Jie Tan +1VLM Adaptation Multimodal Pretraining
Universiti Malaya · Universiti Tunku Abdul Rahman · National University of Singapore
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal Models Multimodal Pretraining
1FAIR · University of Oxford · 2Reality Labs
Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language
Aug 5, 2026Xinran Feng, Yi Xie, Chao Zhang +4Cross-Modal Representation Learning Multimodal Pretraining
Thrust of Financial Technology, The Hong Kong University of Science and Technology (Guangzhou) · Technical University of Munich · Heilbronn Data Science Center, Munich Data Science Institute +1
ReMiX-MAE: Learning Missing-Channel Cross-Modal Representations from RGB-Only Clinical Facial Videos for Sympathetic-Mediated Pain Assessment
Aug 3, 2026Nan Bi, Taoyue Wang, Lijun Yin +1Missing-Modality Learning Masked Autoencoders
School of Computing, Binghamton University, Binghamton, NY, USA · Department of Anesthesiology, SUNY Upstate Medical University, Syracuse, NY, USA
HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
Aug 3, 2026Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu +1Multi-Source Domain Adaptation Chest X-Ray Classification
ARTORG Center for Biomedical Engineering Research, University of Bern · Kaiko.AI, Zurich, Switzerland · Shanghai Jiao Tong University, Shanghai, China
Douyin Multimodal Embedding Model Technical Report
Aug 3, 2026Haonan Chen, Chu Li, Zhicheng Wang +4Multimodal IR Multimodal Pretraining
ByteDance Douyin Search Multimodal Team · Renmin University of China GSAI
GALA: Generative Aligned Learning for Adaptive Multimodal Representation in the Taobao Shangou Recommender System
Jul 31, 2026Jiping Liu, Zhongmin Zhang, Zisen Sang +7Cross-Modal Alignment Multimodal Pretraining
Rajax Network Technology (Taobao Shangou of Alibaba) · Shanghai & Beijing, China · Central South University +1
Data Pyramid for Embodied Manipulation: A Survey
Jul 27, 2026Yifan Ye, Yankai Fu, Yaoxu Lv +26Robotic Data Collection Robot Foundation Models
PKU · NTU · HKUST +8
Scaling GUI Agents with Visual State Transitions
Jul 27, 2026Xiangyan Liu, Kaixin Li, Haonan Wang +6World Models GUI Agents
NUS · UTS · University of Liverpool +1
mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction
Jul 25, 2026Cheng Guo, Qiming Cao, Shengkai Xu +4Synthetic Data Pretraining Sim-to-Real Transfer
University of North Carolina at Charlotte, Charlotte, NC, USA · Purdue University, West Lafayette, IN, USA
LunarFM: A Shared Multimodal Representation of the Moon's Surface
Jul 24, 2026Marc Girona-Mata, Jakob Gawlikowski, Sumit Goski +6Geospatial Representation Learning Multimodal Pretraining
German Aerospace Center (DLR) · European Space Resources Innovation Center (ESRIC) · Universidad de Antioquia
Scaling Native Multimodal Pre-Training From Scratch
Jul 24, 2026Haoyuan Wu, Aoqi Wu, Hai Wang +3Cross-Modal Learning Multimodal Pretraining
1The Chinese University of Hong Kong · 2LLM Department, Tencent
Multimodal Pretraining for Generalizable EEG Representation Learning
Jul 23, 2026Targol Bakhtiarvand, Jugal Kalita, Adham AtyabiCross-Subject EEG Decoding Electroencephalography
Department of Computer Science, University of Colorado Colorado Springs, Colorado Springs, CO, USA
Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design
Jul 22, 2026Xiaoliang Shi, Zichen Wang, Runze Ma +2Multimodal Pretraining Protein Design
Shanghai Jiao Tong University
scMIR: a vision-language foundation model for single-cell light microscopy image representation
Jul 21, 2026Yifan Shang, Jiahui Tan, Xiangxiang Zeng +1Cross-Modal Representation Learning Visual Representation Learning
Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China
Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation
Jul 20, 2026Xiaohan Ye, Xu Chen, Zihan Gong +15Multimodal IR Multimodal Pretraining
Taobao & Tmall Group of Alibaba
Monkey King Bang: A Unified Scientific Multimodal Foundation Model
Jul 17, 2026Hesen Chen, Xinyu Su, Xiaomeng Yang +11AI for Science Unified Multimodal Models
SAIS Team, Shanghai Academy of AI for Science
Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Jul 17, 2026Xunkai Li, Guohao Fu, Yuming Ai +4Multimodal Federated Learning Cross-Modal Alignment
Beijing Institute of Technology, Beijing, 100811, China
Video = World + Event Stream
Jul 16, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +24Audio-Visual Understanding Streaming Video Understanding
Wan Team, Alibaba Group
Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging
Jul 16, 2026Sara Ketabi, Matthias W. Wagner, Cynthia Hawkins +3Contrastive Learning 3D Medical Imaging
University of Toronto · The Hospital for Sick Children · University Hospital Augsburg +1
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal Learning Cross-Modal Representation Learning
Swiss Federal Institute of Technology Lausanne (EPFL) · Apple
Empowering Long-form Omni-modal Understanding with Robust Audio Perception
Jul 11, 2026Kaiying Yan, Luoyi Sun, Xiao Zhou +1Audio-Visual Understanding Multimodal Pretraining
SAI, Shanghai Jiao Tong University, shanghai, China · Zhejiang University, Hangzhou, Zhejiang, china · Shanghai AI Lab, shanghai, China
A Foundation Model for Multimodal Event Sequences in Financial Applications
Jul 10, 2026Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4Financial Services Multimodal Pretraining
Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation
MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
Jul 8, 2026Hyunjae Kim, Dain Kim, Pan Xiao +25Medical Imaging Foundation Models Training Data Curation
Yale University, New Haven, CT, USA · Korea University, Seoul, South Korea · Washington University in St. Louis, St. Louis, MO, USA +4
GaussFusion: Towards Multimodal 3D Gaussian Pretraining
Jul 7, 2026Zhixuan You, Jihua Zhu, Yiding Sun +53D Gaussian Splatting Cross-Modal Representation Learning
School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710100, Shaanxi, China · Wuhu HIT Robot Technology Research Institute Co., Ltd., Wuhu, Anhui, China
Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing
Jul 3, 2026Ayush Prasad, Swarnalee MazumderMissing-Modality Learning Masked Autoencoders
Space and Earth Observation Centre, Finnish Meteorological Institute, Finland · Data Management, German Climate Computing Centre, Germany
Rosetta: Composable Native Multimodal Pretraining
Jul 1, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Unified Multimodal Models Multimodal Pretraining
HKUST · Tencent Hunyuan
Orca: The World is in Your Mind
Jun 29, 2026Yihao Wang, Yuheng Ji, Mingyu Cao +54World Model Learning Multimodal Pretraining
Heterogeneous Tactile Transformer
Jun 29, 2026Jianxin Bi, Qiang Wang, Jayaram Reddy +4Representation Learning Multimodal Pretraining
National University of Singapore · Carnegie Mellon University · 3Smart Systems Institute, NUS
NIVA: A Multimodal Foundation Model for Actionable Earth System Intelligence
Jun 26, 2026Anisha Pal, Aodhan Sweeney, Kyle Heyblom +1Climate Science Multimodal Pretraining
Independent Researcher · Work done while at Planette AI · Planette AI, USA.
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Jun 25, 2026Xumin Yu, Zuyan Liu, Zhenyu Yang +5Visual Representation Learning Multimodal Pretraining
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1
MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning
Jun 23, 2026Revant Teotia, Adrien Bardes, Michael Rabbat +3Audio-Visual Representation Learning Joint-Embedding Predictive Architecture
1FAIR at Meta · 2New York University
BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Jun 20, 2026Qizhi Pei, Zhimeng Zhou, Yi Duan +9Unified Multimodal Models Multimodal Pretraining
1Gaoling School of Artificial Intelligence, Renmin University of China, 2OpenDataLab, Shanghai Artificial Intelligence Laboratory · 3Zhejiang University, 4Shanghai Innovation Institute · 2OpenDataLab, Shanghai Artificial Intelligence Laboratory, 3Zhejiang University, 4Shanghai Innovation Institute +5

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li+6

Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics

Unified Multimodal Models Multimodal Pretraining Multimodal Generation Flow-Based Generative Modeling

Open paper AskPDF arXiv

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/Multimodal-Flow.

Momentum

Related topics

Latest papers 119