Multimodal Pretraining

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 121

All topics

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language
Jun 20, 2026Qizhi Pei, Zhimeng Zhou, Yi Duan +9Unified Multimodal Models Multimodal Pretraining
1Gaoling School of Artificial Intelligence, Renmin University of China, 2OpenDataLab, Shanghai Artificial Intelligence Laboratory · 3Zhejiang University, 4Shanghai Innovation Institute · 2OpenDataLab, Shanghai Artificial Intelligence Laboratory, 3Zhejiang University, 4Shanghai Innovation Institute +5
MixProLAP: Mixture-Induced Uncertainty Modeling for Probabilistic Language-Audio Pretraining
Jun 18, 2026Yu Nakagome, Jaesong Lee, Soo-Whan ChungMultimodal Pretraining Audio-Language Models
LINE WORKS Corporation, Japan · NAVER Cloud Corporation, South Korea
Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying
Jun 18, 2026Jonathan Hecht, Lukas Arzoumanidis, Ziyue Li +1Self-Supervised Pre-Training Geospatial Representation Learning
Computational Methods Lab, HafenCity University Hamburg, Germany. · Institute of Geodesy and Geoinformation, University of Bonn, Germany. · Dept. of Operations & Technology, Technical University of Munich; Heilbronn Data Science Center; Munich Data Science Institute.
Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model
Jun 16, 2026Jinghan Wu, Jing Li, Ivor W. Tsang +1Coreference Resolution VLM Adaptation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China · Centre for Frontier AI Research and the Institute of High-Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138634
See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RL
Jun 16, 2026Yilian Liu, Sicong Leng, Guoshun Nan +7Multimodal Pretraining Visually Grounded Reasoning
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · China Telecom, China
FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models
Jun 15, 2026Jiaju Han, Ben Zhang, Xuemeng Sun +6Remote Sensing Image Understanding Multimodal Pretraining
China University of Petroleum-Beijing at Karamay · University of Electronic Science and Technology of China · Tianjin University
CAP: Towards PPG Universal Representation Learning with Patient-level Supervision
Jun 13, 2026Chenyang He, Xinyi Shao, Shun Huang +4Representation Learning Cross-Modal Representation Learning
Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Peking University Beijing, Beijing, China · Independent Researcher +1
Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging
Jun 12, 2026Haoxu Huang, Long Chen, Jingyun Chen +8Medical Imaging Foundation Models Joint-Embedding Predictive Architecture
New York University, Center for Data Science, New York, NY, 10001, USA · NYU Grossman School of Medicine, Department of Radiology, New York, NY, 10016, USA · State University of New York at Binghamton, School of Computing, Binghamton, NY 13902, USA +7
SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs
Jun 11, 2026Zhengyu Wu, Xu Wang, Hongchao Qin +4Graph Representation Learning Cross-Modal Alignment
GLACIER: A Multimodal Student-Teacher Foundation Model for Molecular Property Prediction
Jun 9, 2026Emily Nguyen, Yongchan Hong, Harsh Toshniwal +2Multimodal Pretraining Molecular Property Prediction
Department of Medical Biochemistry and Biophysics Science for Life Laboratory Karolinska Institutet Stockholm, Sweden · Amazon
Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
Jun 8, 2026Jonathan F. Carter, Lionel TarassenkoSleep Stage Classification Time Series Classification
Institute of Biomedical Engineering University of Oxford
CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning
Jun 8, 2026Penghui Yang, Long Xing, Xiaoyi Dong +10Image Captioning Multimodal Pretraining
Tsinghua University · University of Science and Technology of China · Microsoft +4
A robust PPG foundation model using multimodal physiological supervision
Jun 5, 2026Eloy Geenjaar, Vince Calhoun, Scott Daly +4Multimodal Robustness Multimodal Pretraining
Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, USA · Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, USA · Dolby Laboratories, San Francisco, USA
BRepCLIP: Contrastive Multimodal Pretraining on BRep Primitives for CAD Understanding
Jun 3, 2026Muhammad Usama, Didier Stricker, Mohammad Sadil Khan +1Multimodal Pretraining Multimodal Contrastive Learning
1DFKI, Germany · 2RPTU Kaiserslautern-Landau, Germany
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
May 31, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +10Vision-Language Models Multimodal Pretraining
The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute
UniRTL: Unifying Code and Graph for Robust RTL Representation Learning
May 29, 2026Yi Liu, Hongji Zhang, Lei Chen +2Representation Learning Cross-Modal Representation Learning
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR · Noah’s Ark Lab, Huawei, Hong Kong SAR
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
May 28, 2026Jusuk Lee, Seungjae Lee, Jonghun Shin +6Cross-Modal Learning Cross-Modal Representation Learning
Seoul National University · University of Maryland, College Park · Georgia Institute of Technology
Archon: A Unified Multimodal Model for Holistic Digital Human Generation
May 28, 2026Chong Bao, Shichen Liu, Lijun Yu +9Unified Multimodal Models Multimodal Pretraining
State Key Lab of CAD&CG, Zhejiang University · Google · The work was done during an internship at Google. +1
VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading
May 27, 2026Jinzhou Wu, Zhengwu Ma, Jixing Li +2VLM Evaluation Vision-Language Models
Department of Mechanical and Vehicle Engineering, Chongqing University · Department of Linguistics and Translation, City University of Hong Kong · McGovern Institute for Brain Research, Massachusetts Institute of Technology
FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales
May 27, 2026Jorge L. Rodriguez, Victor Angulo Morales, Areej Alwahas +6Remote Sensing Image Segmentation Geospatial Foundation Models
Biological and Environmental Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals
May 26, 2026Phu X. Nguyen, Konstantinos Kontras, Wei Dai +5Representation Learning Cross-Modal Representation Learning
1KU Leuven · 3MIT · 4DFKI +1
FTibSuite: A Comprehensive Resource Suite for Tibetan Vision-Language Modeling
May 26, 2026Guixian Xu, Yide Liang, Zeli Su +5VLM Evaluation Vision-Language Models
Hainan International College, Minzu University of China · School of Information Engineering, Minzu University of China · Institute of Automation, Chinese Academy of Sciences +1
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
May 24, 2026Prabuddhi Wariyapperuma, Rajitha de Silva, Marc Hanheide +2Lidar-Based 3D Object Detection Masked Autoencoders
University of Lincoln, Lincoln Centre for Autonomous Systems, Lincoln, UK. · University of Cambridge, Institute for Manufacturing, Department of Engineering, Cambridge, UK.
SLIP-RS: Structured-Attribute Language-Image Pre-Training for Remote Sensing Object Detection
May 22, 2026Chenxu Wang, Yuxuan Li, Yunheng Li +3Open-Vocabulary Object Detection Remote Sensing Image Understanding
1VCIP, CS, Nankai University · 2NKIARI, Shenzhen Futian · 3National University of Defense Technology
Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models
May 21, 2026Yuting He, Chenyu You, Shuo LiMedical Imaging Foundation Models Multimodal Pretraining
Case Western Reserve University · Stony Brook University
An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation
May 20, 2026Xiaofeng Liu, Qianru Zhang, Thibault Marin +4Cross-Modality Medical Image Segmentation Medical Imaging Foundation Models
Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA
SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining
May 20, 2026Nassim Ait Ali Braham, Aaron Banze, Conrad M. Albrecht +3Geospatial Foundation Models Multimodal Pretraining
Chair of Data Science in Earth Observation, Technical University of Munich, Germany · Remote Sensing Technology Institute, German Aerospace Center (DLR), Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany +3
Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding
May 18, 2026Paul Quinlan, Jeremy Levasseur, Qingguo Li +1Cross-Modal Learning Multivariate Time Series Forecasting
1InertialAI · Department of Electrical and Computer Engineering, Queen’s University · Department of Mechanical and Materials Engineering, Queen’s University
ITGPT: Generative Pretraining on Irregular Timeseries
May 15, 2026Antoine Honoré, Ming XiaoIrregular Time-Series Modeling Self-Supervised Time Series Representation Learning
⋆Division of Information Science and Engineering, KTH Royal Institute of Technology, Stockholm, Sweden
GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation
May 14, 2026Yuhao Liu, Sadeer Al-Kindi, Ashok Veeraraghavan +1Geospatial Foundation Models Geospatial Representation Learning
Department of Electrical and Computer Engineering, Rice University, Houston, TX 77005 · Center for Cardiovascular Computational and Precision Health, Department of Cardiology, DeBakey Heart and Vascular Center, Houston Methodist, Houston, TX 77030

Jun 20, 2026, cs.CLJ/K move · Enter open · S save

BioMatrix: Towards a Comprehensive Biological Foundation Model Spanning the Modality Matrix of Sequences, Structures, and Language

Qizhi Pei, Zhimeng Zhou, Yi Duan, Yiyang Zhao+8

1Gaoling School of Artificial Intelligence, Renmin University of China, 2OpenDataLab, Shanghai Artificial Intelligence Laboratory · 3Zhejiang University, 4Shanghai Innovation Institute · 2OpenDataLab, Shanghai Artificial Intelligence Laboratory, 3Zhejiang University, 4Shanghai Innovation Institute+5

Unified Multimodal Models Multimodal Pretraining Protein Language Models Multimodal Generation Multimodal Foundation Models Molecular Representation Learning+1

Open paper AskPDF arXiv

We present BioMatrix, the first multimodal foundation model that natively integrates sequences, structures, and natural language for both molecules and proteins within a single decoder-only architecture. Existing biological foundation models pursue native multimodality and broad entity coverage separately: those that fuse multiple modalities under a shared objective remain confined to a single entity type, while those spanning multiple entity types either omit explicit structural modeling or rely on adapter-based designs in which the model cannot natively generate the very modalities it can read. BioMatrix closes this gap by mapping molecular sequences (supporting both SMILES and SELFIES notations), molecular structures, protein sequences, protein structures, and natural language into a shared discrete token space through a unified tokenization scheme, so that all modalities are consumed and produced uniformly under a single next-token prediction objective -- without external encoders, projection adapters, or modality-specific output heads. Built upon the Qwen3 language model (1.7B and 4B), BioMatrix is continually pretrained on 304.4 billion tokens spanning general and domain-specific text, sequence and structure views of molecules and proteins, and cross-modal corpora that interleave biomolecular entities with scientific text and link distinct entities through molecule-protein and protein-protein interaction data. After tuning on a comprehensive suite of downstream applications covering 80 tasks across 6 categories -- encompassing single-entity and multi-entity understanding and generation tasks across and within modalities -- BioMatrix achieves state-of-the-art or competitive performance on 77 out of 80 tasks, demonstrating that a single, natively multimodal generalist model can effectively match or surpass specialized approaches across a wide range of biological tasks.

Momentum

Related topics

Latest papers 121