Multimodal Foundation Models

Overview

Multimodal foundation models study how to train systems that combine inputs such as text, images, audio, video, medical scans, and sensor data, then use them for tasks from question answering to generation and control. Common approaches connect pretrained modality-specific encoders to language models with projectors or adapters, align representations through contrastive learning, or train unified autoregressive and diffusion models to understand and generate multiple modalities. Evaluation spans visual reasoning and retrieval benchmarks, long-video and audio-visual tasks, and real-world settings such as clinical analysis, document understanding, and robotics. A central tension is that strong scores on individual tasks do not ensure grounded, consistent use of every modality, while broader capabilities also raise challenges in data quality, inference cost, robustness, and safety.

Latest papers

CardsList

365 papers11 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

    Nov 19, 2026Changli Tang, Jie Yang, Tianyi Wang +9Multimodal Large Language ModelsMultimodal Foundation Models

  2. MatVerse: A Foundation Model for Materials Science

    Nov 19, 2026Xuanyu Zhu, Qian Tan, Bo Chen +15Materials ScienceMultimodal Foundation Models

  3. AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration

    Nov 16, 2026Xiangrui Xu, Sicong Liu, Zhiwen Yu +2Multimodal Foundation ModelsEdge-Cloud Collaborative Inference

  4. FinVision: Multimodal AI for Visualizing Financial Narratives

    Nov 15, 2026Yixuan Tang, Hui Wang, Yi Yang +1Multimodal GenerationData Visualization

  5. Unbundling Multimodal Vision Encoding

    Nov 12, 2026Ellis Brown, Chris Hoang, Shengyi Qian +9Multimodal Foundation ModelsMultimodal Learning

  6. A Time-Frequency-Condition Multimodal Fusion Foundation Model for Universal Rotating-Machinery Fault Diagnosis under Multi-Element Heterogeneous Scenarios

    Nov 11, 2026Xinyu Zou, Jian Ma, Liang Ma +3Industrial Fault DiagnosisBearing Fault Diagnosis

  7. A Generative Foundation Model for Multimodal Tokamak Tasks

    Nov 5, 2026Yingshuai Ji, Haodong Feng, Guohong Deng +2Multimodal Foundation ModelsPhysical Foundation Models

  8. Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

    Oct 27, 2026Jialv Zou, Xiaotong Li, Zhengrong Yue +6Discrete Visual TokenizationVisual Representation Learning

  9. YingLan: A Multimodal Ocean Foundation Model for Unified Representation Learning from Heterogeneous Observations

    Oct 21, 2026Sen Jia, Siqi Wei, Junyu Wang +8Multimodal Foundation ModelsSpatiotemporal Forecasting

  10. Generalizing Multimodal Foundational Models in Context-Driven Class Incremental Learning

    Oct 19, 2026Avi Gupta, Nilotpal Sinha, Vishnu Raj +4Class-Incremental LearningMultimodal Foundation Models

  11. Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

    Oct 19, 2026Zhuofan Josh Ying, Guanglong Sun, Nolan Ho +6Neural EncodingParameter-Efficient Fine-Tuning

  12. OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

    Oct 15, 2026Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong +7Video EditingVideo Diffusion Models

  13. MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

    Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal LearningMultimodal Foundation Models

  14. Toward a Multimodal Foundation Model for Plasma State Reconstruction

    Oct 10, 2026Bin Xia, Chuanfei Dong, Boting Li +9Multimodal Foundation ModelsMissing-Modality Learning

  15. Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding

    Oct 7, 2026Parastoo Azizeddin, Omid Sharafi, Maryam M. ShanechiEEG DecodingMultimodal Foundation Models

  16. SAREO-FM: Decoupled Semantic Supervision for SAR-EO Foundation Models

    Oct 7, 2026Jeonghyeok Do, Munchurl KimRemote Sensing Image UnderstandingMultimodal Pretraining

  17. RoMod: Temporal Routing Modulation via Mixture-of-Experts for Video Anomaly Detection

    Oct 4, 2026Chao Huang, Pengfei Wei, Benfeng Wang +5Video Anomaly DetectionMultimodal Foundation Models

  18. Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition

    Oct 2, 2026Eeshaan Jain, Linus Bleistein, Bart Deplancke +1Efficient Multimodal InferenceMultimodal Foundation Models

  19. ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation Models

    Sep 30, 2026Tobia Poppi, Silvia Cappelletti, Samuele Poppi +4Cross-Modal AlignmentMultimodal Multi-Turn Safety Alignment

  20. Language as the Interface: Foundation-Model Contrastive Learning Links Transcriptomes and Electrophysiology

    Sep 29, 2026Junbo Shen, Jinying Gao, Bo LeiCross-Modal AlignmentMultimodal Foundation Models

  21. AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

    Sep 28, 2026Konstantinos D. Polyzos, Eleni Oikonomou, Tara JavidiMultimodal Large Language ModelsKernel Methods

  22. InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

    Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal PretrainingMultimodal QA

  23. AstraLOD3: Zero-shot multimodal agentic reconstruction of LOD3 building models

    Sep 23, 2026Bryan G. Pantoja-Rosero3D ReconstructionMulti-View 3D Reconstruction

  24. OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalities

    Sep 22, 2026Yizhou Liu, Jinghang Han, Kaixiang Qiu +8Multimodal GroundingPhysical Reasoning