Multimodal Foundation Models

Overview

Multimodal foundation models study how to train systems that combine inputs such as text, images, audio, video, medical scans, and sensor data, then use them for tasks from question answering to generation and control. Common approaches connect pretrained modality-specific encoders to language models with projectors or adapters, align representations through contrastive learning, or train unified autoregressive and diffusion models to understand and generate multiple modalities. Evaluation spans visual reasoning and retrieval benchmarks, long-video and audio-visual tasks, and real-world settings such as clinical analysis, document understanding, and robotics. A central tension is that strong scores on individual tasks do not ensure grounded, consistent use of every modality, while broader capabilities also raise challenges in data quality, inference cost, robustness, and safety.

Latest papers

CardsList

365 papers11 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

    Apr 22, 2025Wei Zhuo, Zhiyue Tang, Wufeng Xue +3Few-Shot Semantic SegmentationMultimodal Foundation Models

  2. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

    Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded ReasoningVLM Reasoning

  3. Histopathology Multi-modal Embedding for Pathology Composed Retrieval

    Feb 11, 2025Qifeng Zhou, Wenliang Zhong, Thao M. Dang +4Computational PathologyHealthcare

  4. MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

    Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5HealthcareMedical Image Analysis