Multimodal Foundation Models

Multimodal foundation models study how to train systems that combine inputs such as text, images, audio, video, medical scans, and sensor data, then use them for tasks from question answering to generation and control. Common approaches connect pretrained modality-specific encoders to language models with projectors or adapters, align representations through contrastive learning, or train unified autoregressive and diffusion models to understand and generate multiple modalities. Evaluation spans visual reasoning and retrieval benchmarks, long-video and audio-visual tasks, and real-world settings such as clinical analysis, document understanding, and robotics. A central tension is that strong scores on individual tasks do not ensure grounded, consistent use of every modality, while broader capabilities also raise challenges in data quality, inference cost, robustness, and safety.