Multimodal Foundation Models

Follow this topic Ask about this topic

Multimodal foundation models study how to train systems that combine inputs such as text, images, audio, video, medical scans, and sensor data, then use them for tasks from question answering to generation and control. Common approaches connect pretrained modality-specific encoders to language models with projectors or adapters, align representations through contrastive learning, or train unified autoregressive and diffusion models to understand and generate multiple modalities. Evaluation spans visual reasoning and retrieval benchmarks, long-video and audio-visual tasks, and real-world settings such as clinical analysis, document understanding, and robotics. A central tension is that strong scores on individual tasks do not ensure grounded, consistent use of every modality, while broader capabilities also raise challenges in data quality, inference cost, robustness, and safety.

Latest papers

354 papers17 papers in the last four weeks, up 31% on the four weeks before. 0.1% of all new papers.

CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation
Jan 12, 2026Chaoyu Li, Fei Tao, Pooyan FazliTest-Time Scaling for VLMs Test-Time Scaling
Arizona State University · NewsBreak
CLIMP: Contrastive Language-Image Mamba Pretraining
Jan 11, 2026Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1Vision-Language Pretraining VLM Robustness
Tel-Aviv University · IBM Research
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8
Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
Dec 16, 2025George-Andrei Dima, Răzvan-Alexandru Smădu, Dumitru-Clementin CercelVLM Adaptation Image Captioning
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania
OpenPhone: Mobile Agentic Foundation Models
Oct 24, 2025Yangqin Jiang, Chao HuangOn-Device Language Model Inference GUI Agents
The University of Hong Kong
From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Oct 14, 2025Weiyang Jin, Yuwei Niu, Jiaqi Liao +4Unified Multimodal Models T2I Generation
HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute +2
HunyuanImage 3.0 Technical Report
Sep 28, 2025Tencent Hunyuan Foundation Model TeamUnified Multimodal Models Autoregressive Image Generation
Tencent Hunyuan Foundation Model Team
LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
Sep 28, 2025Shubhang Bhatnagar, Andy Xu, Kar-Han Tan +1Mixed-Precision Quantization LLM Quantization
University of Illinois Urbana-Champaign · Work done as an intern at HP Inc. · University of California, Los Angeles +1
Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
Sep 23, 2025Shufan Li, Jiuxiang Gu, Kangning Liu +4Masked Diffusion Models Unified Multimodal Models
Adobe · UCLA
Reconstruction Alignment Improves Unified Multimodal Models
Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal Models Cross-Modal Alignment
1UC Berkeley · University of Washington · 3Duke University
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM Evaluation Multimodal Reasoning
KAIST · MBZUAI
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
Aug 17, 2025Xuhui Zhan, Tyler DerrCross-Modal Learning Cross-Modal Representation Learning
Data Science Institute, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States. · Computer Science Department, Vanderbilt University, 2201 West End Ave, Nashville, 37235, Tennessee, United States.
Position: Reasoning After Perception Means Reasoning Without Vision
Jul 21, 2025Hongcheng Gao, Zihao Huang, Jingyi Tang +12Multimodal Understanding Visual Reasoning
Tsinghua University · BIT · Peking University +6
Can Argus Judge Them All? Comparing VLMs Across Domains
Jun 23, 2025Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali +5VLM Evaluation VLM Robustness
Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India +3
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
May 22, 2025Runpeng Yu, Xinyin Ma, Xinchao WangMultimodal Large Language Models LLM Inference Acceleration
National University of Singapore
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal Understanding Vision-Language Models
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic Fairness Multimodal Large Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA
ChronoSteer: Bridging Large Language Model and Time Series Foundation Model via Synthetic Cross-Modal Alignment Dataset
May 15, 2025Chengsen Wang, Qi Qi, Zhongwen Rao +2Cross-Modal Learning Multivariate Time Series Forecasting
Beijing University of Posts and Telecommunications · Huawei Noah’s Ark Lab
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded Reasoning VLM Reasoning
University of Cambridge
Histopathology Multi-modal Embedding for Pathology Composed Retrieval
Feb 11, 2025Qifeng Zhou, Wenliang Zhong, Thao M. Dang +4Computational Pathology Healthcare
The University of Texas at Arlington, Arlington TX 76010, USA
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Healthcare Medical Image Analysis
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation

Jan 12, 2026J/K move · Enter open · S save

CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation

Chaoyu Li, Fei Tao, Pooyan Fazli

Arizona State University · NewsBreak

Test-Time Scaling for VLMs Test-Time Scaling Video Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Foundation Models+2

Open paper AskPDF arXiv

Vision-language models achieve strong performance across a wide range of multimodal understanding and reasoning tasks, yet their multi-step reasoning remains unstable. Repeated sampling over the same input often produces divergent reasoning trajectories and inconsistent final predictions. To address this, we introduce two complementary approaches inspired by test-time scaling: (1) CASHEW, an inference-time framework that stabilizes reasoning by iteratively aggregating multiple candidate trajectories into higher-quality reasoning traces, with visual verification of object presence providing grounding cues during aggregation, and (2) CASHEW-RL, a learned variant that internalizes this aggregation behavior within a single model. CASHEW-RL is trained using Group Sequence Policy Optimization (GSPO) with a composite reward that encourages correct answers grounded in minimal yet sufficient visual evidence, while adaptively allocating reasoning effort based on task difficulty. This training objective enables robust self-aggregation at inference. Extensive experiments on 13 image understanding, video understanding, and video reasoning benchmarks show significant performance improvements, with CASHEW/CASHEW-RL achieving average gains of +4.1/+5.3 percentage points on Qwen3-VL-4B and +2.6/+3.0 percentage points on Qwen3-VL-8B, respectively.