Multimodal Model Evaluation

Follow this topic Ask about this topic

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing
Mar 12, 2026Mingxin Liu, Ziqian Fan, Zhaokai Wang +13Image Editing Evaluation Text-Guided Image Editing
Shanghai Jiao Tong University · South China University of Technology · Fudan University +2
SciMDR: Advancing Scientific Multimodal Document Reasoning
Mar 12, 2026Ziyu Chen, Yilun Zhao, Chengye Wang +3Multimodal Reasoning Multimodal QA
University of Chicago · Yale University · TCS Research
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Mar 12, 2026Ruirui Chen, Weifeng Jiang, Chengwei Qin +3Moral Reasoning in Language Models Theory of Mind
Institute of High Performance Computing (IHPC) and 2Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Hong Kong University of Science and Technology (Guangzhou), China
Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
Mar 12, 2026Junjie Wu, Xuan Kan, Zihao He +3Vision-Language Model-as-a-Judge Multimodal Model Evaluation
Meta AI · Hong Kong University of Science and Technology · Emory University
MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models
Mar 12, 2026Michiko Yoshitake, Yuta Suzuki, Ryo Igarashi +2Multimodal Model Evaluation Multimodal Large Language Models
OMRON SINIC X, Tokyo, Japan · Department of Applied Physics, Osaka Univ, Suita, Japan · Department of Mechanical Engineering, Univ. Tokyo, Tokyo, Japan
SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG
Mar 10, 2026Fredrik K. Gustafsson, Xiao Gu, Mattia Carletti +3Electrocardiogram Foundation Models Multimodal Model Evaluation
Department of Engineering Science, University of Oxford · NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance · Big Data Institute, Nuffield Department of Population Health, University of Oxford +2
MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
Mar 10, 2026Yunhang Qian, Xiaobin Hu, Jiaquan Yu +6Multi-Agent Systems Multimodal Agents
National University of Singapore · University of Science and Technology of China · Fudan University +3
MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities
Mar 10, 2026Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le +1Multimodal Model Evaluation Missing-Modality Learning
VNU University of Engineering and Technology, Hanoi, Vietnam
EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
Mar 10, 2026Chengjun Yu, Xuhan Zhu, Chaoqun Du +4Multimodal Large Language Models Long-Horizon Reasoning
University of Science and Technology of China · Foundation Model Team, Li Auto Inc. · Tsinghua University
Logics-Parsing-Omni Technical Report
Mar 10, 2026Xin An, Jingyi Cai, Xiangyang Chen +22Multimodal Understanding Semantic Parsing
Alibaba Group
MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent Evaluation Agent Evaluation
Sprinklr AI, Gurugram, India
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1
GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
Mar 9, 2026Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2Multimodal Graph Learning Multimodal Model Evaluation
NYU Shanghai · New York University · East China Normal University +1
A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning
Mar 9, 2026Tianyu Yang, Sihong Wu, Yilun Zhao +6Multimodal Reasoning Multimodal Model Evaluation
University of Notre Dame · Yale University · Columbia University +2
MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals
Mar 9, 2026Junyu Shen, Zhendong She, Chenghanyu Zhang +13Multimodal Large Language Models Multimodal Pretraining
Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications +6
MJ1: Multimodal Judgment via Grounded Verification
Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMs MLLM Evaluation
Haize Labs
BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence
Mar 9, 2026Biao Xiang, Soyeon Caren Han, Yihao DingMulti-Hop QA Multimodal QA
The University of Melbourne Melbourne, Australia · The University of Western Australia Perth, Australia
MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations
Mar 8, 2026Jiyao Liu, Junzhi Ning, Chenglong Ma +14MLLM Evaluation Medical VLMs
Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1
Multimodal Large Language Models as Image Classifiers
Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language Models Multimodal Classification
Visual Recognition Group, Czech Technical University in Prague
When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models
Mar 6, 2026Qitong Wang, Haoran Dai, Haotian Zhang +2Backdoor Attacks Multimodal Diffusion Models
University of Delaware · Illinois Institute of Technology · Columbia University
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
Mar 5, 2026Dannong Xu, Zhongyu Yang, Jun Chen +6Multimodal IR Multimodal Model Evaluation
INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3
UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
Mar 5, 2026Yanlin Li, Minghui Guo, Kaiwen Zhang +13Multimodal Learning Multimodal Generation
NUS · SCUT · NTU +2
SIQA: Toward Reliable Scientific Image Quality Assessment
Mar 5, 2026Wenzhe Li, Liang Chen, Junying Wang +6Image Quality Assessment Multimodal Model Evaluation
TongJi University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary
Mar 5, 2026Alexandru Florea, Shansong Wang, Mingzhe Hu +5Clinical Reasoning Multimodal QA
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine · Department of Biomedical Engineering, Georgia Institute of Technology
Visuospatial Perspective Taking in Multimodal Language Models
Mar 4, 2026Jonathan Prunty, Seraphina Zhang, Patrick Quinn +3Multimodal Large Language Models Visual Spatial Reasoning
Leverhulme Centre for the Future of Intelligence, University of Cambridge · Department of Psychology, University of Cambridge · Microsoft Research Asia
RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review
Mar 4, 2026Zhaoyi Sun, Minal Jagtiani, Wen-wai Yim +4Radiology Report Evaluation Medical Image Benchmarks
University of Washington, Seattle, WA 98195, USA · Microsoft Health AI, Redmond, WA 98052, USA
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?
Mar 3, 2026Zimo Wen, Boxiu Li, Wanbo Zhang +11Unified Multimodal Models Multimodal Model Evaluation
Microsoft Research Asia · Shanghai Jiao Tong University · Fudan University +2
VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning
Mar 3, 2026Ruiyang Zhang, Qianguo Sun, Chao Song +2Multimodal Search Agents Multimodal Tool Use
FST and ICI, University of Macau, China · IDEA
Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language
Mar 3, 2026Michelle Stegeman, Lena Philipp, Fennie van der Graaf +20Computational Pathology Medical Imaging Foundation Models
and on behalf of the UNICORN consortium
An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification
Mar 3, 2026L. Julián Lechuga López, Farah E. Shamout, Tim G. J. RudnerSelective Prediction Uncertainty Calibration
New York University · New York University Abu Dhabi · University of Toronto

Mar 12, 2026J/K move · Enter open · S save

GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing

Mingxin Liu, Ziqian Fan, Zhaokai Wang, Leyao Gu+12

Shanghai Jiao Tong University · South China University of Technology · Fudan University+2

Image Editing Evaluation Text-Guided Image Editing Multimodal Reasoning Image Editing Multimodal Model Evaluation

Open paper AskPDF arXiv

Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natural images and shallow commonsense reasoning, offering limited assessment of this capability under structured, domain-specific constraints. In this work, we introduce GRADE, the first benchmark to assess discipline-informed knowledge and reasoning in image editing. GRADE comprises 520 carefully curated samples across 10 academic domains, spanning from natural science to social science. To support rigorous evaluation, we propose a multi-dimensional evaluation protocol that jointly assesses Discipline Reasoning, Visual Consistency, and Logical Readability. Extensive experiments on 20 state-of-the-art open-source and closed-source models reveal substantial limitations in current models under implicit, knowledge-intensive editing settings, leading to large performance gaps. Beyond quantitative scores, we conduct rigorous analyses and ablations to expose model shortcomings and identify the constraints within disciplinary editing. Together, GRADE pinpoints key directions for the future development of unified multimodal models, advancing the research on discipline-informed image editing and reasoning. Our benchmark and evaluation code are publicly released.