Multimodal Model Evaluation

Ask about this topic

Overview

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

CardsList

770 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

  1. CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs

    Mar 12, 2026Ruirui Chen, Weifeng Jiang, Chengwei Qin +3Moral Reasoning in Language ModelsTheory of Mind

    Institute of High Performance Computing (IHPC) and 2Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Hong Kong University of Science and Technology (Guangzhou), China

  2. Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

    Mar 12, 2026Junjie Wu, Xuan Kan, Zihao He +3Vision-Language Model-as-a-JudgeMultimodal Model Evaluation

    Meta AI · Hong Kong University of Science and Technology · Emory University

  3. MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models

    Mar 12, 2026Michiko Yoshitake, Yuta Suzuki, Ryo Igarashi +2Multimodal Model EvaluationMultimodal Large Language Models

    OMRON SINIC X, Tokyo, Japan · Department of Applied Physics, Osaka Univ, Suita, Japan · Department of Mechanical Engineering, Univ. Tokyo, Tokyo, Japan

  4. SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG

    Mar 10, 2026Fredrik K. Gustafsson, Xiao Gu, Mattia Carletti +3Electrocardiogram Foundation ModelsMultimodal Model Evaluation

    Department of Engineering Science, University of Oxford · NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance · Big Data Institute, Nuffield Department of Population Health, University of Oxford +2

  5. MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems

    Mar 10, 2026Yunhang Qian, Xiaobin Hu, Jiaquan Yu +6Multi-Agent SystemsMultimodal Agents

    National University of Singapore · University of Science and Technology of China · Fudan University +3

  6. MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

    Mar 10, 2026Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le +1Multimodal Model EvaluationMissing-Modality Learning

    VNU University of Engineering and Technology, Hanoi, Vietnam

  7. EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

    Mar 10, 2026Chengjun Yu, Xuhan Zhu, Chaoqun Du +4Multimodal Large Language ModelsLong-Horizon Reasoning

    University of Science and Technology of China · Foundation Model Team, Li Auto Inc. · Tsinghua University

  8. Logics-Parsing-Omni Technical Report

    Mar 10, 2026Xin An, Jingyi Cai, Xiangyang Chen +22Multimodal UnderstandingSemantic Parsing

    Alibaba Group

  9. MM-tau-p2^2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

    Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent EvaluationAgent Evaluation

    Sprinklr AI, Gurugram, India

  10. Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

    Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language ModelsModality Gap in VLMs

    Johns Hopkins University · Amazon · New York University +1

  11. GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

    Mar 9, 2026Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2Multimodal Graph LearningMultimodal Model Evaluation

    NYU Shanghai · New York University · East China Normal University +1

  12. A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning

    Mar 9, 2026Tianyu Yang, Sihong Wu, Yilun Zhao +6Multimodal ReasoningMultimodal Model Evaluation

    University of Notre Dame · Yale University · Columbia University +2

  13. MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

    Mar 9, 2026Junyu Shen, Zhendong She, Chenghanyu Zhang +13Multimodal Large Language ModelsMultimodal Pretraining

    Tsinghua University · Tianjin University · Beijing University of Posts and Telecommunications +6

  14. MJ1: Multimodal Judgment via Grounded Verification

    Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMsMLLM Evaluation

    Haize Labs

  15. BRIDGE: Benchmark for multi-hop Reasoning In long multimodal Documents with Grounded Evidence

    Mar 9, 2026Biao Xiang, Soyeon Caren Han, Yihao DingMulti-Hop QAMultimodal QA

    The University of Melbourne Melbourne, Australia · The University of Western Australia Perth, Australia

  16. MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

    Mar 8, 2026Jiyao Liu, Junzhi Ning, Chenglong Ma +14MLLM EvaluationMedical VLMs

    Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1

  17. Multimodal Large Language Models as Image Classifiers

    Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language ModelsMultimodal Classification

    Visual Recognition Group, Czech Technical University in Prague

  18. When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models

    Mar 6, 2026Qitong Wang, Haoran Dai, Haotian Zhang +2Backdoor AttacksMultimodal Diffusion Models

    University of Delaware · Illinois Institute of Technology · Columbia University

  19. MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

    Mar 5, 2026Dannong Xu, Zhongyu Yang, Jun Chen +6Multimodal IRMultimodal Model Evaluation

    INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3

  20. UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark

    Mar 5, 2026Yanlin Li, Minghui Guo, Kaiwen Zhang +13Multimodal LearningMultimodal Generation

    NUS · SCUT · NTU +2

  21. SIQA: Toward Reliable Scientific Image Quality Assessment

    Mar 5, 2026Wenzhe Li, Liang Chen, Junying Wang +6Image Quality AssessmentMultimodal Model Evaluation

    TongJi University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

  22. Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary

    Mar 5, 2026Alexandru Florea, Shansong Wang, Mingzhe Hu +5Clinical ReasoningMultimodal QA

    Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine · Department of Biomedical Engineering, Georgia Institute of Technology

  23. Visuospatial Perspective Taking in Multimodal Language Models

    Mar 4, 2026Jonathan Prunty, Seraphina Zhang, Patrick Quinn +3Multimodal Large Language ModelsVisual Spatial Reasoning

    Leverhulme Centre for the Future of Intelligence, University of Cambridge · Department of Psychology, University of Cambridge · Microsoft Research Asia

  24. RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review

    Mar 4, 2026Zhaoyi Sun, Minal Jagtiani, Wen-wai Yim +4Radiology Report EvaluationMedical Image Benchmarks

    University of Washington, Seattle, WA 98195, USA · Microsoft Health AI, Redmond, WA 98052, USA

  25. UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?

    Mar 3, 2026Zimo Wen, Boxiu Li, Wanbo Zhang +11Unified Multimodal ModelsMultimodal Model Evaluation

    Microsoft Research Asia · Shanghai Jiao Tong University · Fudan University +2

  26. VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning

    Mar 3, 2026Ruiyang Zhang, Qianguo Sun, Chao Song +2Multimodal Search AgentsMultimodal Tool Use

    FST and ICI, University of Macau, China · IDEA

  27. Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language

    Mar 3, 2026Michelle Stegeman, Lena Philipp, Fennie van der Graaf +20Computational PathologyMedical Imaging Foundation Models

    and on behalf of the UNICORN consortium

  28. An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification

    Mar 3, 2026L. Julián Lechuga López, Farah E. Shamout, Tim G. J. RudnerSelective PredictionUncertainty Calibration

    New York University · New York University Abu Dhabi · University of Toronto

  29. Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

    Mar 3, 2026Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5MLLM EvaluationMultimodal Model Evaluation

    Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1

  30. Efficient and High-Fidelity Omni Modality Retrieval

    Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IRCross-Modal Retrieval

    University of Maryland, College Park, USA · Monash University, Australia