Multimodal Model Evaluation

Ask about this topic

Overview

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

CardsList

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

  1. Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models

    Apr 1, 2026Zixiang Peng, Yongxiu Xu, Qin-Yi Zhang +5Unified Multimodal ModelsLLM Safety Benchmarks

    Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences Beijing, China · Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing, China

  2. OmniSch: A Multimodal PCB Schematic Benchmark For Structured Diagram Visual Reasoning

    Mar 31, 2026Taiting Lu, Kaiyuan Lin, Yuxin Tian +13Spatial Reasoning BenchmarksMultimodal Model Evaluation

    Pennsylvania State University, USA · Independent Researcher · Shanghai Jiao Tong University, China +2

  3. Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

    Mar 31, 2026Fumihiko Tsuchiya, Taiki Miyanishi, Shunsuke Yasuki +5Long-Video QAMultimodal Large Language Models

    The University of Tokyo, Japan · Institute of Science Tokyo, Japan · National Institute of Informatics, Japan

  4. ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation

    Mar 31, 2026Yinuo Liu, Zi Qian, Heng Zhou +7Tool Use in VLMsMultimodal Tool Use

    Qwen Business Unit, Alibaba Group, China · Huazhong University of Science and Technology, China · Zhejiang University, China

  5. Mind the Gap: A Framework for Assessing Pitfalls in Multimodal Active Learning

    Mar 31, 2026Dustin Eisenhardt, Yunhee Jeong, Florian BuettnerActive LearningModality Imbalance

    German Cancer Research Center (DKFZ), Heidelberg, Germany · German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany · Institute of Informatics, Goethe University Frankfurt, Frankfurt am Main, Germany +3

  6. Assessing Multimodal Chronic Wound Embeddings with Expert Triplet Agreement

    Mar 31, 2026Fabian Kabus, Julia Hindel, Jelena Bratulić +6Medical Image RetrievalMultimodal Embedding

    Institute of Medical Biometry and Statistics (IMBI), Medical Faculty and Medical Center, University of Freiburg · Department of Computer Science, Faculty of Engineering, University of Freiburg · Department of Dermatology, Medical Faculty and Medical Center, University of Freiburg

  7. MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

    Mar 30, 2026Fangda Ye, Yuxin Hu, Pengxiang Zhu +19AI Agent BenchmarksMultimodal Agents

  8. CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

    Mar 30, 2026Yongkang Du, Xiaohan Zou, Minhao Cheng +1Analogical ReasoningMultimodal Reasoning

    Pennsylvania State University

  9. SACRED: A Faithful Annotated Multimedia Multimodal Multilingual Dataset for Classifying Connectedness Types in Online Spirituality

    Mar 28, 2026Qinghao Guan, Yuchen Pan, Donghao Li +6Multimodal DatasetMultimodal Classification

    Department of Communication and Media Research, University of Zurich, Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Zurich, Switzerland · Department of Informatics, University of Zurich, Zurich, Switzerland +1

  10. Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

    Mar 28, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan ParvezModality Reliance in Multimodal ModelsMultimodal Model Evaluation

    University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)

  11. RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs

    Mar 27, 2026Logan Lawrence, Mustafa Chasmai, Rangel Daroya +8Fine-Grained Image ClassificationMultimodal Model Evaluation

    Computer Vision Lab, UMass Amherst

  12. MA-Bench: Towards Fine-grained Micro-Action Understanding

    Mar 27, 2026Kun Li, Jihao Gu, Fei Wang +3MLLM EvaluationMultimodal Large Language Models

    CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2

  13. SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

    Mar 27, 2026Cai Selvas-Sala, Lei Kang, Lluis GomezVLM UnlearningMachine Unlearning Evaluation

    Computer Vision Center · Universitat Polit`ecnica de Catalunya · Universitat Aut`onoma de Barcelona

  14. Good Scores, Bad Data: A Metric for Multimodal Coherence

    Mar 26, 2026Vasundra SrinivasanMultimodal Model EvaluationMultimodal Fusion

  15. Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

    Mar 26, 2026Abdullah Hamdi, Changchun Yang, Xin GaoVideo Object SegmentationMedical Image Analysis

    King Abdullah University of Science and Technology (KAUST)

  16. Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

    Mar 26, 2026Ünsal Öztürk, Hatef Otroshi Shahreza, Sébastien MarcelAlgorithmic FairnessMultimodal Large Language Models

    Idiap Research Institute, Switzerland

  17. GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing

    Mar 26, 2026Xuran Hu, Zhitong Xiong, Zhongcheng Hong +3Remote Sensing Image UnderstandingLarge Vision-Language Models

    KTH Royal Institute of Technology · Technical University of Munich · The Hong Kong University of Science and Technology (Guangzhou)

  18. Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math

    Mar 26, 2026Dingjie Song, Tianlong Xu, Yi-Fan Zhang +6Educational AssessmentMultimodal Model Evaluation

    Lehigh University, Bethlehem, PA, USA · Squirrel Ai Learning, Bellevue, WA, USA · Institute of Automation, Chinese Academy of Sciences, Beijing, China +2

  19. NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

    Mar 25, 2026Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska +5Medical VLMsMultimodal Model Evaluation

    Faculty of computer science and engineering, Ss. Cyril and Methodius University, Skopje, North Macedonia.

  20. UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

    Mar 25, 2026Yicheng Xu, Jiangning Zhang, Zhucun Xue +5Multimodal ICLUnified Multimodal Models

    1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1

  21. POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan

    Mar 25, 2026Marta Moscati, Muhammad Saad Saeed, Marina Zanoni +8Missing-Modality LearningSpeech Processing

    Institute of Computational Perception, Johannes Kepler University Linz, Austria · University of Michigan-Flint, USA · Sapienza University of Rome, Italy +5

  22. HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents

    Mar 25, 2026Lexin Wang, Shenghua Liu, Yiwei Wang +5Multimodal Model EvaluationTable QA

    Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · University of Queensland

  23. OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

    Mar 25, 2026Seunghee Kim, Bumkyu Park, Kyudan Jung +5Multimodal GroundingUnified Multimodal Models

    Hanyang University · Seoul National University · KAIST AI +1

  24. I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes

    Mar 24, 2026Shijia Zhou, Saif M. Mohammad, Barbara Plank +1Figurative Language UnderstandingMultimodal Large Language Models

    MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · National Research Council Canada, Ottawa, Canada

  25. From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics

    Mar 24, 2026Paolo Cupini, Francesco PierriLLM-Assisted AnnotationMultimodal Large Language Models

    Politecnico di Milano

  26. ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding

    Mar 24, 2026Ao Cheng, Xingming Li, Xuanyu Ji +5Multimodal Spatial ReasoningMaritime Navigation

    National University of Defense Technology · Intelligent Game and Decision Lab · The Chinese University of Hong Kong, Shenzhen

  27. MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding

    Mar 24, 2026Purui Bai, Tao Wu, Jiayang Sun +3Video UnderstandingVideo-Language Model Evaluation

    MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China · SIST, ShanghaiTech University, Shanghai, China

  28. Think 360°: Evaluating the Width-centric Reasoning Capability of MLLMs Beyond Depth

    Mar 24, 2026Mingrui Chen, Hexiong Yang, Haogeng Liu +2Multimodal ReasoningMultimodal Model Evaluation

    School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR&MAIS, Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +1

  29. CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark

    Mar 23, 2026Junzhao Zhang, Hsiu-Yuan Huang, Chenming Tang +2Sarcasm DetectionMultimodal Model Evaluation

    National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · School of Software and Microelectronics, Peking University

  30. Is AI Ready for Multimodal Hate Speech Detection? A Comprehensive Dataset and Benchmark Evaluation

    Mar 23, 2026Rui Xing, Qi Chai, Jie Ma +5Hate Speech DetectionHarmful Meme Detection

    MOE KLINNS Lab, Xi’an Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou) · School of Cyber Science and Engineering, Xi’an Jiaotong University +1