Multimodal Model Evaluation

Overview

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

CardsList

796 papers39 papers in the last four weeks, down 53% on the four weeks before. 0.3% of all new papers.

  1. Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

    Nov 17, 2026Siddharth Ravi, Anders R. Bargum, Hugo Anselme +5Multimodal Model EvaluationAudio-Visual Understanding

  2. OmniTongue: A Comprehensive MLLM Benchmark for Automated Tongue Diagnosis in Traditional Chinese Medicine

    Nov 17, 2026Liuyi Xu, Yun Guo, Shusheng Li +3Medical VLMsMedical Diagnosis

  3. EEGProbe: Do Vision-Language Models Really Understand EEG? A Benchmark for Signal-Grounded EEG Signals Evaluation

    Nov 16, 2026Haozhe Zhu, Dan Yuan, Chenghao Gu +3VLM EvaluationMultimodal Model Evaluation

  4. RAVEN-Bench: A Paired EO-IR Video QA Benchmark for Aerial Multimodal Understanding

    Nov 13, 2026Yu Hu, Jianyang Gu, Yue Cao +5Video Question AnsweringRemote Sensing Benchmarks

  5. Report of the 8th LSVOS Challenge: Mobile Unified Multimodal Understanding

    Nov 12, 2026Chang Liu, Henghui Ding, Tong Yang +10Unified Multimodal ModelsMultimodal Model Evaluation

  6. TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State

    Nov 9, 2026Xiangyang Wu, Chen Ju, Xu Chen +4Multimodal IRInformation Retrieval

  7. Evaluating classification and explainability in multimodal petrographic image analysis

    Nov 7, 2026Thomas James Wood, Oktay Karakuş, Can EyüpoğluMultimodal ClassificationExplainable Image Classification

  8. From Weapons to Events: Evaluating Evidential Support in Multimodal AI for Policing

    Nov 6, 2026Armin Mustafa, Syed Sameed HusainMultimodal Model EvaluationTemporal Video Understanding

  9. Zero-Shot Benchmarking of Unified Multimodal Models for Shallow-Water UAV Image Distortion Assessment and Restoration

    Nov 6, 2026Dario Scilla, Omar A. Lopez, Mattia Rosso +2Image RestorationUnderwater Image Enhancement

  10. MM-DyGraph: A Dataset, Benchmark, and Model for Multimodal Dynamic Graphs

    Nov 6, 2026Zongyuan Wu, Pengjie Wang, Lingshan Chen +5Multimodal Graph LearningMultimodal Dataset

  11. NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

    Nov 5, 2026Pengkai Wang, Wei-Wei Zhang, Yan Li +13Clinical ReasoningClinical Decision-Making

  12. Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images

    Nov 4, 2026Oishi Banerjee, Eujin Lee, Sung Eun Kim +4Multimodal Model EvaluationMultimodal Reasoning

  13. Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining

    Nov 4, 2026Daichi Hotta, Yohei Kobashi, Takeshi Kojima +2Language Model PretrainingRecurrent Transformers

  14. PhoneValue-Bench: A Multimodal Benchmark and Datasets for Consumer Smartphone Recommerce Valuation

    Oct 31, 2026Zihan Deng, Jun GaoMultimodal DatasetMultimodal Model Evaluation

  15. Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?

    Oct 30, 2026Ziyuan Wang, Genliang Zhu, Xiaobo ChenMultimodal Model EvaluationMultimodal Learning

  16. Do Transaction Prices Reflect Visual Preference? A Diagnostic Study with Multimodal Language Models

    Oct 30, 2026Fengxian Ji, Jingpu Yang, Fan Zhang +6Multimodal Model EvaluationPreference Elicitation

  17. Seeing Is Not Enough: Knowledge-Grounded Active Inquiry from Uncertain Visual Evidence

    Oct 28, 2026Yucheng Liu, Everett X. WangMultimodal ClassificationClinical Decision-Making

  18. Alpha-TCM: Auditable Knowledge Integration for Multimodal TCM Inference

    Oct 28, 2026Yucheng Liu, Everett X. WangMultimodal ReasoningMultimodal Fusion

  19. DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

    Oct 27, 2026Saifullah Saifullah, David Dembinsky, Stefan Agne +2Multimodal Model InterpretabilityExplainability Evaluation

  20. PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy

    Oct 27, 2026Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi +3MLLM EvaluationMultimodal Model Evaluation

  21. Reliability-Aware Short-Term Glycemic Risk Prediction Using an Anchor-Based Comparative Framework of Sequence, Multimodal, and State-Updating Digital-Twin Models

    Oct 26, 2026Pir Bakhsh Khokhar, Carmine Gravino, Fabio PalombaDigital TwinsMultimodal Model Evaluation

  22. RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature

    Oct 23, 2026Jiaxing Sun, Junyuan Gao, Ruikang Zhang +16Multimodal DatasetDocument Information Extraction

  23. The Value of System 1 Models in Medicine

    Oct 22, 2026Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben +17Clinical Language Model EvaluationMultimodal Model Evaluation

  24. PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

    Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI GroundingMultimodal Model Evaluation