Multimodal Model Evaluation

Ask about this topic

Overview

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

CardsList

770 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

  1. Auditable Hierarchical Inference with Multimodal Evidence and Bayesian Active Inquiry

    Oct 28, 2026Yucheng Liu, Everett X. WangMultimodal ClassificationClinical Decision-Making

    Department of Engineering Science, University of Oxford, Oxford, United Kingdom · Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, China

  2. Alpha-TCM: A Nine-Layer Framework for TCM-Guided Multimodal Inference and Structured-Prior Auditing

    Oct 28, 2026Yucheng Liu, Everett X. WangMultimodal ReasoningMultimodal Fusion

    Department of Engineering Science, University of Oxford, Oxford, United Kingdom · Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, Beijing, China

  3. DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

    Oct 27, 2026Saifullah Saifullah, David Dembinsky, Stefan Agne +2Multimodal Model InterpretabilityExplainability Evaluation

    Smart Data and Knowledge Services (SDS), German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, Kaiserslautern, 67663, Germany. · Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Erwin-Schrödinger-Str. 52, Kaiserslautern, 67663, Germany. · DeepReader GmbH, Kaiserslautern, 67663, Germany.

  4. PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy

    Oct 27, 2026Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi +3MLLM EvaluationMultimodal Model Evaluation

    Advanced Micro Devices, Inc. · University of Toronto

  5. Reliability-Aware Short-Term Glycemic Risk Prediction Using an Anchor-Based Comparative Framework of Sequence, Multimodal, and State-Updating Digital-Twin Models

    Oct 26, 2026Pir Bakhsh Khokhar, Carmine Gravino, Fabio PalombaDigital TwinsMultimodal Model Evaluation

    Department of Computer Science, University of Salerno, Fisciano, Italy

  6. Seeing the Options: Typed Decisions over Visual Candidates with Reusable Option Memory

    Oct 24, 2026Fengxin LiEfficient VLM InferenceMultimodal Model Evaluation

    Harbin Institute of Technology, Shenzhen

  7. RxnExBench: Benchmarking Vision-Language Models for Structured Chemical Reaction Extraction from Scientific Literature

    Oct 23, 2026Jiaxing Sun, Junyuan Gao, Ruikang Zhang +16Multimodal DatasetDocument Information Extraction

    Shanghai Artificial Intelligence Laboratory · Peking University · South China Normal University +7

  8. The Value of System 1 Models in Medicine

    Oct 22, 2026Xiufeng Yang, Konstantinos Vrettos, Zeineb Ben Chaaben +17Clinical Language Model EvaluationMultimodal Model Evaluation

    Department of Diagnostic and Interventional Radiology, School of Medicine and Health, Technical University of Munich, Munich, Germany · Artificial Intelligence and Translational Imaging Lab, Department of Radiology, School of Medicine, University of Crete, Heraklion, Greece · Department of Medical Imaging, University Hospital of Heraklion, Heraklion, Crete, Greece +6

  9. PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

    Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI GroundingMultimodal Model Evaluation

    Sapienza University of Rome · Warmwind AG

  10. TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

    Oct 22, 2026Zhaolu Kang, Yidi Wang, Yile Li +4Multimodal Model EvaluationMultimodal QA

    Hunyuan, Tencent · Peking University

  11. Asking a Multimodal LLM Whether Someone Fell: Zero-Shot Screening of Low-Resolution Thermal Video in a Laboratory and an Occupied Home

    Oct 21, 2026Marcos Junca, Mauro JuncaFall DetectionHealthcare

    Senecta Inc.

  12. Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models

    Oct 21, 2026Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili SabetMLLM EvaluationMultimodal Large Language Models

    Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning +2

  13. MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents

    Oct 21, 2026Hong Ting Tsang, Boyi Zhang, Haoyu Huang +7Multimodal MemoryLong-Term Conversational Memory

    The Hong Kong University of Science and Technology · University of Edinburgh

  14. EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

    Oct 20, 2026Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi +17Audio-Visual UnderstandingAudio-Visual QA

    Shanghai Jiao Tong University · Shanghai Innnovation Institute · Inner Mongolia University of Technology +8

  15. DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

    Oct 20, 2026Yuhang Chen, Guangyue Liu, Shilong LiuMultimodal Model EvaluationMultimodal Large Language Models

    Huazhong University of Science and Technology · Princeton University

  16. Testing Associative Creativity in Humans and Multimodal LLMs with Difficult-to-Name Visual Stimuli

    Oct 20, 2026Manar Ali, Lisa Gottschalk, Simeon Junker +6Creativity AssessmentMultimodal Large Language Models

    CRC 1646 ‘Linguistic Creativity in Communication’ Bielefeld University, Bielefeld, Germany

  17. Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum

    Oct 20, 2026Da Huang, Jichao Wang, Junhui Wu +8Robotic ControlLanguage Model-Based Control

  18. RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

    Oct 20, 2026Siyuan Ma, Yu Qi, Cheng Ye +8Vision-Language Model-as-a-JudgeVideo Generation Evaluation

    The Chinese University of Hong Kong · Northeastern University · University of California, San Diego +3

  19. TriGeoBench: A Multimodal Mathematical Geometry Benchmark with Figures and TikZ

    Oct 20, 2026Jing Wang, Yitong Yang, Hao Dou +6Mathematical Reasoning BenchmarksMultimodal Reasoning

    School of Computing and Artificial Intelligence, Shanghai University of Finance and Economics, Shanghai, China · School of Information Management and Engineering, Shanghai University of Finance and Economics, Shanghai, China

  20. CARLAR Influence: A Structured Hallucination Risk in Multimodal Cheque Extraction and an Exploratory Compressed-Image Countermeasure

    Oct 19, 2026Joong Ho, Choi, Aditya Paul +1Document Information ExtractionMultimodal Large Language Models

    Joshua · BNY, Pittsburgh, PA, USA

  21. Small Average Error, Wide Individual Error: General-Purpose LLMs Estimating Energy Intake from Meal Photos in a Feeding Trial

    Oct 19, 2026Xiaomeng LiLLM ReliabilityMultimodal Model Evaluation

    Link Heart Limited, Houston, TX, USA

  22. UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models

    Oct 19, 2026Jianing Wang, Youwen Jian, Derong Wang +5Model CalibrationVLM Calibration

    Fudan University · Institute of Information Engineering, Chinese Academy of Sciences

  23. EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models

    Oct 19, 2026Jiaqi Liu, Ruinan Jin, Yitong Tang +4LLM EvaluationMultimodal Model Evaluation

    The University of British Columbia · Vector Institute · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

  24. GroundedVDR: Grounding Video-to-Web Evidence Dependencies in Video Deep Research

    Oct 18, 2026Xiyuan Liang, Yu Wang, Zuhao Yang +7Multimodal Model EvaluationEvidence-Grounded Generation

    NTU · HKUST(GZ) · ModelBest +2

  25. UBD-Bench: Benchmarking and Analyzing Unimodal Branch Degradation in Multimodal Perception for Autonomous Driving

    Oct 18, 2026Ang Liu, Xiang Gu, Ruikai Li +10Autonomous Driving PerceptionMultimodal Model Evaluation

    Beihang University · Xiaomi EV · University of Macau

  26. Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

    Oct 16, 2026Yi Fang, Gaurav Srivastava, Sirui Ding +1Multimodal FusionMultimodal Model Evaluation

    Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

  27. ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark

    Oct 16, 2026Joel Julin, Souraja Kundu, Liza Dahiya +6Multimodal Model EvaluationMultimodal Grounding

    Carnegie Mellon University · Amazon

  28. VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

    Oct 16, 2026Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza +11Assistive Technology for Blind and Low-Vision PeopleVisually Impaired Navigation Assistance

    Harvard AI and Robotics Lab, Harvard University · Télécom Paris, Institut Polytechnique de Paris · Texas A&M University +2

  29. Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies

    Oct 15, 2026Yiran Huang, Zeyong Zhang, Bharat Biswal +2Multimodal Large Language ModelsMultimodal Model Evaluation

    Department of Data Science, New Jersey Institute of Technology, Newark, NJ 07102, USA · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

  30. A Plausible Explanation Is Enough to Mislead Multimodal Judges

    Oct 15, 2026Anglin Cai, Zhiqi Wang, Xiaozhi Yang +3Bias in LLM-as-a-JudgeVision-Language Model-as-a-Judge

    Northwestern University · The Pennsylvania State University · University of Pennsylvania +1