Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models
Jul 24, 2026Bannapol Limanond, Masanori Suganuma, Takayuki OkataniMedical VLMs Multimodal Model Evaluation
Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning
Jul 23, 2026Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini +2VLM Unlearning Algorithmic Fairness
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy
Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
Jul 22, 2026Qiwei Ma, Chunping Qiu, Xinjun Cheng +5Remote Sensing Image Understanding Remote Sensing VQA
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China · Intelligent Game and Decision Lab (IGDL), Beijing, 100091, China · cYuelushan Center for Industrial Innovation, Changsha, 410082, China
MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction
Jul 22, 2026Yue Zhao, Hongxu Liu, Feiyu Wang +5Data Visualization Code Generation
Shandong Second Medical University · Shandong University · Bairong Inc. +2
Silent Failures in Multimodal Agentic Search:A Diagnostic Taxonomy and Cross-Judge Evaluation
Jul 22, 2026Zhengxian Wu, Junjie Gao, Kai YangMultimodal Grounding Agentic Search
Ant Group · Hangzhou, China
Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended Generation LLM-as-a-Judge
Meta AI
MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings
Jul 21, 2026Ziyi Wang, Yuhang Wu, Dongxu Piao +3Theory of Mind Multimodal Large Language Models
College of AI, Tsinghua University · Department of Biostatistics and Bioinformatics, Duke University
Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
Jul 21, 2026Tuo Liang, Zhe Hu, Disheng Liu +2Multimodal Large Language Models Computational Creativity
Case Western Reserve University · The Hong Kong Polytechnic University
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference Evaluation Pairwise Preference Evaluation
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences Beijing, China
Now We Know? A Systematic Comparison of TerraMind and THOR
Jul 20, 2026Frederick Schindlegger, Kenzo Bounegta, Eva Gmelich Meijling +5Geospatial Foundation Models Remote Sensing
University of Münster, Münster, Germany · 2ESA Φ-lab, ESRIN, Frascati, Italy · 3Rubicon, Paris, France +2
How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks
Jul 20, 2026Tahiya ChowdhurySpoken Dialogue Systems Multimodal Model Evaluation
Colby College Waterville, Maine, United States
Can Multimodal Large Language Models Understand OCT?
Jul 18, 2026Baochen Fu, Wenzhi Deng, Baihao Jin +5Medical Imaging Medical Image Analysis
1Shandong University · 4Shandong Provincial Qianfoshan Hospital · University of Science and Technology of China +2
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Jul 17, 2026Like Liu, Zhengzheng Xu, Haitao He +3Multimodal Grounding Aerial Robotics
Northwestern Polytechnical University Xi’an, Shaanxi, China
SLAPBench: Benchmarking Multimodal Large Language Models for Four-Finger SLAP Fingerprint Verification
Jul 17, 2026Bibesh Pyakurel, M. G. Sarwar MurshedBiometric Identification Multimodal Large Language Models
Computer Science Department, University of Wisconsin–Green Bay, Green Bay, WI, USA
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Jul 16, 2026Shaoxiong Zhan, Shi Hu, Boyu Feng +7Multimodal Model Evaluation
Tsinghua University
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Jul 16, 2026Patrick Phuoc Do, Chau M. Ta, Chaoli WangScientific Visualization Data Visualization
University of Notre Dame
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Jul 16, 2026Yijiang Li, Huiqi Zou, Bingyang Wang +1VLM Evaluation VLM Hallucination
UC San Diego · Northeastern University · Georgia Institute of Technology +1
AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning
Jul 14, 2026Yanghai Wang, Jiahao Wang, Jiafu Tang +9Audio-Visual Understanding Audio-Visual Alignment
NJU-LINK Team, Nanjing University · Kling Team, Kuaishou Technology · Institute of Automation, Chinese Academy of Sciences
Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding
Jul 13, 2026Kerui Chen, Jinglu Wang, Xiaoyi Zhang +1Active Perception Sports Video Analysis
Zhejiang University · Microsoft Research Asia
GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
Jul 13, 2026Suhaas Garre, Emily Ritchie, Sushant Mehta +1Multimodal Grounding Multimodal Reasoning
Surge AI
Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video
Jul 13, 2026Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar +1Visual Object Tracking Video-Language Model Evaluation
Capabilities of Claude Fable 5 on Biomedical Challenge Problems
Jul 12, 2026Dominic Okonkwo, Magnus Hodgson, Temitope I. David +1Biomedical QA LLM Refusal Behavior
School of Computing, University of Georgia · Department of Chemistry, University of Illinois · Institute of Bioinformatics, University of Georgia
Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing
Jul 10, 2026Mohammad Dabaja, Turgay CelikRemote Sensing Image Understanding Remote Sensing Small Object Detection
Department of ICT, University of Agder, Grimstad, Norway
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Jul 10, 2026Runhan Shi, Quan Zhou, Yuqian Xu +14Healthcare Clinical Language Model Evaluation
JD Health International Inc. · Shanghai Jiao Tong University · National University of Singapore +3
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Jul 9, 2026Matteo Santelmo, Xiuying Wei, Israa Fakih +5VLM Evaluation LLM Evaluation
Ecole Polytechnique Fédérale de Lausanne (EPFL), Switzerland
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
Jul 9, 2026Haibin Tian, Huichao Xie, Xuelin Qian +3Multimodal Grounding Multimodal Model Evaluation
School of Automation, Northwestern Polytechnical University, Xi’an 710072, China · College of Missile Engineering, Rocket Force University of Engineering, Xi’an 710038, China · School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
Rethinking Multimodal Time-Series Forecasting Evaluation
Jul 8, 2026Haoxin Liu, Yichen Zhou, Rajat Sen +2Multivariate Time Series Forecasting Zero-Shot Time Series Forecasting
Georgia Institute of Technology · Google Research
HoloCount: A Holistic Visual Counting Benchmark for MLLMs
Jul 7, 2026Jinhong Deng, Limeng Qiao, Guanglu WanMultimodal Robustness Object Counting
Meituan
Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation
Jul 7, 2026Shenxi Liu, Kan Li, Mingyang Zhao +2Multimodal IR Temporal Video Grounding
School of Computer Science and Technology, Beijing Institute of Technology · Department of Computing, The Hong Kong Polytechnic University · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music
Jul 7, 2026Tomáš Sourada, Katia Vendrame, Jan HajičAudio-Language Model Evaluation Synthetic Benchmark Generation
Charles University · Brno University of Technology

Jul 24, 2026, cs.CVJ/K move · Enter open · S save

Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models

Bannapol Limanond, Masanori Suganuma, Takayuki Okatani

Medical VLMs Multimodal Model Evaluation Medical Image Benchmarks

Open paper AskPDF arXiv

This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models have demonstrated significant potential in the field of medical vision-language tasks. However, it is becoming increasingly clear that evaluating these models' performance, whether they are applied to natural or medical images, is challenging. The critical question is whether the models can accurately understand an input image while associating it with relevant input text. To address this, Medical-Checklist imposes a binary test on the models: they are given an image and two captions, where one is correct and the other incorrect, and the model must select the correct one. The incorrect caption contains a single medical concept (word or phrase) that is inaccurately substituted from the correct caption. Although the task is simple, this simplicity enables the unified assessment of diverse multimodal models designed and learned on different principles. It also enables us to verify whether models correctly understand a wide range of medical concepts across various medical sub-domains. Medical-Checklist is designed to reduce potential biases in data and to enable evaluation of the models' ability to handle out-of-distribution inputs, which were difficult in existing datasets. When evaluating four state-of-the-art medical multimodal models with Medical-Checklist, it was revealed that despite their excellent performance in specific tasks such as Med-VQA, they may not correctly understand images, suggesting a long journey ahead for clinical application. The dataset and code will be made public upon acceptance.

Momentum

Related topics

Latest papers 496