Multimodal Model Evaluation

Follow this topic Ask about this topic

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
Mar 3, 2026Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5MLLM Evaluation Multimodal Model Evaluation
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
Efficient and High-Fidelity Omni Modality Retrieval
Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IR Cross-Modal Retrieval
University of Maryland, College Park, USA · Monash University, Australia
MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
Mar 2, 2026Jiachun Li, Shaoping Huang, Zhuoran Jin +5Multi-Image Reasoning Multimodal Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences
Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring
Mar 2, 2026Aditya Shukla, Yining Yuan, Ben Tamo +7Summarization Evaluation Clinical Language Model Evaluation
Georgia Institute of Technology · Shepherd Center
Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs
Mar 2, 2026Hengwei Ye, Yuanting Guan, Yuxuan Ge +7Multimodal Model Evaluation Multimodal Large Language Models
ShanghaiTech University
PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval
Mar 2, 2026Tianyi Xu, Rong Shan, Junjie Wu +11Multimodal IR Image Retrieval
Shanghai Jiao Tong University, Shanghai, China · OPPO, Shenzhen, China
Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
Mar 1, 2026Xuan Lu, Kangle Li, Haohang Huang +3Multimodal IR Cross-Modal Retrieval
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative +1
MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
Mar 1, 2026Xuying Ning, Dongqi Fu, Tianxin Wei +7Multimodal RAG Multimodal Search Agents
University of Illinois Urbana-Champaign · Meta · IBM Research +1
UNICBench: UNIfied Counting Benchmark for MLLM
Feb 28, 2026Chenggang Rong, Tao Han, Zhiyuan Zhao +5Multimodal Large Language Models Multimodal Dataset
Northwestern Polytechnical University · Hong Kong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom +2
Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs
Feb 28, 2026Yiran Zhao, Lu Zhou, Xiaogang Xu +3Algorithmic Fairness Demographic Bias in Language Models
Nanjing University of Aeronautics and Astronautics · Collaborative Innovation Center of Novel Software Technology and Industrialization · The Chinese University of Hong Kong +2
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks
Feb 28, 2026Hengjian Gao, Kaiwei Zhang, Shibo Wang +8AI Agent Benchmarks Multimodal Model Evaluation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Shanghai University of Electric Power +1
CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
Feb 28, 2026Yannian Gu, Zhongzhen Huang, Linjie Mu +3Clinical Language Model Evaluation Multimodal Model Evaluation
Shanghai Jiao Tong University, Shanghai, China · SenseTime Research, China · Shanghai Innovation Institute, Shanghai, China
How Well Do Multimodal Models Reason on ECG Signals?
Feb 27, 2026Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu +11Electrocardiogram Interpretation Multimodal Model Evaluation
University of Illinois Urbana Champaign · Google Inc · Rush University +4
Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Feb 27, 2026Gregory Kang Ruey Lau, Hieu Dao, Nicole Kan Hui Lin +1LLM Uncertainty Estimation Multimodal Large Language Models
Department of Computer Science, National University of Singapore
Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?
Feb 27, 2026Hongbo Jiang, Jie Li, Yunhang Shen +4Unified Multimodal Models VLM Reasoning
Tencent Youtu Lab, Shanghai, China · Xiamen University, Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen, China.
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Feb 27, 2026Kejing Yin, Haizhou Xu, Wenfang Yao +5Multimodal Fusion Multimodal Model Evaluation
Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
Feb 26, 2026Peiyao Xiao, Xiaogang Li, Xinyi Gao +7LLM Evaluation Multimodal Model Evaluation
Alibaba Group Beijing, China
Same Words, Different Judgments: How Preferences Vary Across Modalities
Feb 26, 2026Aaron Broukhim, Nadir Weibel, Eshin JollyHuman Preference Evaluation Inter-Rater Reliability
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
Feb 25, 2026Boqi Chen, Xudong Liu, Jiachuan Peng +5Medical Diagnosis Medical VLMs
ETH Zurich · MBZUAI · Amazon +4
EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs
Feb 25, 2026Wenjie Tian, Zhixian Zhao, Jingbin Hu +4Multimodal Large Language Models Emotion-Aware Dialogue Generation
Northwestern Polytechnical University, Xi’an, China · University of Surrey, Guildford, United Kingdom
FewMMBench: A Benchmark for Multimodal Few-Shot Learning
Feb 25, 2026Mustafa Dogan, Ilker Kesen, Iacer Calixto +2Multimodal Model Evaluation Multimodal Large Language Models
Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam +4
WildSVG: Towards Reliable SVG Generation Under Real-Word Conditions
Feb 24, 2026Marco Terral, Haotian Zhang, Tianyang Zhang +8Vector Graphics Generation Multimodal Model Evaluation
QuiverAI · Columbia University · Illinois Institute of Technology +3
Predicting Sentence Acceptability Judgments in Multimodal Contexts
Feb 24, 2026Hyewon Jang, Nikolai Ilinykh, Sharid Loáiciga +2LLM Evaluation Multimodal Model Evaluation
University of Gothenburg · University of Melbourne · Queen Mary University of London +1
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Feb 24, 2026Zhenyu Wang, Xiaofen Xing, Yirong Chen +1Role-Playing Agent Evaluation Multimodal Agent Evaluation
South China University of Technology, Guangzhou 510641, China · Foshan University, Foshan 528225, China
Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition
Feb 22, 2026Xia Hu, Honglei Zhuang, Brian Potetz +4Multimodal Reasoning Multimodal QA
Google DeepMind
MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs
Feb 20, 2026Ziqiao Shang, Ling-Yue Ge, Zian Xu +10Multimodal Reasoning Multimodal Model Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AMAP, Alibaba Group +1
Rodent-Bench
Feb 20, 2026Thomas Heap, Laurence Aitchison, Emma Cahill +1Animal Behavior Analysis Video-Language Model Evaluation
University of Bristol, Bristol, UK
MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions
Feb 19, 2026Rebecca Salganik, Teng Tu, Fei-Yueh Chen +8Music Information Retrieval Multimodal Model Evaluation
University of Rochester, NY, USA · National University of Singapore, Singapore · SJTU Paris Elite Institute of Technology, Shanghai, China +2
Projective Psychological Assessment of Large Multimodal Models Using Thematic Apperception Tests
Feb 19, 2026Anton Dzega, Aviad Elyashar, Ortal Slobodin +2Personality Modeling in Language Models Multimodal Model Evaluation
Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel · Department of Computer Science, Shamoon College of Engineering, Israel · School of Education, Ben-Gurion University of the Negev, Israel +1

Mar 3, 2026J/K move · Enter open · S save

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao+4

Kyoto University · CyberAgent. · The Institute of Statistical Mathematics+1

MLLM Evaluation Multimodal Model Evaluation Multimodal Reasoning Item Response Theory

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.