Multimodal Model Evaluation

Follow this topic Ask about this topic

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

MirrorBench: Evaluating Self-centric Intelligence in MLLMs by Introducing a Mirror
Apr 16, 2026Shengyu Guo, Tongrui Ye, Jianbo Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai AI Lab
Rethinking Patient Education as Multi-turn Multi-modal Interaction
Apr 16, 2026Zonghai Yao, Zhipeng Tang, Chengtao Lin +5Multimodal Grounding Healthcare
1VA Bedford Health Care · 2UMass Amherst · 4Yale University +3
The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models
Apr 15, 2026Akshay Paruchuri, Ishan Chatterjee, Henry Fuchs +2Modality Reliance in Multimodal Models Multimodal Large Language Models
Università della Svizzera italiana · Stanford University · University of Washington +1
MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging
Apr 15, 2026Zhijie Bao, Fangke Chen, Licheng Bao +4Multimodal Model Evaluation Medical VLMs
School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China +2
From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms
Apr 14, 2026Nicholas Pather, Joshua Fouché, Sitwala Mundia +5Document Information Extraction Document Image Understanding
CSAM, University of the Witwatersrand & Grai Labs, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa · Faculty of Health Sciences, University of the Witwatersrand, South Africa +5
MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games
Apr 14, 2026Shufang Lin, Muyang Chen, Xiabing Zhou +3Multimodal Intent Recognition Multimodal Large Language Models
The Chinese University of Hong Kong, Shenzhen Shenzhen, GuangDong, China · Capital Normal University Beijing, China
Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs
Apr 14, 2026Jianhao Chen, Shiqin Wang, Haoyang Chen +4Adversarial Attacks on VLMs VLM Evaluation
School of Computer Science, Wuhan University, Wuhan, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China +1
MLLM-as-a-Judge Exhibits Model Preference Bias
Apr 13, 2026Shuitsu Koyama, Yuiga Wada, Daichi Yashima +1Bias in LLM-as-a-Judge Multimodal Model Evaluation
Keio University, Japan
PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers
Apr 13, 2026Lei Xiong, Huaying Yuan, Zheng Liu +2AI Agent Benchmarks Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China
Enhancing Multimodal Large Language Models for Ancient Chinese Character Evolution Analysis via Glyph-Driven Fine-Tuning
Apr 13, 2026Rui Song, Lida Shi, Ruihua Qi +2Multimodal Large Language Models LLM Fine-Tuning
College of Computer Science and Technology, Jilin University, China · Key Laboratory of Ancient Chinese Script, Culture Relics and Artificial Intelligence, Jilin University, China · School of Artificial Intelligence, Jilin University, China +1
TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training
Apr 12, 2026Yinyi Luo, Wenwen Wang, Hayes Bai +6Unified Multimodal Models Multimodal Model Evaluation
Carnegie Mellon University · William & Mary · Auburn University +1
MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark
Apr 12, 2026Junzhi Ning, Jiashi Lin, Yingying Fang +9Medical Image Benchmarks Multimodal Model Evaluation
Shanghai AI Laboratory · Imperial College London · Shanghai Jiao Tong University +1
Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts
Apr 11, 2026Ruijia Li, Mingzi Zhang, Zengyi Yu +2Cultural Bias in VLMs VLM Evaluation
School of Computer Science and Technology, East China Normal University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · Faculty of Education, East China Normal University
Demographic and Linguistic Bias Evaluation in Omnimodal Language Models
Apr 11, 2026Alaa ElobaidMultimodal Model Evaluation Unified Multimodal Models
Freie Universität Berlin, Berlin, Germany
From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
Apr 10, 2026Yu Wu, Guangzeng Han, Ibra Niang Niang +6Plant Phenotyping VLM Evaluation
Computer Science, University of Memphis, Memphis, 38111, TN, United States · Fisher Delta Research Extension and Education Center, University of Missouri, Portageville, 63873, MO, United States · University of Arkansas Division of Agriculture, Batesville, 72501, AR, United States +1
SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
Apr 10, 2026Xiyang Huang, Jiawei Lin, Keying Wu +9Video-Language Model Evaluation Multimodal Model Evaluation
School of Artificial Intelligence, Wuhan University · Center for Language and Information Research, Wuhan University · Southwest Jiaotong University +3
Unified Multimodal Uncertain Inference
Apr 9, 2026Dengjia Zhang, Alexander Martin, William Jurayj +3Probability Calibration Multimodal Reasoning
Johns Hopkins University · Human Language Technology Center of Excellence
Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
Apr 9, 2026Xun Zhu, Fanbin Mo, Xi Chen +6Multimodal Large Language Models Medical LLMs
Department of Electronic Engineering Tsinghua University Beijing, China · School of Artificial Intelligence BUPT Beijing, China · College of AI Tsinghua University Beijing, China +1
FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios
Apr 8, 2026Xiangru Jian, Hao Xu, Wei Pang +13Multimodal Model Evaluation Multimodal Learning
University of Waterloo, Canada · University of Sydney, Australia · Singapore Management University, Singapore +7
What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
Apr 8, 2026Songze Li, Xiaoke Guo, Tianqi Liu +5GUI Grounding Multimodal Reasoning
Zhejiang University · ZJU-Ant Group Joint Lab of Knowledge Graph
"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?
Apr 7, 2026Naen Xu, Jiayi Sheng, Changjiang Li +7VLM Evaluation Multimodal Understanding
Zhejiang University · Beihang University · Palo Alto Networks +4
Simulating Validity: Modal Decoupling in MLLM Generated Feedback on Science Drawings
Apr 5, 2026Arne Bewersdorff, Nejla Yuruk, Xiaoming ZhaiScience Education Multimodal Grounding
University of Georgia, AI4STEM Education Center, Athens GA 30602, USA · Gazi University, Department of Mathematics and Science Education, Ankara, 06560, Turkiye
FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
Apr 4, 2026Zeyu Wang, Jingye Xu, Xiaogang Li +7Scientific Reasoning in Language Models Multimodal Reasoning
Alibaba Group, Beijing, China · Skylenage, Beijing, China
ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
Apr 4, 2026Zitong Xu, Huiyu Duan, Shengyao Qin +6Image Captioning Evaluation Image Captioning
Shanghai Jiao Tong University · University of Electronic and Science Technology of China · Beijing University of Technology +1
Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
Apr 3, 2026Gengwei Zhang, Jie Peng, Zhen Tan +6RL for VLMs Multimodal Reasoning
University of North Carolina at Chapel Hill · University of Science and Technology of China · Arizona State University +1
PolyReal: A Benchmark for Real-World Polymer Science Workflows
Apr 3, 2026Wanhao Liu, Weida Wang, Jiaqing Xie +12Multimodal Model Evaluation Multimodal Large Language Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University +5
OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Apr 3, 2026Felix Henry, Lihan Lin, Jiangyou Zhu +4Mobile GUI Automation GUI Agents
XPeng Motors
CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning
Apr 2, 2026Junyoung Sung, Seungwoo Lyu, Minjun Kim +3Multimodal Reasoning VLM Reasoning
Dept. of CSE, Korea University · Google DeepMind
M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency
Apr 1, 2026Abolfazl Ansari, Delvin Ce Zhang, Zhuoyang Zou +2Claim Verification Factual Consistency Evaluation
The Pennsylvania State University, USA · University of Sheffield, UK
Multimodal Language Models Cannot Spot Spatial Inconsistencies
Apr 1, 2026Om Khangaonkar, Hadi J. Rad, Hamed PirsiavashMulti-View Consistency 3D Spatial Reasoning
University of California, Davis · Shell · TU Delft

Apr 16, 2026J/K move · Enter open · S save

MirrorBench: Evaluating Self-centric Intelligence in MLLMs by Introducing a Mirror

Shengyu Guo, Tongrui Ye, Jianbo Zhang, Zicheng Zhang+2

Shanghai AI Lab

Multimodal Large Language Models AI Agent Benchmarks Embodied AI Multimodal Model Evaluation

Open paper AskPDF arXiv

Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated remarkable advances in perception and reasoning, suggesting their potential for embodied intelligence. While recent studies have evaluated embodied MLLMs in interactive settings, current benchmarks mainly target capabilities to perceive, understand, and interact with external objects, lacking a systematic evaluation of self-centric intelligence. To address this, we introduce MirrorBench, a simulation-based benchmark inspired by the classical Mirror Self-Recognition (MSR) test in psychology. MirrorBench extends this paradigm to embodied MLLMs through a tiered framework of progressively challenging tasks, assessing agents from basic visual perception to high-level self-representation. Experiments on leading MLLMs show that even at the lowest level, their performance remains substantially inferior to human performance, revealing fundamental limitations in self-referential understanding. Our study bridges psychological paradigms and embodied intelligence, offering a principled framework for evaluating the emergence of general intelligence in large models. Project page: https://fflahm.github.io/mirror-bench-page/.