Multimodal Model Evaluation

Follow this topic Ask about this topic

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Chart-to-Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
What MLLMs Learn about When they Learn about Multimodal Reasoning
Oct 2, 2025Jiwan Chung, Neel Joshi, Pratyusha Sharma +2Multimodal Reasoning Multimodal Model Evaluation
Microsoft Research AI Frontiers · Yonsei University · Seoul National University
NeMo: Needle in a Montage for Video-Language Understanding
Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video Grounding Video-Language Model Evaluation
The Chinese University of Hong Kong · Phoenix TV · Stanford University +2
Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments
Sep 25, 2025Junfeng Yan, Biao Wu, Meng Fang +1Multimodal Model Evaluation Human-Computer Interaction
Australian Artificial Intelligence Institute, Sydney, Australia · University of Liverpool, Liverpool, United Kingdom
The Platonic Universe: Do Foundation Models See the Same Sky?
Sep 23, 2025UniverseTBD, :, Trinidad Borrell +11Cross-Modal Alignment Multimodal Model Evaluation
Independent Researcher · AstroAI · Harvard-Smithsonian CfA +4
Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection
Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2MLLM Evaluation Sarcasm Detection
University of Groningen, The Netherlands
Measuring Epistemic Humility in Multimodal Large Language Models
Sep 11, 2025Bingkui Tong, Jiaer Xia, Sifeng Shang +1VLM Hallucination Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates. · Hong Kong Baptist University, Hong Kong.
Explain Before You Answer: A Survey on Compositional Visual Reasoning
Aug 24, 2025Fucai Ke, Joy Hsu, Zhixi Cai +10Compositional Reasoning Visual Reasoning
Monash University · Stanford University · University of Washington +3
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM Evaluation Multimodal Reasoning
KAIST · MBZUAI
VGGSounder: Audio-Visual Evaluations for Foundation Models
Aug 11, 2025Daniil Zverev, Thaddäus Wiedemer, Ameya Prabhu +3Audio-Visual Understanding Multimodal Classification
Technical University of Munich, MCML · University of Tübingen · Tübingen AI Center +1
Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes
Jun 11, 2025Zhaoyang Wei, Bowen Jiang, Xumeng Han +6Multimodal Grounding Multimodal Robustness
University of Chinese Academy of Sciences(UCAS) · Tencent CDG
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
A Survey of Generative Categories and Techniques in Multimodal Generative Models
May 29, 2025Longzhen Han, Awes Mubarak, Almas Baimagambetov +2Multimodal Generation Multimodal Model Evaluation
School of Architecture, Technology and Engineering, University of Brighton, Lewes Road, BN2 4GJ, United Kingdom · University of Khorfakkan, Sharjah, United Arab Emirates
Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning
May 26, 2025Sanghyuk Chun, Olga RussakovskyCross-Modal Alignment Multimodal Model Evaluation
1Princeton University
MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation
May 23, 2025Jihan Yao, Yushi Hu, Wenyuan Wang +10Multimodal Generation Multimodal Model Evaluation
University of Washington · Allen Institute for AI
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
May 23, 2025Haoyu Sun, Huichen Will Wang, Jiawei Gu +2Software Engineering Web Application Generation
Tongji University · Shanghai AI Laboratory · University of Washington +1
Redemption Score: A Multi-Modal Evaluation Framework for Image Captioning via Distributional, Perceptual, and Linguistic Signal Triangulation
May 22, 2025Ashim Dahal, Ankit Ghimire, Saydul Akbar Murad +1Image Captioning Image Captioning Evaluation
University of Southern Mississippi, Hattiesburg, MS, USA · School of Business & Technology, Emporia State University, Emporia, KS, USA
MMS-VPR: A Fine-Grained Multimodal Street-Level Visual Place Recognition Dataset and Evaluation Benchmark for Dense Pedestrian Environments
May 18, 2025Yiwei Ou, Xiaobin Ren, Ronggui Sun +3Visual Place Recognition Multimodal Model Evaluation
The University of Auckland · Harbin Institute of Technology, Shenzhen
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic Fairness Multimodal Large Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Feb 13, 2025Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31Visually Grounded Reasoning VLM Reasoning
University of Cambridge
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation

Oct 20, 2025J/K move · Enter open · S save

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang+7

CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University

Data Visualization Chart-to-Code Generation Code Generation Code Generation Evaluation Multimodal Foundation Models Chart Understanding+2

Open paper AskPDF arXiv

We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (LMMs). Chart2Code is explicitly designed from a user-driven perspective, capturing diverse real-world scenarios and progressively increasing task difficulty. It consists of three levels: Level 1 (Chart Reproduction) reproduces charts from a reference figure and user query; Level 2 (Chart Editing) involves complex modifications such as changing chart types or adding elements; and Level 3 (Long-Table to Chart Generation) requires models to transform long, information-dense tables into faithful charts following user instructions. To our knowledge, this is the first hierarchical benchmark that reflects practical chart2code usage while systematically scaling task complexity. In total, Chart2Code contains 2,023 tasks across 22 chart types, paired with multi-level evaluation metrics that assess both code correctness and the visual fidelity of rendered charts. We benchmark 25 state-of-the-art (SoTA) LMMs, including both proprietary and the latest open-source models such as GPT-5, Qwen2.5-VL, InternVL3/3.5, MiMo-VL, and Seed-1.6-VL. Experimental results demonstrate that even the SoTA model GPT-5 averages only 0.57 on code-based evaluation and 0.22 on chart-quality assessment across the editing tasks, underscoring the difficulty of Chart2Code. We anticipate this benchmark will drive advances in multimodal reasoning and foster the development of more robust and general-purpose LMMs. Our code and data are available on Chart2Code.