Multimodal Model Evaluation

Follow this topic Ask about this topic

Multimodal model evaluation studies whether systems can perceive, reason over, and generate content using combinations of text, images, audio, video, documents, and structured signals. Researchers build task-specific benchmarks and assess outputs with human judgments, programmatic checks, structured rubrics, and measures of evidence grounding, calibration, and safety. Controlled tests such as modality ablations, counterfactual inputs, and robustness checks help reveal shortcuts and failures that aggregate accuracy can hide. A central challenge is showing that a correct answer reflects reliable use of the relevant evidence, while keeping evaluations reproducible, representative, and comparable across settings.

Latest papers

772 papers87 papers in the last four weeks, up 28% on the four weeks before. 0.5% of all new papers.

ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response
Apr 23, 2026Stephan Xie, Ben Cohen, Mononito Goswami +6Time Series Reasoning Time Series QA
Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA · Datadog AI Research, New York, NY, USA · Amazon Web Services, Seattle, WA, USA
Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry
Apr 22, 2026Syed Nazmus Sakib, Nafiul Haque, Shahrear Bin Amin +4Visual Question Answering Plant Disease Classification
Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Computer Science and Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Agronomy, Gazipur Agricultural University, Gazipur, Bangladesh +1
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Apr 22, 2026Menghe Ma, Siqing Wei, Yuecheng Xing +5Optical Music Recognition Multimodal Reasoning
Beijing University of Posts and Telecommunications Beijing, China · China Conservatory of Music Beijing, China · Nanyang Technological University Singapore, Singapore
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
Exploring Spatial Intelligence from a Generative Perspective
Apr 22, 2026Muzhi Zhu, Shunyao Jiang, Huanyi Zheng +9Spatial Reasoning Unified Multimodal Models
Zhejiang University, State Key Laboratory of CAD & CG · Ant Group · Zhejiang University of Technology +1
CHASM: Unveiling Covert Advertisements on Chinese Social Media
Apr 22, 2026Jingyi Zheng, Tianyi Hu, Yule Liu +5Social Media Analysis Online Advertising
Hong Kong University of Science and Technology (Guangzhou) · Aarhus Univeristy
CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs
Apr 22, 2026Xingcheng Zhou, Hao Guo, Rui Song +5Visual Question Answering Video QA
Technical University of Munich · University of California, Los Angeles
X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis
Apr 22, 2026Gui Wang, Zehao Zhong, YongSong Zhou +6Healthcare Medical Diagnosis
School of Computer Science and Software Engineering, Shenzhen University · 2Tsinghua University · 5Wenzhou Medical University +3
SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark
Apr 22, 2026Gui Wang, YongSong Zhou, Kaijun Deng +4Spatiotemporal Reasoning Healthcare
School of Computer Science and Software Engineering, Shenzhen University · University of Nottingham · School of AI, Shenzhen University +1
WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring
Apr 22, 2026Mobin Habibpour, Niloufar Alipour Talemi, John Spodnik +2Thermal Imaging Visual Question Answering
Clemson University
Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
Apr 21, 2026Jing Jin, Hao Liu, Yan Bai +9Reasoning Evaluation Multimodal Reasoning
1Tsinghua University · 3Central South University · 2Meituan Inc +3
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models
Apr 21, 2026Josue Torres-Fonseca, Naihao Deng, Yinpei Dai +5Safety Evaluation of VLA Models LLM Safety Benchmarks
University of Michigan · Boise State University
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
Apr 20, 2026Shaden Alshammari, Kevin Wen, Abrar Zainal +5Mathematical Reasoning Benchmarks Multilingual Language Model Evaluation
MIT · HUMAIN · Individual Researcher +1
Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
Apr 20, 2026Samuel G. Balter, Ethan Jerzak, Connor T. JerzakMultimodal Large Language Models Multimodal Reasoning
University of Texas at Austin · National University of Singapore (NUS)
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts
Apr 20, 2026Run Xu, Lu Li, Rongzhao Zhang +1Image Captioning Multimodal Generation
Nanyang Technological University · Tongji University · Shanghai Artificial Intelligence Laboratory
Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
Apr 19, 2026Yunkai Dang, Yifan Jiang, Yizhu Jiang +3Selective Prediction Confidence Estimation in Language Models
School of Artificial Intelligence, Nanjing University.
Adverse-to-the-eXtreme Panoptic Segmentation: URVIS 2026 Study and Benchmark
Apr 18, 2026Yiting Wang, Nolwenn Peyratout, Tim Brodermann +13Image Segmentation Panoptic Segmentation
Organizers of the URVIS 2026 Challenge
Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models
Apr 18, 2026Bruce A. Bassett, Amy Rouillard, Sitwala Mundia +8Language Model Safety Evaluation Healthcare
School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · Wits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · Grai Labs, Cape Town, South Africa +3
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
Apr 18, 2026Xinru Yan, Boxi Cao, Yaojie Lu +4Unified Multimodal Models Neural Network Interpretability
University of Chinese Academy of Sciences, Beijing, China · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing
Apr 17, 2026Toby Perrett, Matthew Bouchard, William McCarthy3D Shape Editing Parametric CAD Modeling
Autodesk Research
Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs
Apr 17, 2026Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N. Balasubramanian +1Multimodal Model Evaluation Visual Spatial Reasoning
IIT, Hyderabad · Microsoft Research India
Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
Apr 17, 2026Manh Luong, Tamas Abraham, Junae Kim +6LLM Safety Benchmarks Language Model Safety Evaluation
Monash University, Australia · Defence Science and Technology Group, Australia
ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams
Apr 17, 2026Qiang Xu, Shengyuan Bai, Yu Wang +6Visual Reasoning Multimodal Model Evaluation
International Digital Economy Academy, Shenzhen, China · University of Wisconsin–Madison
RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees
Apr 17, 2026Yichen Xu, Yuanhang Liu, Chuhan Wang +5Sports Video Analysis Video Reasoning
Renmin University of China Beijing, China · Sichuan University Chengdu, China
From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
Apr 16, 2026Ke Xu, Yuhao Wang, Yu WangLLM Agent Evaluation Agent Evaluation
Shanghai Jiao Tong University
Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems
Apr 16, 2026Nishanth Madhusudhan, Vikas Yadav, Alexandre LacosteMultimodal QA Multimodal Reasoning
ServiceNow Research

Apr 23, 2026J/K move · Enter open · S save

ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response

Stephan Xie, Ben Cohen, Mononito Goswami, Junhong Shen+5

Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA · Datadog AI Research, New York, NY, USA · Amazon Web Services, Seattle, WA, USA

Time Series Reasoning Time Series QA Multimodal QA Multimodal Foundation Models Multimodal Model Evaluation Temporal QA

Open paper AskPDF arXiv

Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising yet underexplored capability of foundation models. In this work, we present ARFBench, a TSQA benchmark that evaluates the understanding of multimodal foundation models (FMs) on time series anomalies prevalent in software incident data. ARFBench consists of 750 questions across 142 time series and 5.38M data points from 63 production incidents sourced exclusively from internal telemetry at Datadog. We evaluate leading proprietary and open-source LLMs, VLMs, and time series FMs and observe that frontier VLMs perform markedly better than existing baselines; the leading model (GPT-5) achieves a 62.7% accuracy and 51.9% F1. We next demonstrate the promise of specialized multimodal approaches. We develop a novel TSFM + VLM hybrid prototype which we post-train on a small set of synthetic and real data that yields comparable overall F1 and accuracy with frontier models. Lastly, we find models and human domain experts exhibit complementary strengths. We define a model-expert oracle, a best-of-2 oracle selector over model and expert answers, yielding 82.8% F1 and 87.2% accuracy and establishing a new superhuman frontier for future TSQA models. The benchmark is available at https://huggingface.co/datasets/Datadog/ARFBench.