Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 48% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Mar 19, 2026Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2Multimodal Large Language Models AI Agent Benchmarks
University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1
SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models
Mar 17, 2026Tianyu Xie, Jinfa Huang, Yuexiao Ma +11Audio-Visual Understanding Spoken Dialogue Systems
Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · Department of Computer Science, University of Rochester, Rochester, NY, USA +2
NeSy-Route: A Neuro-Symbolic Benchmark for Constrained Route Planning in Remote Sensing
Mar 17, 2026Ming Yang, Zhi Zhou, Shi-Yu Tian +3Remote Sensing Image Understanding Remote Sensing
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China
DamageArbiter: A Multimodal Arbitration Framework for Disaster Damage Assessment from Street-View Imagery
Mar 16, 2026Yifan Yang, Lei Zou, Wenjing Gong +6Disaster Damage Assessment Multimodal Classification
Department of Geography, Texas A&M University, College Station, USA · Department of Landscape Architecture & Urban Planning, Texas A&M University, College Station, USA · Department of Industrial and Systems Engineering, University of Florida, Gainesville, USA +2
GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing
Mar 12, 2026Mingxin Liu, Ziqian Fan, Zhaokai Wang +13Image Editing Evaluation Text-Guided Image Editing
Shanghai Jiao Tong University · South China University of Technology · Fudan University +2
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Mar 12, 2026Ruirui Chen, Weifeng Jiang, Chengwei Qin +3Moral Reasoning in Language Models Theory of Mind
Institute of High Performance Computing (IHPC) and 2Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Hong Kong University of Science and Technology (Guangzhou), China
EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
Mar 10, 2026Chengjun Yu, Xuhan Zhu, Chaoqun Du +4Multimodal Large Language Models Egocentric Video Understanding
University of Science and Technology of China · Foundation Model Team, Li Auto Inc. · Tsinghua University
Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language
Mar 3, 2026Michelle Stegeman, Lena Philipp, Fennie van der Graaf +20Computational Pathology Medical Imaging Foundation Models
and on behalf of the UNICORN consortium · Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands · Department of Medical Imaging, Radboud University Medical Center, Nijmegen, The Netherlands +2
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
Beyond Pixels: A Vector-to-Graph Framework for Reliable Schematic Auditing
Feb 12, 2026Chengwei Ma, Zhen Tian, Zhou Zhou +5Multimodal Large Language Models Multimodal Model Evaluation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · Guangdong Power Grid Co., Ltd., Yangjiang Power Supply Bureau, Yangjiang, China · Shanghai University, Shanghai, China +1
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Feb 11, 2026Sethuraman T, Savya Khosla, Aditi Tiwari +12VLM Robustness Video-Language Model Evaluation
University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6
ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
Feb 10, 2026Yijie Lin, Guofeng Ding, Haochen Zhou +3Multimodal IR Visual Spatial Reasoning
College of Computer Science, Sichuan University, Chengdu, China · National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University, China
UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models
Feb 9, 2026Cheng Yang, Chufan Shi, Bo Shui +7Unified Multimodal Models Image Generation
University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1
MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training
Feb 6, 2026Lucia Gordon, Serge Belongie, Christian Igel +1Test-Time Adaptation Multimodal Robustness
Harvard University, USA · University of Copenhagen, Denmark
RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity
Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model Evaluation Audio QA
University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3
Quantifying the Gap between Understanding and Generation within Unified Multimodal Models
Feb 2, 2026Chenlong Wang, Yuhang Chen, Zhihan Hu +4Unified Multimodal Models Cross-Modal Alignment
Independent Researcher · University of Maryland · University of Waterloo +1
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Audio-Language Model Evaluation Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Social Caption: Evaluating Social Understanding in Multimodal Models
Jan 21, 2026Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Carnegie Mellon University
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
Jan 20, 2026Aahana Basappa, Pranay Goel, Anusri Karra +3Reasoning Evaluation Multimodal Large Language Models
Centennial High School, Frisco, Texas, USA · Algoverse AI Research, Palo Alto, Cali- fornia, USA · Lebanon Trail High School, Frisco, Texas, USA +1
M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
Jan 13, 2026Juntao Jiang, Jiangning Zhang, Yali Bi +7Medical VLMs Multimodal CoT Reasoning
1Zhejiang University · University of Science and Technology of China · 3East China Normal University +2
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8
Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment
Jan 8, 2026Han Zhu, Jiale Chen, Chengkun Cai +8Language Model Safety Evaluation Multimodal Large Language Models
Hong Kong University of Science and Technology · University College London · AISpeech +1
Unexplored flaws in multiple-choice VQA make benchmarking unreliable
Nov 27, 2025Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf +3Visual Question Answering Prompt Sensitivity
Technical University of Munich · Volkswagen AG · Helmholtz AI +1
CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?
Nov 12, 2025Peiyu Li, Xiaobao Huang, Ting Hua +1Generative Modeling Multimodal Model Evaluation
University of Notre Dame
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
Oct 20, 2025Yulin Luo, Chun-Kai Fan, Menghang Dong +19Long-Horizon Robotic Manipulation Multimodal Large Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer2026 Science, Peking University · Institute for Brain and Intelligence, Fudan University · University of Science andJul Technology Beijing +2
From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Oct 20, 2025Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu +8Data Visualization Code Generation
CSU-JPG, Central South University · National University of Singapore · Nanyang Technological University
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
NeMo: Needle in a Montage for Video-Language Understanding
Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video Grounding Video-Language Model Evaluation
The Chinese University of Hong Kong · Phoenix TV · Stanford University +2
The Platonic Universe: Do Foundation Models See the Same Sky?
Sep 23, 2025UniverseTBD, :, Trinidad Borrell +11Cross-Modal Alignment Multimodal Model Evaluation
Independent Researcher · AstroAI · Harvard-Smithsonian CfA +4

Mar 19, 2026, cs.CVJ/K move · Enter open · S save

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

Thomas De Min, Subhankar Roy, Stéphane Lathuilière, Elisa Ricci+1

University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK+1

Multimodal Large Language Models AI Agent Benchmarks Proactive Assistance Multimodal Foundation Models Multimodal Agents Multimodal Model Evaluation+1

Open paper AskPDF arXiv

Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can MLLMs exhibit a similar "proactive" behavior by requesting simple user interventions? To investigate this, we introduce ProactiveBench, a benchmark built from seven repurposed datasets that tests proactiveness across different tasks such as recognizing occluded objects, enhancing image quality, and interpreting coarse sketches. We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains. Surprisingly, we found that conversation histories and in-context learning introduce negative biases, hindering performance. Finally, we explore a simple fine-tuning strategy based on reinforcement learning: its results suggest that proactiveness can be learned, even generalizing to unseen scenarios. We publicly release ProactiveBench as a first step toward building proactive multimodal models.

Momentum

Related topics

Latest papers 496