Multimodal Model Evaluation

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 44% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 496

All topics

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
Jul 7, 2026Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen +1Multimodal Grounding LLM Uncertainty Estimation
Amazon
Evaluating and Understanding Model Editing for Medical Vision Language Models
Jul 6, 2026Guli Zhu, Chenwei Wu, Liyue ShenKnowledge Editing Medical VLMs
EECS, University of Michigan, Ann Arbor, MI 48105, USA
Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards
Jul 6, 2026Tianhao Niu, Ziyu Han, Qiguang Chen +5Data Visualization Code Generation Evaluation
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China
PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement
Jul 3, 2026Xianren Zhang, Delvin Ce Zhang, Dongwon Lee +1Privacy Leakage in Language Models MLLM Unlearning
The Pennsylvania State University · University of Sheffield
K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos
Jul 2, 2026Khush Attarde, Yusuf Ali, Megha Thukral +3Video Understanding Multimodal Large Language Models
Ogmen Robotics Inc., Massachusetts, USA · Georgia Institute of Technology, Atlanta, USA
EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
Jul 2, 2026Gianmarco Spinaci, Lukas Klic, Giovanni ColavizzaEducational Assessment Multimodal QA
Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy · Villa i Tatti – The Harvard University Center for Italian Renaissance Studies, Firenze, Italy · Center for Computational and Digital Humanities, University of Copenhagen, Copenhagen, Denmark
MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
Jul 2, 2026Yuanzhi Liu, Shousheng Zhao, Bo Zhou +2VLM Evaluation Benchmark Construction
1Beijing University of Posts and Telecommunications, Beijing, China
MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding
Jul 2, 2026Yuan Wang, Shujian Gao, Songtao Jiang +2Video Understanding Streaming Video Understanding
College of Computer Science and Technology, Zhejiang University, Zhejiang, China · Insititute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pennsylvania, USA
AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization
Jul 1, 2026Tianhong Zhou, Mingyang Han, Boyu Li +8Audio-Visual Synchronization Audio-Visual Alignment
Tsinghua University, Beijing, China · Alibaba Group, China · Fudan University, Shanghai, China
EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes
Jul 1, 2026Jihyeok Jung, Jeewu Lee, Sanghyeop Kim +2Egocentric Video Understanding AI Agent Benchmarks
KAIST AI · Sogang University · Ministry of Science and ICT
MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments
Jun 30, 2026Qingyun Liu, Jiwen Zhang, Jingyi Hu +2Multi-Agent Collaboration AI Agent Benchmarks
1Fudan University · 3The Chinese University of Hong Kong · 2Shanghai Innovation Institute
Learning to Deny: Action Denial in Multimodal Large Language Models
Jun 30, 2026Raiyaan Abdullah, Shehreen Azad, Yogesh Singh RawatMultimodal Large Language Models Causal Reasoning in Videos
Institute of Artificial Intelligence, University of Central Florida, USA
OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning
Jun 29, 2026Haocong He, Chenfei Liao, Zichen Wen +13Visual Spatial Reasoning Multimodal Large Language Models
SJTU · HKUST(GZ) · Shanghai AI Lab +6
From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation
Jun 29, 2026Shuchang Ye, Jinqiang Yu, Zhujun Xiao +6Video Content Moderation Multimodal Model Evaluation
1TikTok
Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation
Jun 29, 2026Chonghuinan Wang, Zhikai Chen, Chunwei Wang +9Unified Multimodal Models Multimodal Model Evaluation
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1
MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs
Jun 29, 2026Yuxuan Fan, Gyusik Seo, Jing Hao +3Audio-Visual Understanding Multimodal Large Language Models
1NTU Singapore · 2The University of Hong Kong · 3Johns Hopkins University +2
Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
Jun 29, 2026Yuchen He, Peizhi Ying, Liqi Cheng +4Document Information Extraction Multimodal Large Language Models
State Key Lab of CAD&CG, Zhejiang University · Zhejiang University Hangzhou, Zhejiang, China · Guangdong University of Technology Guangzhou, Guangdong, China
Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation Evaluation Multimodal Reasoning
UCSB · University of Washington · UCLA
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Jun 28, 2026Xiaomeng Fan, Wei Wu, Yuwei Wu +9Direct Preference Optimization Multimodal Large Language Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Department of Electrical and Computer System Engineering, Monash University
The Contagion Tensor: A Framework for Measuring Output-Distribution Coupling in Multi-Agent LLM Systems -- and Auditing the Claims It Enables
Jun 27, 2026Zewen LiuMulti-Agent LLM Systems LLM Auditing
IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
Jun 26, 2026Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar +10Multimodal Large Language Models Clinical Language Model Evaluation
Amazon Health AI
PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception
Jun 26, 2026Yana Wei, Hongbo Peng, Yanlin Lai +14Rubric-Based Evaluation Automated Evaluation
Johns Hopkins University · StepFun · Tsinghua University +1
AirGroundBench: Probing Spatial Intelligence in Multimodal Large Models under Heterogeneous Multi-View Embodied Collaboration
Jun 26, 2026Haotian Li, Yida Wang, Leyuan Wang +7Multi-View Consistency Multimodal Reasoning
Beijing University of Posts and Telecommunications · University of Electronic Science and Technology of China · Tsinghua University +1
HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models
Jun 25, 2026Jiajun Wu, Haoyu Kang, Yining Sun +13Video Understanding Video Content Moderation
Central South University · Tsinghua University · South China Normal University +10
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Evaluation Pitfalls and Challenges in Multimedia Event Extraction
Jun 25, 2026Philipp Seeberger, Steffen Freisinger, Tobias Bocklet +1Multimodal Grounding Multimodal Model Evaluation
Technische Hochschule Nürnberg Georg Simon Ohm
SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context
Jun 25, 2026Qinkai Zhang, Yanyan Zhao, Xin Lu +3LLM Personalization Multimodal Large Language Models
Harbin Institute of Technology
DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues
Jun 25, 2026Geng Li, Yuxin PengVisual Reasoning Multimodal Large Language Models
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence
Jun 25, 2026Jaden Moon, Arvind Pillai, Andrew CampbellUncertainty-Aware Multimodal Fusion Multimodal Fusion
Dartmouth College, United States
What We are Missing in Multimodal LLM Evaluation?
Jun 24, 2026Po-han Li, Shenghui Chen, Sandeep Chinchali +1Spatiotemporal Reasoning Multimodal Large Language Models
The University of Texas at Austin, USA

Jul 7, 2026, cs.CVJ/K move · Enter open · S save

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen, Tal Remez

Amazon

Multimodal Grounding LLM Uncertainty Estimation Multimodal Model Evaluation VLM Hallucination Mitigation

Open paper AskPDF arXiv

Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.

Momentum

Related topics

Latest papers 496