Video-Language Model Evaluation

Follow this topic Ask about this topic

The field develops tests and metrics for whether models can answer questions, describe events, localize evidence, and follow instructions in videos, including long, streaming, and audio-visual settings. Common approaches include human-annotated benchmarks, controlled tests that shuffle frames or alter evidence, and measures of temporal grounding, hallucination, efficiency, and response timing. Results often show that strong question-answering scores can coexist with weak event ordering, evidence localization, and resistance to misleading cues. A central challenge is separating genuine use of video evidence from shortcuts, while making results comparable across models and realistic about deployment constraints.

Latest papers

103 papers20 papers in the last four weeks, down 5% on the four weeks before. 0.1% of all new papers.

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
Mar 7, 2026Xueqing Yu, Bohan Li, Yan Li +1Abstention in VLMs VLM Evaluation
Peking University · ByteDance
RIVER: A Real-Time Interaction Benchmark for Video LLMs
Mar 4, 2026Yansong Shi, Qingsong Zhao, Tianxiang Jiang +3Video-Language Model Evaluation Streaming Video Understanding
School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · College of Computer Science and Artificial Intelligence, Fudan University +1
Beyond Static Artifacts: A Forensic Benchmark for Video Deepfake Reasoning in Vision Language Models
Feb 25, 2026Zheyuan Gu, Qingsong Zhao, Yusong Wang +6Deepfake Detection Video-Language Model Evaluation
Institute of Artificial Intelligence, China Telecom (TeleAI) · Peking University · Fudan University
Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs
Feb 23, 2026Chang Liu, Yunfan Ye, Qingyang Zhou +5VLM Reasoning Physical Reasoning in Video
National University of Defense Technology, Changsha, China · Hunan University, Changsha, China
Rodent-Bench
Feb 20, 2026Thomas Heap, Laurence Aitchison, Emma Cahill +1Animal Behavior Analysis Video-Language Model Evaluation
University of Bristol, Bristol, UK
KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understanding
Feb 19, 2026Boda Lin, Yongjie Zhu, Xiaocheng Gong +2Video Captioning Video-Language Model Evaluation
Kuaishou Technology
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Feb 11, 2026Sethuraman T, Savya Khosla, Aditi Tiwari +12VLM Robustness Video-Language Model Evaluation
University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal Understanding Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
NeMo: Needle in a Montage for Video-Language Understanding
Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video Grounding Video-Language Model Evaluation
The Chinese University of Hong Kong · Phoenix TV · Stanford University +2
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Sep 2, 2025Mennatullah SiamVision-Language Grounding Video-Language Model Evaluation
TempCore: Are Video QA Benchmarks Temporally Grounded?
Sep 1, 2025Hyunjong Ok, Jaeho LeeVLM Evaluation Video-Language Model Evaluation
POSTECH
How Well Do Vision-Language Models Understand Movies?
Jun 6, 2025Emmanouil Zaranis, António Farinhas, Saul Santos +28Video-Language Model Evaluation Multimodal Understanding
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ILLC, University of Amsterdam +12
VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
May 23, 2025Shenghui Chen, Po-han Li, Sandeep Chinchali +1Information Bottleneck Video Summarization
The University of Texas at Austin

Mar 7, 2026J/K move · Enter open · S save

VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding

Xueqing Yu, Bohan Li, Yan Li, Zhenheng Yang

Peking University · ByteDance

Abstention in VLMs VLM Evaluation Video-Language Model Evaluation Vision-Language Models Long-Video Understanding

Open paper AskPDF arXiv

Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.