Video-Language Model Evaluation

Overview

The field develops tests and metrics for whether models can answer questions, describe events, localize evidence, and follow instructions in videos, including long, streaming, and audio-visual settings. Common approaches include human-annotated benchmarks, controlled tests that shuffle frames or alter evidence, and measures of temporal grounding, hallucination, efficiency, and response timing. Results often show that strong question-answering scores can coexist with weak event ordering, evidence localization, and resistance to misleading cues. A central challenge is separating genuine use of video evidence from shortcuts, while making results comparable across models and realistic about deployment constraints.

Latest papers

CardsList

115 papers16 papers in the last four weeks, down 27% on the four weeks before. 0.1% of all new papers.

  1. V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

    Nov 13, 2026Keuntae Kim, Dongjun Kim, Yong Suk ChoiVideo-Language ModelsTemporal Reasoning in VLMs

  2. Evaluating Video LLMs' Understanding of Corner Cases in Autonomous Driving

    Nov 12, 2026Gabriel Manalu, Minh Son Hoang, Hyeontaek Hwang +6Video-Language Model EvaluationAutonomous Driving Benchmarks

  3. CineMME: From Dialogue Grounding to Narrative Reasoning in Multimodal Video Understanding

    Nov 11, 2026Jin Liu, Dawei Du, Yexiang Liu +7Video-Language Model EvaluationAudio-Visual Understanding

  4. PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation

    Nov 10, 2026Huiyi Chen, Weitai Kang, Shicheng Fan +5Physical Consistency in Video GenerationVideo Generation Evaluation

  5. What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis

    Nov 3, 2026Christian Pipal, Eva-Maria Vogel, Morgan WackModality Reliance in Multimodal ModelsVideo-Language Model Evaluation

  6. DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas

    Oct 31, 2026Xinyue Huo, Pengfei Chen, Jiannan Ge +5Video UnderstandingLong-Video QA

  7. Selecting or Solving? What Best-of-N Verification Delivers in Video Reasoning

    Oct 31, 2026Martin Q. Ma, Yuxiao Qu, Ruslan Salakhutdinov +3Video ReasoningVideo-Language Model Evaluation

  8. LF-VideoQA: Evaluating Long-Video Understanding with Long-Form Theory-of-Mind Questions

    Oct 30, 2026Shaden Shaar, Bradon Thymes, Majd Aldaye +2Long-Video QALong-Video Understanding

  9. CodeCap-Bench: Can Code Serve as a Good Video Caption?

    Oct 27, 2026Jiafu Tang, Yifan Xu, Eric Tang +11Video CaptioningVideo Caption Evaluation

  10. PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

    Oct 20, 2026Heran Hu, Yuhang Dong, Zhucun Xue +3Video-Language Model EvaluationVideo Understanding

  11. CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models

    Oct 20, 2026Marc Chapus, Carlos Crispim-Junior, Véronique Eglin +1Counting in VLMsVisual Counting

  12. From Cause to Effect: Generating Intermediate Event Chains in Videos

    Oct 19, 2026Ridouane Ghermi, Vicky Kalogeiton, Ivan LaptevCausal Reasoning in VideosVideo-Language Model Evaluation

  13. From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

    Oct 15, 2026Boyu Cui, Junhao Luo, Zhihao Wang +5Proactive Video UnderstandingVideo-Language Model Evaluation

  14. VisualTrust: Video Reasoning under On-Screen Claim Framing

    Oct 15, 2026Huachao Zhu, Junjielong Xu, Sicheng Li +5Video-Language Model EvaluationVLM Robustness

  15. RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

    Oct 13, 2026Mahsa Khoshnoodi, Sarah Adel BargalCounting in VLMsVideo-Language Models

  16. OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

    Oct 13, 2026Xuyang Liu, Shihao Wang, Xinyu Wei +5Video-Language Model EvaluationMultimodal Grounding

  17. Meta-Video: Efficient Standardized Evaluation for Video Understanding

    Oct 13, 2026Soumya Shamarao Jahagirdar, Anna Kukleva, Vishaal Udandarao +6Video-Language Model EvaluationVideo Understanding

  18. Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

    Oct 6, 2026Dan Ben-Ami, Kobi Cohen, Chaim BaskinStreaming Video UnderstandingVideo-Language Model Evaluation

  19. InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

    Oct 5, 2026Enxin Song, Suhao Yu, Yifei Xu +7Streaming Video UnderstandingVideo-Language Model Evaluation

  20. Video-Index: A Curated Meta-Benchmark for Video Understanding

    Oct 1, 2026Enxin Song, Yinuo Xu, Shusheng Yang +2Video UnderstandingVideo-Language Model Evaluation

  21. SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

    Sep 29, 2026Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy +1Video-Language Model EvaluationVideo Reasoning