Video-Language Model Evaluation

Ask about this topic

Momentum

18 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 55

All topics
CardsList

  1. Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation

    Oct 6, 2026Lichen Zhu, Yiheng Wang, Yueqian Lin +2Video-Language Model EvaluationVideo QA

    Duke University, Durham, NC, USA

  2. Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

    Oct 6, 2026Dan Ben-Ami, Kobi Cohen, Chaim BaskinStreaming Video UnderstandingVideo-Language Model Evaluation

    INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel

  3. InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

    Oct 5, 2026Enxin Song, Suhao Yu, Yifei Xu +7Streaming Video UnderstandingVideo-Language Model Evaluation

    University of Pennsylvania · Stanford University · University of Illinois at Urbana-Champaign +3

  4. Video-Index: A Curated Meta-Benchmark for Video Understanding

    Oct 1, 2026Enxin Song, Yinuo Xu, Shusheng Yang +2Video UnderstandingVideo-Language Model Evaluation

    University of Pennsylvania · New York University · Princeton University

  5. Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics

    Sep 30, 2026Zhixi Zhu, Kristina GligoricVLM EvaluationVideo-Language Model Evaluation

    Johns Hopkins University

  6. SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

    Sep 29, 2026Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy +1Video-Language Model EvaluationVideo Reasoning

    New York University

  7. Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning

    Sep 29, 2026Yanan Wang, Tingsong Li, Kaixun Jiang +3Direct Preference OptimizationVideo-Language Model Evaluation

    Zhejiang University · Alibaba Token Hub, Alibaba Group · University of Science and Technology of China +2

  8. ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

    Sep 28, 2026Gueter Josmy Faure, Min-Hung Chen, Hao Ping Wang +3Video-Language Model EvaluationTemporal Video Understanding

    National Taiwan University · NVIDIA

  9. PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

    Sep 28, 2026Shayekh Bin Islam, Hwanjun SongLLM-as-a-JudgeVideo-Language Model Evaluation

    KAIST

  10. Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

    Sep 24, 2026Shuzhi Gong, Fengze Sun, Yuansan LiuTemporal Video GroundingVideo-Language Model Evaluation

    The University of Melbourne

  11. CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

    Sep 23, 2026Shuo Xing, Pooja Verlani, Balu Adsumilli +1Video UnderstandingVideo-Language Model Evaluation

    Texas A&M University, College Station, USA · Google Inc., Mountain View, USA

  12. Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows

    Sep 23, 2026Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang +4Video UnderstandingVideo-Language Model Evaluation

    University of Washington, USA · Georgia Institute of Technology, USA · Google DeepMind, USA

  13. A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery

    Sep 22, 2026Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1Video-Language Model EvaluationHierarchical Representation Learning

    Data Science Center HAVA-Lab, University of Amsterdam · Video & Image Sense Lab, Informatics Institute, University of Amsterdam · Amsterdam UMC Location University of Amsterdam, Surgery +2

  14. Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

    Sep 15, 2026Zhaoyang Wei, Zipeng Wang, Yushe Cao +10Audio-Visual UnderstandingVideo-Language Model Evaluation

    University of Chinese Academy of Sciences, China · Tsinghua University, China · Tencent, China

  15. Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models

    Sep 15, 2026Rwiddhi Chakraborty, Yinong, Wang +7Video-Language Model EvaluationVideo QA

    Oliver · University of Copenhagen · Carnegie Mellon University +4

  16. DementiaCare-Bench: A Modality-Validated Video Benchmark

    Sep 14, 2026Afrouz Sheikholeslami, Yuankai Qi, Xuyun Zhang +4Alzheimer's DiseaseHealthcare

    School of Computing, Macquarie University, Sydney, NSW, Australia · School of Electrical and Computer Engineering, University of Sydney, Sydney, NSW, Australia · University of California, Merced, CA, USA

  17. LiveProBench: Can Streaming Video Models Really Interact Like Humans?

    Sep 11, 2026Kaixuan Du, Xin Wan, Hang Zhang +4Streaming Video UnderstandingVideo-Language Model Evaluation

    School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba

  18. Can We Trust Video Hallucination Detectors? VidHalLoc for Evaluating the Evaluators

    Sep 9, 2026Xinyu Chen, Adnan Mahmood, Mark DrasHallucination Detection in VLMsHallucination Detection

  19. MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs

    Sep 8, 2026Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche +7Video-Language Model EvaluationVideo-Language Models

    Electrical and Computer Engineering Department Northeastern University, Boston, MA, USA

  20. Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics

    Sep 8, 2026Ruibo Ming, Lei Sun, Deheng Zhang +8Video-Language Model EvaluationVideo-Language Models

    1INSAIT, Sofia University “St. Kliment Ohridski” · 2Adobe Research · 3Snap Research

  21. Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?

    Sep 8, 2026Bangshuo Zhu, Wei Song, Yuxin Cao +4VLM RobustnessAdversarial Attacks on VLMs

    School of Computer Science and Engineering, University of New South Wales, New South Wales 2052, Australia · School of Information and Communication Technology, Griffith University, Queensland 4111, Australia · School of Computing, National University of Singapore, Singapore 117417, Singapore +2

  22. Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

    Sep 3, 2026Prakhar KhatriVideo-Language Model EvaluationLong-Video Understanding

    Independent Researcher

  23. TempCloze: Can Video-LLMs Identify the Missing Middle?

    Sep 1, 2026Wenqi Pei, Henry Hengyuan Zhao, Yilai Liu +4Video-Language Model EvaluationVideo-Language Models

    The University of Hong Kong · National University of Singapore · Peking University

  24. ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits

    Sep 1, 2026Sethuraman T, Savya Khosla, Onkar Kishor Susladkar +6Video-Language Model EvaluationVideo-Language Models

    University of Illinois Urbana-Champaign · Adobe Research

  25. TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film

    Aug 30, 2026Kaishuu Shinozaki-Conefrey, Olivier Pascaud, Robin Courant +3Audio-Visual UnderstandingVideo-Language Model Evaluation

    LIX, Ecole Polytechnique, IP Paris, Palaiseau, France · New York University, New York, NY, USA · Ecole nationale supérieure Louis-Lumière, Saint-Denis, France +1

  26. DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos

    Aug 30, 2026Bomiao Wang, Zekai Shao, Jiexiang Lan +3Video UnderstandingVideo-Language Model Evaluation

    Fudan University · The Hong Kong University of Science and Technology (Guangzhou)

  27. NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

    Aug 13, 2026Yuheng Huang, Jianlang Chen, Jiayang Song +6Video UnderstandingVideo-Language Model Evaluation

    The University of Tokyo, Tokyo, Japan · Kyushu University, Fukuoka, Japan · Macau University of Science and Technology, Macau, China +2

  28. EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

    Aug 13, 2026Weitao Chen, Hu Jiaxin, Xie Tianyidan +15Egocentric Video QAVideo-Language Model Evaluation

    Nanjing University, China · Huawei Technologies Co., Ltd., China · Tianjin University, China

  29. From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning

    Aug 10, 2026Han Zhang, Yilin Zhao, Zaid Pervaiz Bhat +5Video-Language Model EvaluationVideo Anomaly Detection

    NVIDIA · Santa Clara University

  30. VideoVIBE: A Video-Grounded Diagnostic Benchmark for One-Shot Interactive Website Generation

    Aug 10, 2026Jiajun Xu, Yanghao Zhou, Jingyun Liao +6Web Application GenerationVideo-Language Model Evaluation

    University of Technology Sydney · Beijing Institute of Technology · Hunan University +4

  31. Action- and Language-Conditioned Video Assessment for Embodied Control

    Aug 8, 2026Hwanhee Kim, Jaehyun Jang, Seungmin Cha +3Video-Language Model EvaluationLong-Horizon Agent Evaluation

    Robotics Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

  32. Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

    Aug 7, 2026Wenzhang Sun, Chunfeng Wang, Xiangchen Yin +3Video-Language Model EvaluationVideo-Language Models

  33. The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

    Aug 6, 2026Sarvesh Baskar, Zikui Cai, Shayan Shabihi +5Video-Language Model EvaluationTemporal Video Understanding

    University of Maryland, College Park, USA · Scale AI, USA

  34. Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos

    Aug 5, 2026Yang Wang, Yanan Ma, Yiqi Liu +7Video-Language Model EvaluationMultimodal Reasoning

    University of Manchester · University of Sheffield · 2Durham University +1

  35. CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

    Aug 5, 2026Mukhtiar Ali, Harsh Dubey, Sugam Mishra +1Video-Language Model EvaluationVideo Captioning

    South Dakota State University Brookings, SD, USA

  36. Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models

    Aug 4, 2026Yuxin Cao, Wei Song, Jingling Xue +1Selective PredictionVideo-Language Model Evaluation

    National University of Singapore, Singapore · University of New South Wales, Australia

  37. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    Aug 3, 2026Zhongjie Ba, Shengwang Xu, Peng Cheng +4Video-Language Model EvaluationVideo QA

    State Key Lab. of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security Hangzhou, Zhejiang, China · Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

  38. RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos?

    Aug 3, 2026Hongjie Zhou, Shiqin Wang, Haoyang Chen +5Video-Language Model EvaluationRemote Sensing

    Wuhan University · Zhongguancun Academy

  39. Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

    Jul 14, 2026Jae Joong LeeVLM EvaluationVideo-Language Model Evaluation

    Department of Computer Science Purdue University West Lafayette, Indiana, USA

  40. Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video

    Jul 13, 2026Mohammad Al-Ratrout, Shayla Sharmin, Aditya Raikwar +1Visual Object TrackingVideo-Language Model Evaluation

  41. Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation

    Jun 22, 2026Yu Cao, Ziquan Liu, Zhensong Zhang +3VLM EvaluationPhysical Consistency in Video Generation

    Queen Mary University of London · Huawei Darwin Research Center · Imperial College London

  42. Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models

    Jun 19, 2026Bita Azari, Zoe Stanley, Avneet Batra +4VLM EvaluationVideo-Language Model Evaluation

    Simon Fraser University, Canada

  43. Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?

    Jun 8, 2026Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh +5LLM Fine-TuningSynthetic Data Generation

    Qualcomm AI Research · York University · Vector Institute for AI

  44. VCIFBench: Evaluating Complex Instruction Following for Video Understanding

    Jun 3, 2026Huangchen Xu, Yuan Wu, Yi ChangVLM EvaluationVideo Understanding

    School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University · 3International Center of Future Science, Jilin University

  45. Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models

    May 26, 2026Oscar Chew, Serhii Honcharenko, Qian-Hui Chen +4Temporal Video GroundingVideo-Language Model Evaluation

    Texas A&M University · National Taiwan University · Stanford University +1

  46. When Vision Speaks for Sound

    May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model EvaluationAudio-Visual Understanding

    dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1

  47. EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistant

    May 11, 2026Zichen Wen, Boxue Yang, Junlong Ke +5Streaming Video UnderstandingSelf-Training

    EPIC Lab, Shanghai Jiao Tong University · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) +1

  48. TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

    May 11, 2026Junzhe Chen, Siyuan Meng, Yuxi Chen +3Video-Language Model EvaluationVideo-Language Models

    Tianjin University

  49. PushupBench: Your VLM is not good at counting pushups

    Apr 25, 2026Shengzhi Li, Jiarun Chen, Karun Sharma +2VLM EvaluationVideo-Language Model Evaluation

  50. VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

    Apr 2, 2026Jiahao Meng, Yue Tan, Qi Xu +9Video-Language Model EvaluationLong-Video QA

    PKU · WHU · CASIA +2

  51. Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models

    Feb 11, 2026Sethuraman T, Savya Khosla, Aditi Tiwari +12VLM RobustnessVideo-Language Model Evaluation

    University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6

  52. SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

    Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Audio-Language Model EvaluationMultimodal Grounding

    Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada

  53. NeMo: Needle in a Montage for Video-Language Understanding

    Sep 29, 2025Zi-Yuan Hu, Shuo Liang, Duo Zheng +10Temporal Video GroundingVideo-Language Model Evaluation

    The Chinese University of Hong Kong · Phoenix TV · Stanford University +2

  54. TempCore: Are Video QA Benchmarks Temporally Grounded?

    Sep 1, 2025Hyunjong Ok, Jaeho LeeVLM EvaluationVideo-Language Model Evaluation

    POSTECH

  55. VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

    May 23, 2025Shenghui Chen, Po-han Li, Sandeep Chinchali +1VLM EvaluationVideo Summarization

    The University of Texas at Austin