VidNum: Diagnosing VLM Failure Modes in Video-Grounded Numerical Reasoning

Shaoyang Cui; Lingbei Meng; Yaodi Luo; Peize He

cs.CVApr 4, 2026

VidNum: Diagnosing VLM Failure Modes in Video-Grounded Numerical Reasoning

Authors: Shaoyang Cui, Lingbei Meng, Yaodi Luo, Peize He

Organizations: School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · University of Electronic Science and Technology of China, Chengdu, Sichuan, China

Abstract

Video-grounded numerical reasoning requires Vision-Language Models (VLMs) to identify, track, and combine quantitative evidence across frames, actions, and scene changes. Existing benchmarks provide fragmented coverage: general VideoQA includes counting among broader tasks, while dedicated benchmarks focus on repetition counting, ultra-long-video enumeration, or instructional mathematics. We introduce VidNum, a manually curated and independently verified benchmark containing 1,167 multiple-choice questions. Its three task groups distinguish Direct and Distinct Enumeration, Conditioned and Structured Enumeration, and Compositional Quantitative Reasoning. Question-level annotations further identify the evidence target, counting structure, and required reasoning operation. The best evaluated VLM reaches 59.8% accuracy, compared with 98.2% for human annotators, and no evaluated open-weight model exceeds 45%. Stratified analyses reveal that failures are not uniformly distributed: structured target construction and action-grounded compositional reasoning form recurring bottlenecks across models. Zero-shot chain-of-thought prompting is not a reliable remedy: it recovers some errors but breaks previously correct answers, with effects that vary across models and task structures. VidNum therefore supports diagnostic analysis beyond a single aggregate score.

Figures & tables

Benchmark	Scale	Target diversity	Conditioned targets	Quantity composition	Item-level labels	Primary setting
General video question answering
TGIF-QA ( 2017 )	30,397 QA	✗	✗	✗	✗	Repetition counting
MVBench ( 2024 )	400 MCQ	✓	✗	✗	⚫	General VideoQA
Counting and quantitative video benchmarks
Countix ( 2020 )	8,757 videos	✗	✗	✗	✗	Cycle counting
UCFRep ( 2020 )	526 videos	✗	✗	✗	✗	Cycle counting

Table 1: Comparison with representative counting and quantitative video benchmarks.

Figure 1: Overview of VidNum . The benchmark covers three diagnostic task groups: Direct and Distinct Enumeration (DDE), Conditioned and Structured Enumeration (CSE), and Compositional Quantitative Reasoning (CQR). Examples illustrate different requirements for target specification, visual grounding, and quantity composition. The right panel shows the question distribution by target type, where [A], [O], and [E] denote action, object, and event targets.

Figure 2: Computation-oriented overview of the VidNum task taxonomy.

Figure 3: Item-level prediction transitions induced by zero-shot CoT across VidNum task groups. Each stacked bar is computed on questions with parsable NoCoT and CoT outputs. Green indicates items recovered by CoT, orange indicates previously correct items broken by CoT, and the right panel reports the net change, recovered minus broken, in percentage points. Thin gray strips show the end-to-end fraction of CoT outputs that could not be parsed. Models are grouped by family and ordered by scale within each family.

Figure 4: Action disadvantage across VidNum task groups.

Explore similar work

Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting
Mar 31, 2026Fumihiko Tsuchiya, Taiki Miyanishi, Shunsuke Yasuki +5Long-Video QA Multimodal Large Language Models
The University of Tokyo, Japan · Institute of Science Tokyo, Japan · National Institute of Informatics, Japan
NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models
Aug 15, 2026Yiming Fu, Fangjun Li, Xiujin Liu +7VLM Evaluation VLM Interpretability
Cognitive Robotics Lab, Department of Computer Science, The University of Manchester, Manchester M13 9PL, UK · University of Michigan, MI 48109, USA · Sheffield Hallam University, Sheffield S1 1WB, UK +2
Counting to Four is still a Chore for VLMs
Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoVision-Language Models VLM Evaluation
MBZUAI

Mar 31, 2026J/K move · Enter open · S save

Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

Fumihiko Tsuchiya, Taiki Miyanishi, Shunsuke Yasuki, Mahiro Ukai+4

The University of Tokyo, Japan · Institute of Science Tokyo, Japan · National Institute of Informatics, Japan

Long-Video QA Multimodal Large Language Models Long-Video Understanding Multimodal Model Evaluation

Open paper AskPDF arXiv

Final-answer video QA can show whether a model predicts the right number, but not which instances it counted, when the supporting evidence occurs, or why it failed. We diagnose long-video quantitative reasoning in multimodal large language models (MLLMs) through three coupled abilities: enumerating query-relevant instances, temporally grounding supporting evidence, and aggregating the evidence into counts. To support this analysis, we build EC-Bench, an evidence-annotated evaluation suite with 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries across six reasoning categories, and human-verified evidence spans. We evaluate 22 open-source and proprietary MLLMs using timestamped visual frames and transcripts. The best average scores reach only 29.98% Enumeration F1 and 23.74% Counting accuracy, compared with human performance of 78.57% and 82.97%, respectively. Our analyses show that counting errors are rarely isolated arithmetic mistakes: Enumeration F1 is strongly associated with Counting accuracy, temporal grounding quality is associated with lower counting error, and Counting accuracy drops as supporting evidence becomes more distributed. These findings recast long-video counting as evidence retrieval, temporal grounding, deduplication, and aggregation across the video, rather than simple numerical prediction.

Abstract

Explore similar work

Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

Counting to Four is still a Chore for VLMs