VidNum: Diagnosing VLM Failure Modes in Video-Grounded Numerical Reasoning
Authors: Shaoyang Cui, Lingbei Meng, Yaodi Luo, Peize He
Organizations: School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · University of Electronic Science and Technology of China, Chengdu, Sichuan, China
Video-grounded numerical reasoning requires Vision-Language Models (VLMs) to identify, track, and combine quantitative evidence across frames, actions, and scene changes. Existing benchmarks provide fragmented coverage: general VideoQA includes counting among broader tasks, while dedicated benchmarks focus on repetition counting, ultra-long-video enumeration, or instructional mathematics. We introduce VidNum, a manually curated and independently verified benchmark containing 1,167 multiple-choice questions. Its three task groups distinguish Direct and Distinct Enumeration, Conditioned and Structured Enumeration, and Compositional Quantitative Reasoning. Question-level annotations further identify the evidence target, counting structure, and required reasoning operation. The best evaluated VLM reaches 59.8% accuracy, compared with 98.2% for human annotators, and no evaluated open-weight model exceeds 45%. Stratified analyses reveal that failures are not uniformly distributed: structured target construction and action-grounded compositional reasoning form recurring bottlenecks across models. Zero-shot chain-of-thought prompting is not a reliable remedy: it recovers some errors but breaks previously correct answers, with effects that vary across models and task structures. VidNum therefore supports diagnostic analysis beyond a single aggregate score.
Figures & tables
Benchmark
Scale
Target diversity
Conditioned targets
Quantity composition
Item-level labels
Primary setting
General video question answering
TGIF-QA ( 2017 )
30,397 QA
✗
✗
✗
✗
Repetition counting
MVBench ( 2024 )
400 MCQ
✓
✗
✗
⚫
General VideoQA
Counting and quantitative video benchmarks
Countix ( 2020 )
8,757 videos
✗
✗
✗
✗
Cycle counting
UCFRep ( 2020 )
526 videos
✗
✗
✗
✗
Cycle counting
Table 1: Comparison with representative counting and quantitative video benchmarks.
Figure 1: Overview of VidNum . The benchmark covers three diagnostic task groups: Direct and Distinct Enumeration (DDE), Conditioned and Structured Enumeration (CSE), and Compositional Quantitative Reasoning (CQR). Examples illustrate different requirements for target specification, visual grounding, and quantity composition. The right panel shows the question distribution by target type, where [A], [O], and [E] denote action, object, and event targets.
Figure 2: Computation-oriented overview of the VidNum task taxonomy.
Figure 3: Item-level prediction transitions induced by zero-shot CoT across VidNum task groups. Each stacked bar is computed on questions with parsable NoCoT and CoT outputs. Green indicates items recovered by CoT, orange indicates previously correct items broken by CoT, and the right panel reports the net change, recovered minus broken, in percentage points. Thin gray strips show the end-to-end fraction of CoT outputs that could not be parsed. Models are grouped by family and ordered by scale within each family.
Figure 4: Action disadvantage across VidNum task groups.
Final-answer video QA can show whether a model predicts the right number, but not which instances it counted, when the supporting evidence occurs, or why it failed. We diagnose long-video quantitative reasoning in multimodal large language models (MLLMs) through three coupled abilities: enumerating query-relevant instances, temporally grounding supporting evidence, and aggregating the evidence into counts. To support this analysis, we build EC-Bench, an evidence-annotated evaluation suite with 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries across six reasoning categories, and human-verified evidence spans. We evaluate 22 open-source and proprietary MLLMs using timestamped visual frames and transcripts. The best average scores reach only 29.98% Enumeration F1 and 23.74% Counting accuracy, compared with human performance of 78.57% and 82.97%, respectively. Our analyses show that counting errors are rarely isolated arithmetic mistakes: Enumeration F1 is strongly associated with Counting accuracy, temporal grounding quality is associated with lower counting error, and Counting accuracy drops as supporting evidence becomes more distributed. These findings recast long-video counting as evidence retrieval, temporal grounding, deduplication, and aggregation across the video, rather than simple numerical prediction.
Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, comprising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis reveals that model architecture explains the largest proportion of performance variance (partial ω2=0.325), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity signals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associated with the language model component. Code and data are publicly available at https://github.com/fuy3/NumerosityVLM-Benchmark, and https://huggingface.co/datasets/fuy3/NumerosityVLM.
Yiming Fu, Fangjun Li, Xiujin Liu +7
Cognitive Robotics Lab, Department of Computer Science, The University of Manchester, Manchester M13 9PL, UK · University of Michigan, MI 48109, USA · Sheffield Hallam University, Sheffield S1 1WB, UK +2
Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.
Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo