Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.
Figures & tables
Figure 1: Example frames from CountVideoBench illustrating coverage across cardinality (top) and object categories (bottom). Camera motion and isolated object classes isolate counting from recognition and object motion while preserving viewpoint variation.
Figure 2: Ground-truth count distributions in three video benchmarks with counting questions: VideoCount, CG-AV-Counting, and VCBench. Probability mass concentrates on small counts, with long tails of rare large counts carrying negligible statistical weight.
Benchmark
Range
Distinct counts
Med [Q1–Q3]
Top-3 share
MVBench [ 13 ]
0–9
10
3 [2–3]
66.5%
NExT-QA [ 22 ]
1–15
13
2 [2–4]
72.5%
TGIF-QA [ 11 ]
0–10
10
2 [0–4]
67.9%
Video-MME [ 8 ]
0–600000
25
4 [2–6]
53.5%
VideoCount [ 1 ]
1–1203
22
1 [1–2]
93.1%
CG-AV-Counting [ 16 ]
1–76
31
4 [3–7]
49.6%
Table 1: Distribution characteristics of the ground-truth count variable in video benchmarks with counting questions. CountVideoBench is reported in the last row.
Model
EM
Acc@10%
MAE
RMSE
Coverage
∣S∣
PP
AdaReTaKe-7B
15.1
34.4
4.5
6.8
91.1
28
11.3
AdaReTaKe-72B
16.7
39.1
4.2
6.2
97.5
27
12.3
InternVL2.5-8B
14.0
38.2
15.8
28.5
97.9
16
9.4
InternVL2.5-78B
15.8
23.4
4.5
8.5
49.7
17
5.9
LLaVA-Video-7B
26.8
56.3
3.7
6.1
98.9
16
8.0
LLaVA-Video-72B
14.1
27.8
2.5
3.4
69.1
21
10.5
Table 2: Results on CountVideoBench. EM, Acc@10% and Coverage in percent; MAE and RMSE in object-count units (lower is better). Bold marks the best value per column; for MAE/RMSE it ignores † , whose Coverage is 1.2% . Oracle: exact prediction everywhere.
Figure 3: Model coverage by ground-truth count. Coverage is the proportion of videos for which the model produces a numerical answer.
Figure 4: Exact Match accuracy by ground-truth count and model. Each cell reports the percentage of videos for which the model’s predicted count equals the ground-truth exactly.
Figure 5: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos. (a) LLaVA-Video-7B. (b) GPT-5.4. (c) Gemini-3.1-pro.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Ground-truth count distributions in the representative video benchmarks with counting questions. Each panel shows the number of videos at each ground-truth count.
Table 3: Generation parameters. U(a,b) denotes a uniform draw on [a,b] . The polar angle is drawn subject to the placement constraint described below, which induces the effective distribution reported in the last row of the camera block.
Figure 7: Exact Match by ground-truth cardinality on the 104 -video human-study subset, for all evaluated models and for the twenty human participants (bottom row). Each model cell aggregates four videos and therefore takes values in {0,25,50,75,100}% ; each human cell aggregates 80 responses. The human row declines steadily from 99% at the smallest counts to 74% at n=50 , without the isolated peaks and adjacent collapses visible across the model rows.
Respondent
EM
Acc@10%
Coverage
∣S∣
PP
AdaReTaKe-7B
14.4
30.8
90.4
14.0
9.7
AdaReTaKe-72B
16.3
40.4
96.2
18.0
13.3
InternVL2.5-8B
13.5
36.5
97.1
12.0
9.6
InternVL2.5-78B
13.5
19.2
44.2
10.0
5.0
LLaVA-Video-7B
26.9
56.7
99.0
12.0
7.6
LLaVA-Video-72B
15.4
31.7
67.3
17.0
9.8
Appendix
Table 4: Human baseline and model performance on the 104 -video subset. Human rows report the mean over 20 participants; the majority-vote row aggregates the modal answer per video. Support size and perplexity are per-respondent averages.
Figure 8: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 9: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 10: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 11: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 12: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 13: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 14: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 15: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Model
Prompt
EM
Acc@10%
MAE
RMSE
Cov.
∣S∣
PP
InternVL2.5-8B
P1
14.0
38.2
15.8
28.5
97.9
16
9.4
P2
10.1
27.2
7.3
11.7
100.0
29
16.0
P3
19.0
44.8
6.6
13.8
96.5
18
9.9
LLaVA-Video-7B
P1
26.8
56.3
3.7
6.1
98.9
16
8.0
P2
21.7
44.3
4.3
6.5
99.0
20
11.4
P3
18.7
33.6
5.1
8.0
87.4
18
12.1
Appendix
Table 5: Prompt robustness. All nine combinations are evaluated on the full 2,600 videos under identical conditions. EM, Acc@10% and Coverage are in percent; MAE and RMSE in object-count units.
Figure 16: Exact Match by ground-truth count for three models under three question formulations (P1–P3, Appendix E ; P1 is the prompt used throughout the paper). Rows are grouped by model, prompts in order within each group. Aggregate accuracy differs across rows, but every row alternates between a small number of high-accuracy cardinalities and neighboring values at or near zero. Eight cardinalities ( 21 , 22 , 29 , 31 , 38 , 39 , 41 , 42 ) are at exactly 0% in all nine rows.
Shape
Color
Model
Cube
Sphere
Red
Green
Blue
Yellow
Purple
AdaReTaKe-7B
14
16
15
14
15
16
15
AdaReTaKe-72B
17
16
16
15
17
18
17
InternVL2.5-8B
15
13
15
13
15
14
13
InternVL2.5-78B
19
13
21
9
15
21
13
LLaVA-Video-7B
25
29
28
26
27
29
25
Appendix
Table 6: Exact Match (%) by object class, over the full 2,600 videos. Shape columns aggregate 1,300 videos each, colour columns 520 each.
Figure 17: Exact Match against ground-truth count, by object class, for the sixteen models. Each block shows one model, with rows for the two shapes and the five colors; columns are the 26 cardinalities of the benchmark. Color rows aggregate 20 videos per cell and shape rows 50 , so differences of five to ten points between rows at a single cardinality are within sampling noise. Within a model the seven rows describe the same profile: the cardinalities a model reaches, and those at which it collapses, do not depend on the class it is asked to count.
Large vision-language models (VLMs) can recognize what happens in video but fail to count how many times. We introduce PushupBench, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score ∼6%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in lmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)
Final-answer video QA can show whether a model predicts the right number, but not which instances it counted, when the supporting evidence occurs, or why it failed. We diagnose long-video quantitative reasoning in multimodal large language models (MLLMs) through three coupled abilities: enumerating query-relevant instances, temporally grounding supporting evidence, and aggregating the evidence into counts. To support this analysis, we build EC-Bench, an evidence-annotated evaluation suite with 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries across six reasoning categories, and human-verified evidence spans. We evaluate 22 open-source and proprietary MLLMs using timestamped visual frames and transcripts. The best average scores reach only 29.98% Enumeration F1 and 23.74% Counting accuracy, compared with human performance of 78.57% and 82.97%, respectively. Our analyses show that counting errors are rarely isolated arithmetic mistakes: Enumeration F1 is strongly associated with Counting accuracy, temporal grounding quality is associated with lower counting error, and Counting accuracy drops as supporting evidence becomes more distributed. These findings recast long-video counting as evidence retrieval, temporal grounding, deduplication, and aggregation across the video, rather than simple numerical prediction.
Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The dissociation holds on both real and synthetic videos and is not explained by clip duration, position, or scene content. These results indicate that models encode repetition structure but do not reliably report it in their outputs. Project page: https://arenaa.github.io/RepProbe-webpage/
Mahsa Khoshnoodi, Sarah Adel Bargal
Department of Computer Science, Georgetown University