Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.
Figures & tables
Figure 1: Example frames from CountVideoBench illustrating coverage across cardinality (top) and object categories (bottom). Camera motion and isolated object classes isolate counting from recognition and object motion while preserving viewpoint variation.
Figure 2: Ground-truth count distributions in three video benchmarks with counting questions: VideoCount, CG-AV-Counting, and VCBench. Probability mass concentrates on small counts, with long tails of rare large counts carrying negligible statistical weight.
Benchmark
Range
Distinct counts
Med [Q1–Q3]
Top-3 share
MVBench [ 13 ]
0–9
10
3 [2–3]
66.5%
NExT-QA [ 22 ]
1–15
13
2 [2–4]
72.5%
TGIF-QA [ 11 ]
0–10
10
2 [0–4]
67.9%
Video-MME [ 8 ]
0–600000
25
4 [2–6]
53.5%
VideoCount [ 1 ]
1–1203
22
1 [1–2]
93.1%
CG-AV-Counting [ 16 ]
1–76
31
4 [3–7]
49.6%
Table 1: Distribution characteristics of the ground-truth count variable in video benchmarks with counting questions. CountVideoBench is reported in the last row.
Model
EM
Acc@10%
MAE
RMSE
Coverage
∣S∣
PP
AdaReTaKe-7B
15.1
34.4
4.5
6.8
91.1
28
11.3
AdaReTaKe-72B
16.7
39.1
4.2
6.2
97.5
27
12.3
InternVL2.5-8B
14.0
38.2
15.8
28.5
97.9
16
9.4
InternVL2.5-78B
15.8
23.4
4.5
8.5
49.7
17
5.9
LLaVA-Video-7B
26.8
56.3
3.7
6.1
98.9
16
8.0
LLaVA-Video-72B
14.1
27.8
2.5
3.4
69.1
21
10.5
Table 2: Results on CountVideoBench. EM, Acc@10% and Coverage in percent; MAE and RMSE in object-count units (lower is better). Bold marks the best value per column; for MAE/RMSE it ignores † , whose Coverage is 1.2% . Oracle: exact prediction everywhere.
Figure 3: Model coverage by ground-truth count. Coverage is the proportion of videos for which the model produces a numerical answer.
Figure 4: Exact Match accuracy by ground-truth count and model. Each cell reports the percentage of videos for which the model’s predicted count equals the ground-truth exactly.
Figure 5: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos. (a) LLaVA-Video-7B. (b) GPT-5.4. (c) Gemini-3.1-pro.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Ground-truth count distributions in the representative video benchmarks with counting questions. Each panel shows the number of videos at each ground-truth count.
Table 3: Generation parameters. U(a,b) denotes a uniform draw on [a,b] . The polar angle is drawn subject to the placement constraint described below, which induces the effective distribution reported in the last row of the camera block.
Figure 7: Exact Match by ground-truth cardinality on the 104 -video human-study subset, for all evaluated models and for the twenty human participants (bottom row). Each model cell aggregates four videos and therefore takes values in {0,25,50,75,100}% ; each human cell aggregates 80 responses. The human row declines steadily from 99% at the smallest counts to 74% at n=50 , without the isolated peaks and adjacent collapses visible across the model rows.
Respondent
EM
Acc@10%
Coverage
∣S∣
PP
AdaReTaKe-7B
14.4
30.8
90.4
14.0
9.7
AdaReTaKe-72B
16.3
40.4
96.2
18.0
13.3
InternVL2.5-8B
13.5
36.5
97.1
12.0
9.6
InternVL2.5-78B
13.5
19.2
44.2
10.0
5.0
LLaVA-Video-7B
26.9
56.7
99.0
12.0
7.6
LLaVA-Video-72B
15.4
31.7
67.3
17.0
9.8
Appendix
Table 4: Human baseline and model performance on the 104 -video subset. Human rows report the mean over 20 participants; the majority-vote row aggregates the modal answer per video. Support size and perplexity are per-respondent averages.
Figure 8: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 9: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 10: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 11: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 12: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 13: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 14: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Figure 15: Distributions of predicted counts on CountVideoBench. Each bar reports the number of times a given integer value is predicted by the model across the 2,600 videos.
Model
Prompt
EM
Acc@10%
MAE
RMSE
Cov.
∣S∣
PP
InternVL2.5-8B
P1
14.0
38.2
15.8
28.5
97.9
16
9.4
P2
10.1
27.2
7.3
11.7
100.0
29
16.0
P3
19.0
44.8
6.6
13.8
96.5
18
9.9
LLaVA-Video-7B
P1
26.8
56.3
3.7
6.1
98.9
16
8.0
P2
21.7
44.3
4.3
6.5
99.0
20
11.4
P3
18.7
33.6
5.1
8.0
87.4
18
12.1
Appendix
Table 5: Prompt robustness. All nine combinations are evaluated on the full 2,600 videos under identical conditions. EM, Acc@10% and Coverage are in percent; MAE and RMSE in object-count units.
Figure 16: Exact Match by ground-truth count for three models under three question formulations (P1–P3, Appendix E ; P1 is the prompt used throughout the paper). Rows are grouped by model, prompts in order within each group. Aggregate accuracy differs across rows, but every row alternates between a small number of high-accuracy cardinalities and neighboring values at or near zero. Eight cardinalities ( 21 , 22 , 29 , 31 , 38 , 39 , 41 , 42 ) are at exactly 0% in all nine rows.
Shape
Color
Model
Cube
Sphere
Red
Green
Blue
Yellow
Purple
AdaReTaKe-7B
14
16
15
14
15
16
15
AdaReTaKe-72B
17
16
16
15
17
18
17
InternVL2.5-8B
15
13
15
13
15
14
13
InternVL2.5-78B
19
13
21
9
15
21
13
LLaVA-Video-7B
25
29
28
26
27
29
25
Appendix
Table 6: Exact Match (%) by object class, over the full 2,600 videos. Shape columns aggregate 1,300 videos each, colour columns 520 each.
Figure 17: Exact Match against ground-truth count, by object class, for the sixteen models. Each block shows one model, with rows for the two shapes and the five colors; columns are the 26 cardinalities of the benchmark. Color rows aggregate 20 videos per cell and shape rows 50 , so differences of five to ten points between rows at a single cardinality are within sampling noise. Within a model the seven rows describe the same profile: the cardinalities a model reaches, and those at which it collapses, do not depend on the class it is asked to count.