Final-answer video QA can show whether a model predicts the right number, but not which instances it counted, when the supporting evidence occurs, or why it failed. We diagnose long-video quantitative reasoning in multimodal large language models (MLLMs) through three coupled abilities: enumerating query-relevant instances, temporally grounding supporting evidence, and aggregating the evidence into counts. To support this analysis, we build EC-Bench, an evidence-annotated evaluation suite with 152 untrimmed videos longer than 30 minutes, 1,699 open-ended queries across six reasoning categories, and human-verified evidence spans. We evaluate 22 open-source and proprietary MLLMs using timestamped visual frames and transcripts. The best average scores reach only 29.98% Enumeration F1 and 23.74% Counting accuracy, compared with human performance of 78.57% and 82.97%, respectively. Our analyses show that counting errors are rarely isolated arithmetic mistakes: Enumeration F1 is strongly associated with Counting accuracy, temporal grounding quality is associated with lower counting error, and Counting accuracy drops as supporting evidence becomes more distributed. These findings recast long-video counting as evidence retrieval, temporal grounding, deduplication, and aggregation across the video, rather than simple numerical prediction.
Figures & tables
Figure 1: Example tasks from EC-Bench . We diagnose long-video quantitative reasoning through three coupled outputs: enumeration of query-relevant instances, temporal grounding of supporting evidence spans, and counting of the identified instances.
Dataset
#QA
#Tasks
Ultra-long ( ≥ 30min)
Multi- hop
Evidence Span
Audio
Enum.
Answer Type
Long Video Understanding Benchmarks
EgoSchema [ 31 ]
5,063
1
✗
✗
✗
✗
✗
MCQ
ActivityNet - QA [ 55 ]
58,000
3
✗
✗
✗
✗
✗
MCQ
MultiHop - EgoQA [ 6 ]
11,707
6
✗
✓
✓
✗
✗
MCQ
HourVideo [ 4 ]
12,976
4
✓
✗
✗
✗
✗
MCQ
LVBench [ 43 ]
1,549
6
✓
✗
✗
✗
✗
MCQ
Table 1: Comparison of representative datasets for long-video understanding and counting tasks. EC-Bench uniquely supports ultra-long videos, multi-hop reasoning, explicit evidence spans, and open-ended enumeration.
Figure 2: Reasoning categories in EC-Bench. Examples of six quantitative reasoning types evaluated in the benchmark.
Figure 3: Dataset statistics. 3(a) Video duration distribution. 3(b) Per-query answer-size distributions for the Enumeration (upper) and Counting (lower) tasks. 3(c) Evidence span duration distribution and the cumulative distribution of normalized temporal spread between the first and last supporting span.
Enumeration
Counting
Model
Parallel
Causal
Speech
Appear.
Spatial
Cond.
Avg.
Parallel
Causal
Speech
Appear.
Spatial
Cond.
Avg.
Open-source Models
LongLLaVA [ 45 ]
12.16
21.58
12.50
13.99
7.86
5.04
12.19
4.17
11.59
5.00
5.67
6.38
11.68
7.37
LongVA DPO 7B [ 59 ]
14.77
10.79
15.28
24.48
14.18
7.91
14.62
9.03
10.87
5.00
3.55
11.35
13.87
8.92
VideoLLaMA3 2B [ 56 ]
19.05
2.88
15.28
20.14
8.51
4.32
11.83
13.19
18.12
10.00
8.51
5.67
16.06
11.89
VideoLLaMA3 7B [ 56 ]
23.13
5.76
15.71
24.26
11.28
5.97
14.48
14.58
18.12
7.14
14.18
11.35
11.35
12.41
Table 2: Performance of open-source and proprietary MLLMs on EC-Bench. Enumeration F1 and Counting accuracy (%) are reported for six reasoning categories and their overall averages. Best and second-best results in each column are highlighted with darker and lighter shading, respectively.
Figure 6Figure 7
Figure 7: Video statistics of EC-Bench. Left: distribution of video durations (all videos exceed 30 minutes; median 47 minutes). Right: representative examples of video genres in the dataset.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Enumeration
Counting
Model
Academic
Cartoon
Documentary
Live
News
Self-media
Sports
TV
Average
Academic
Cartoon
Documentary
Live
News
Self-media
Sports
TV
Average
Open-source Models
LongLLaVA [ 45 ]
10.00
16.35
12.24
16.67
13.56
8.21
13.79
6.45
11.96
9.09
7.62
6.25
6.14
8.47
9.09
4.70
10.34
7.37
LongVA DPO 7B [ 59 ]
5.00
18.27
17.01
9.17
20.34
12.78
16.38
7.53
14.62
0.00
9.52
4.86
11.40
13.56
6.82
8.72
12.93
8.92
VideoLLaMA3 2B [ 56 ]
25.00
16.99
20.75
14.62
20.69
13.11
16.38
11.83
12.06
9.09
9.52
9.72
13.16
15.25
11.36
11.41
15.52
11.89
VideoLLaMA3 7B [ 56 ]
20.00
13.26
19.05
13.89
20.69
13.11
14.66
12.90
14.96
13.64
11.43
10.42
13.16
23.73
9.85
12.08
16.38
12.96
Appendix
Table 5: Performance of open-source and proprietary MLLMs on EC-Bench across video genres. Enumeration F1 and Counting accuracy (%) are reported for eight video genres and their overall averages. Best and second-best results in each column are highlighted with darker and lighter shading, respectively.
Model
#Frames
Open-source Models
LongLLaVA
128
LongVA DPO 7B
128
VideoLLaMA3 2B
128
VideoLLaMA3 7B
128
LLaVA-Next-Video 7B
64
Appendix
Table 6: Per-model input frame budget.
Failure type
Description
Representative example
Missed instances
The model captures the coarse scene but fails to identify all relevant people, objects, or events, particularly under sparse or crowded evidence settings.
For “What is the maximum number of people trapped in capsules?”, the ground truth is 87, while GPT-4o predicts 3.
Weak temporal grounding
The answer is semantically plausible, but the predicted evidence spans fail to localize all supporting events.
Retrieved intervals are missing, overly broad, or misaligned with the actual evidence.
Transcript grounding failure
The model relies on topic-level transcript or audio cues instead of counting concrete visual occurrences.
For “How many times is AI-generated content shown?”, the model predicts 5 instead of the correct count of 26.
Over-abstraction
The model outputs coarse semantic categories instead of enumerating the required concrete instances.
For “List the technical terms for Changi Airport T4”, specific terms are replaced by broad labels such as “self-service systems”.
Prior-knowledge hallucination
The model injects plausible world knowledge unsupported by video evidence.
For “List the items under ‘GENERATIVE AI’ at 6:54”, the model hallucinates NVIDIA-related terms.
Deduplication failure
The model inconsistently merges or splits repeated appearances across long temporal contexts.
Boundary-crossing or repeated events are counted inconsistently across segments.
Appendix
Table 7: Failure taxonomy on EC-Bench. Representative GPT-4o failures grouped by the stage of long-video quantitative reasoning where the error occurs.
Figure 8: Enumeration and counting accuracy across video genres.
School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · University of Electronic Science and Technology of China, Chengdu, Sichuan, China