RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

Mahsa Khoshnoodi; Sarah Adel Bargal

cs.CVOct 13, 2026

RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

Authors: Mahsa Khoshnoodi, Sarah Adel Bargal

Organizations: Department of Computer Science, Georgetown University

Abstract

Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The dissociation holds on both real and synthetic videos and is not explained by clip duration, position, or scene content. These results indicate that models encode repetition structure but do not reliably report it in their outputs. Project page: https://arenaa.github.io/RepProbe-webpage/

Figures & tables

Figure 1: RepProbe-Real construction. Rest-pose timestamps bound half-open repetition intervals. A clip consists of $k$ consecutive complete repetitions from one source. Left bars mark example windows: $k{=}1$ (pink, Rep 1), two overlapping $k{=}2$ placements (teal, Reps 1–2; blue, Reps 2–3), and $k{=}3$ (orange, all three). Start, middle, and end placements vary clip position while the target count stays fixed.

Figure 2: RepProbe statistics. (a) Clips per repetition count $k$ . RepProbe-Real (blue, stacked by clip position: start, middle, end) is skewed toward small counts because fewer source videos contain many repetitions; RepProbe-Synthetic (orange) is balanced at $500$ clips per count. (b) Clip duration versus $k$ . Real clip duration grows with $k$ (line: median; shaded band: spread across clips), whereas every synthetic clip lasts $4$ s regardless of $k$ .

Figure 3: RepProbe-Synthetic families. Each panel shows five frames spanning one period of one of the ten families. In each family, a single attribute (visibility, size, hue, position, angle, geometry, or occlusion) changes periodically and completes $k$ periods per clip, while everything else stays fixed. Textured flash and Multi object flash add stress conditions: a cluttered background, and flashes distributed across several objects that take turns.

RepProbe-Real
	Overall				OBO (%) by band $\uparrow$
Method	MAE ${}_{n}\downarrow$	MAE $\downarrow$	Exact $\uparrow$	OBO $\uparrow$	Few	Mid	Many
Specialist
ESCounts	0.19	1.07	41.9	77.5	87.5	71.0	65.6
RepNet	0.32	2.20	21.7	53.0	79.1	36.4	21.3
Generalist

Table 1: Repetition counting on RepProbe-Real and RepProbe-Synthetic (generalists only),

k=1

–

20

. Predictions are clipped to

[0,40]

. Bands: Few

k{=}1

–

6

, Mid

k{=}7

–

13

, Many

k{=}14

–

20

. Bold : best generalist in each column. The shaded Many band highlights the high-count collapse on RepProbe-Real.

RepProbe-Real
		All		Few		Mid		Many
Model	Readout	MAE $\downarrow$	OBO $\uparrow$	MAE $\downarrow$	OBO $\uparrow$	MAE $\downarrow$	OBO $\uparrow$	MAE $\downarrow$	OBO $\uparrow$
Cosmos-8B	Model	3.91	37.9	1.91	56.0	3.20	40.7	6.35	19.4
	Probe	2.49	50.3	1.62	68.0	2.99	39.7	2.75	45.6
Qwen3-VL-8B	Model	7.88	24.2	2.45	58.9	6.97	14.0	13.43	4.7
	Probe	3.05	44.7	2.27	58.9	3.23	37.9	3.54	39.4

Table 2: Two ways of reading the count from the same final-layer representation, on the balanced

10

K subset of RepProbe-Real and on RepProbe-Synthetic. The linear count probe (trained on 1k clips) is more accurate than the model’s own answer on both; on RepProbe-Real, the gap is largest at high

k

Figure 4: The count is recoverable by a probe but not by the model’s answer (Cosmos-Reason2, RepProbe-Real). (a) The answer’s probability on the true $k$ falls to chance by $k\!\approx\!11$ ; (b) its argmax collapses onto a few small numbers. (c) The probe keeps far more mass on the true $k$ ; (d) its argmax tracks $k$ across the full range.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	FPC	$k{=}5$	$k{=}10$	$k{=}15$	$k{=}20$
Qwen3-VL-8B	8	2.93	4.09	7.98	7.86
	4	3.50	4.83	6.82	14.18
	2	2.96	5.16	13.64	21.42
	1	3.90	6.13	9.25	14.20
Cosmos-Reason2-8B	8	2.47	2.13	4.46	4.57
	4	2.19	3.14	4.17	6.16

Appendix

Table A.1: Per-cell MAE on the hold-

k

\times

FPC grid (RepProbe-Real),

100

clips per cell.

	Exact $\uparrow$			OBO $\uparrow$
Method	start	middle	end	start	middle	end
ESCounts	$37.6$	$36.9$	$37.0$	$72.5$	$72.3$	$70.7$
RepNet	$16.5$	$17.1$	$16.7$	$42.3$	$43.0$	$42.9$
Cosmos probe (L35)	$24.1$	$26.0$	$24.8$	$49.2$	$51.7$	$49.8$
Qwen probe (L35)	$21.4$	$23.5$	$21.3$	$43.7$	$46.5$	$44.2$
Cosmos-Reason2-8B	$14.2$	$12.3$	$12.4$	$38.1$	$38.1$	$37.3$

Appendix

Table A.2: Counting accuracy by clip position on the balanced

10

k RepProbe-Real set (

\approx 3{,}330

clips in each window). Parse failures scored as

0

, then clipped to

[0,40]

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 167 papers

PushupBench: Your VLM is not good at counting pushups
Apr 25, 2026Shengzhi Li, Jiarun Chen, Karun Sharma +2Multimodal Understanding Video-Language Model Evaluation
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Aug 6, 2026Sarvesh Baskar, Zikui Cai, Shayan Shabihi +5Video-Language Model Evaluation Temporal Video Understanding
University of Maryland, College Park, USA · Scale AI, USA
CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models
Oct 20, 2026Marc Chapus, Carlos Crispim-Junior, Véronique Eglin +1Video-Language Model Evaluation
INSA Lyon, Université Lumière Lyon 2, CNRS, Université Claude Bernard Lyon 1, LIRIS, UMR5205, 69621 Villeurbanne, France

Apr 25, 2026J/K move · Enter open · S save

PushupBench: Your VLM is not good at counting pushups

Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su+1

Multimodal Understanding Video-Language Model Evaluation Large Vision-Language Models VLM Evaluation Temporal Video Understanding Video Foundation Models+1

Open paper AskPDF arXiv

Large vision-language models (VLMs) can recognize what happens in video but fail to count how many times. We introduce PushupBench, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score

\sim

6%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in lmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)

Abstract

Video-Language Models

Explore similar work

PushupBench: Your VLM is not good at counting pushups

The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models