Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation

Ankit Khedia; Astitva Jaiswal; Chaitanya Prasad; Anisha Raju; Aangeeras

cs.CVOct 28, 2026

Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation

Authors: Ankit Khedia, Astitva Jaiswal, Chaitanya Prasad, Anisha Raju, Aangeeras

Organizations: Deccan AI

Abstract

Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video description, and find that they do not. Six frontier models captioned 196 short clips. Trained annotators post-edited every caption against a failure taxonomy fixed before annotation, once as a whole-clip overview and once window by window, and a quality-control reviewer checked each annotation, yielding 1,144 overview verdicts and 4,672 window verdicts. Reference-overlap metrics do not separate the captions reviewers accepted from those they rejected: BLEU-4 reaches an AUC of 0.510 and a CIDEr-style tf-idf cosine 0.528, and restricting references to reviewer corrections leaves BLEU-4 at 0.479 (95% CI 0.431-0.529), indistinguishable from chance. Review granularity changes what is detected. The same annotators record camera and shot errors on 2.7% of captions when reading the overview and on 12.8% when reading window by window, a 4.7x gap that holds across every robustness check we ran, on the mode most likely to need a near-total rewrite. Correct-overview rates span 54.1-72.7%, but only 4 of 15 pairwise gaps survive Bonferroni correction (5 after adjusting for reviewer and clip effects), and accuracy and derived severity order the models differently. Models with statistically indistinguishable accuracy have sharply different failure profiles, which no single scalar shows. We argue that caption evaluation should record what was wrong rather than only how often, and we report the taxonomy, the per-model profiles, the corpus properties that predict difficulty, and the code that regenerates every number in the paper.

Figures & tables

References	Score	Accepted	Rejected	AUC	95% CI	Matched
Approved ( $n$ =751/392)	BLEU-4	0.358	0.355	0.510	[0.473, 0.546]	0.520
	CIDEr-style cosine	1.834	1.782	0.528	[0.488, 0.567]	0.526
Gold ( $n$ =579/352)	BLEU-4	0.206	0.210	0.479	[0.431, 0.529]	0.500
	CIDEr-style cosine	1.768	1.729	0.518	[0.475, 0.562]	0.524
Controls	Overview length (shorter $\to$ accept)	85w	86w	0.524	[0.486, 0.562]	–
	Video-LLM judge, fewer defects †			0.558	[0.519, 0.597]	–

Table 1: Agreement between automatic scores and the human accept/reject verdict on the overview, over all captions with a final verdict. AUC 0.5 means the score carries no information; intervals are 95% bootstrap over clips. Matched : accepted and rejected candidates on the same clip scored against an identical reference set. † Gemini 3.1 Pro judge, 648 accepted / 301 rejected captions from five models (Qwen3.8-27B was not judged). The judge counts defects in the overview and timeline together, and Gemini 3.1 Pro is itself a candidate.

Model	$n$	Any correction	$\geq$ 3 words	$\geq$ 6 words	$\geq$ 16 words
Qwen3.8-Max	193	27.5	22.3	17.6	7.3
Muse Spark 1.2	187	28.3	24.1	19.3	7.5
Kimi K3	194	28.9	24.2	20.6	9.3
Seed 2.1 Turbo	190	32.1	25.8	21.6	12.1
Gemini 3.1 Pro	188	41.5	35.1	27.7	16.0
Qwen3.8-27B	192	47.4	38.0	27.6	12.0

Table 2: Overview intervention rate at increasing evidence thresholds, over all captions with a final verdict. A caption counts as failed only if the correction changed at least that many words (word-level diff, replacements counted at the longer side). Appendix C gives the same table on the 172-clip paired subset.

Figure 1: Overview intervention rate recomputed while progressively discarding small corrections (Table 2 ). The four frontier models in grey hold their order throughout. Gemini 3.1 Pro and Qwen3.8-27B tie at the six-word bar and swap places beyond it: much of Qwen3.8-27B’s excess comes from small corrections.

Failure mode	Overview pass, % ( $k$ )	Window pass, % ( $k$ )	Ratio
Camera / shot	2.7 (31)	12.8 (147)	4.7 $\times$
Temporal / order	0.3 (4)	1.0 (11)	2.8 $\times$
Unsupported inference	2.4 (27)	4.9 (56)	2.1 $\times$
Important omission	11.5 (131)	17.9 (205)	1.6 $\times$
Factual error	23.3 (267)	33.9 (388)	1.5 $\times$

Table 3: Share of captions recorded as carrying each mode, by review granularity, over all 1,144 captions. Overview pass: modes tagged on the rejected overview. Window pass: modes tagged on any rejected window of the same caption.

k

is the number of captions.

Figure 2: The data of Table 3 . Camera and shot is the outlier in both position and ratio.

Variant	Camera / shot ratio
Primary: QC-pass overview, annotator-pass windows, 1,144 captions	4.7 $\times$
Annotator pass at both levels (same person, same sitting)	4.9 $\times$
QC pass at both levels	4.7 $\times$
Camera-over-factual precedence enforced on every unit	4.7 $\times$
172-clip paired subset	5.5 $\times$
Excluding the 12 manipulated clips	4.5 $\times$

Table 4: Ratio of window-pass to overview-pass incidence for camera and shot errors under alternative analysis choices.

Figure 3: Share of overviews accepted whole that contain at least one window the same annotator rejected, with 95% Clopper–Pearson intervals and counts, all captions. The dashed line is the pooled rate.

Figure 4: Rejected windows per caption among the 250 captions with exactly four windows, against what independent windows at each model’s own rejection rate would give. The middle is hollow and both tails are heavy.

Figure 5: Correct-overview rate on the 172-clip paired subset with 95% Clopper–Pearson intervals (Table 5 ). The four leading models overlap almost entirely, and so do the two trailing ones.

	Correct overview		Correct window		Critical (derived)
Model	%	95% CI	%	95% CI	%	95% CI
Qwen3.8-Max	72.7	[65.4, 79.2]	80.7	[76.7, 84.5]	5.8	[2.8, 10.4]
Muse Spark 1.2	72.1	[64.8, 78.7]	68.4	[62.2, 74.1]	16.3	[11.1, 22.7]
Kimi K3	71.5	[64.1, 78.1]	73.7	[68.1, 78.8]	9.9	[5.9, 15.4]
Seed 2.1 Turbo	69.8	[62.3, 76.5]	71.6	[65.6, 77.3]	10.5	[6.3, 16.0]
Gemini 3.1 Pro	58.7	[51.0, 66.2]	65.0	[58.8, 71.1]	8.7	[5.0, 14.0]

Table 5: Primary measures on the 172-clip paired subset (

n=172

captions per model). Correct overview: accepted as written (Clopper–Pearson interval). Correct window: share of windows accepted (bootstrap over clips). Critical: share of captions with at least one rejected window whose corrected text has character-level similarity below 0.55 to the original (Clopper–Pearson). Critical is derived from edit size, not recorded by reviewers (§ 10 ).

Mode	Definition given to annotators Decision test (our summary)	Captions (%)	Near-total (%)	Per 100
Factual error	An observable fact is stated incorrectly: wrong person, object, action, attribute, count, location or spatial relation. Can a reviewer point at a frame and say “not that”?	33.9	16.0	7.5
Important omission	A central event or piece of information needed to describe the content adequately is missing; optional background detail is not marked. Would a reader holding only the caption be misled?	17.9	5.5	1.6
Camera / shot	Camera motion, zoom, cuts, framing, occlusion or reflection is mistaken for something happening in the physical scene. Did the pixels move because the camera moved, or the world?	12.8	21.6	4.1
Segmentation	Windows merge distinct events, miss a shot change, are placed on a grid rather than on content, or are timestamped materially away from the event. Are the timestamps where the events actually are?	13.4	–	–
Unsupported inference	Intent, emotion, identity, cause or relationship is asserted where the video does not support it, including identification from outside knowledge. Could any viewer verify this from the footage alone?	4.9	19.0	1.2
Temporal / order	Event order, timing, duration, repetition or simultaneity is materially misrepresented. Is what happened right but when wrong?	1.0	0.0	0.0

Table 6: The failure-mode taxonomy, frozen before annotation, with each mode’s incidence and severity pooled across the six models (window detector, all 1,144 captions). Definitions are those shown to annotators; the decision tests are our one-line summaries and were not part of the rubric. Captions : share of captions carrying the mode at least once. Near-total : share of that mode’s window repairs rewritten to character-level similarity below 0.55. Per 100 : captions per hundred with at least one near-total window repair carrying the mode. Segmentation is judged once per caption and has no window-level severity. All figures are floors, since modes were recorded only on rejected units.

Figure 6: Gemini 3.1 Pro against Qwen3.8-27B, two models 4.6 points apart on accuracy that cannot be statistically separated: share of captions carrying each mode at least once, window detector, all captions. Qwen3.8-27B is far worse on factual error and far better on segmentation; Gemini 3.1 Pro is the reverse on both and also worse on omission, camera and temporal order.

Model	$n$	Factual	Omission	Camera	Inference	Temporal	Segment.
Qwen3.8-Max	193	24.9	16.6	10.9	4.7	0.5	12.4
Muse Spark 1.2	187	36.4	14.4	13.9	4.3	0.5	14.4
Kimi K3	194	33.5	13.9	11.3	5.2	1.0	10.8
Seed 2.1 Turbo	190	32.1	19.5	13.7	6.8	1.6	17.9
Gemini 3.1 Pro	188	30.3	23.9	14.9	4.8	2.1	18.1
Qwen3.8-27B	192	46.4	19.3	12.5	3.6	0.0	6.8

Table 7: Share of each model’s captions carrying each mode at least once, detected window by window, over all 1,144 captions with a final verdict. Modes are not exclusive: 36% of rejected captions carry two or more under this detector (16% under the overview detector), so rows do not sum to a rejection rate. Appendix C gives the paired-subset version.

Figure 7: Failure-mode incidence by model with 95% Clopper–Pearson intervals, one panel per mode, models ordered by paired accuracy, the highest bar in each panel in red. Camera is flat across the field; factual error and segmentation are not.

Model	Overview	Scene text	Shot text	Segmentation	Rewritten
Qwen3.8-Max	72.5	83.7	96.1	87.6	2.78
Muse Spark 1.2	71.7	74.0	91.6	85.6	2.94
Kimi K3	71.1	77.9	94.4	89.2	3.67
Seed 2.1 Turbo	67.9	77.8	91.7	82.1	4.57
Gemini 3.1 Pro	58.5	70.2	88.5	81.9	10.62
Qwen3.8-27B	52.6	67.5	89.4	93.2	5.51

Table 8: Share of each component left untouched by the annotator, over all 1,144 captions. Denominators differ by column: overview and segmentation are per caption, scene and shot text per window. Rewritten : share of all characters of overview and timeline text changed by the reviewer, with accepted text counted as unchanged (Appendix D ).

Model	Repairs	$\pm$ 5% band	Lengthen, $\pm$ 0%	Lengthen, $\pm$ 10%	change (%)	penalty (pts)
		Lengthen / shorten / same (%)			Median length	Handheld
Gemini 3.1 Pro	310	65 / 14 / 22	72	56	+16.7	+3.1
Seed 2.1 Turbo	247	57 / 13 / 30	75	47	+8.2	+2.7
Qwen3.8-27B	330	50 / 19 / 31	64	42	+5.6	+26.1
Qwen3.8-Max	160	50 / 18 / 32	67	42	+5.0	+4.0
Kimi K3	237	39 / 16 / 45	63	33	+2.4	+13.2

Table 9: Direction of window repairs (changed shot or scene fields of rejected windows), over all 1,144 captions, ordered by the share lengthened at

\pm

5%. Handheld penalty : intervention rate on handheld minus tripod-static footage, from Table 13 .

Model	Accuracy (%)	Two most frequent modes (%)	Handheld	Repairs L / S / =	Direction
Qwen3.8-Max	72.7	factual 24.9; omission 16.6	+4.0	50 / 18 / 32	mixed
Muse Spark 1.2	72.1	factual 36.4; segmentation = omission 14.4	+16.0	38 / 15 / 47	substitutive
Kimi K3	71.5	factual 33.5; omission 13.9	+13.2	39 / 16 / 45	substitutive
Seed 2.1 Turbo	69.8	factual 32.1; omission 19.5	+2.7	57 / 13 / 30	additive
Gemini 3.1 Pro	58.7	factual 30.3; omission 23.9	+3.1	65 / 14 / 22	additive
Qwen3.8-27B	54.1	factual 46.4; omission 19.3	+26.1	50 / 19 / 31	mixed

Table 10: Model profiles. Accuracy: correct-overview rate on the 172-clip paired subset. Modes: the two most frequent, window detector, all captions (Table 7 ). Handheld penalty: points lost on handheld against tripod-static footage (Table 13 ). Repairs: share of window repairs that lengthen, shorten or keep the length within

\pm

5% (Table 9 ). Direction: additive

\geq

55% lengthened, substitutive

<

45%, mixed otherwise.

Property	Correct-overview range across buckets	$\chi^{2}$	d.f.	$p$	$p$
		Caption level			Clip level
Content type	54.5 domestic lifestyle – 72.6 people activity	21.6	4	$<$ 0.001	0.001 ∗
Camera platform	56.2 vehicle mounted – 72.1 tripod static	15.2	5	0.010	0.015
Scene complexity	61.3 medium – 69.0 low	7.1	1	0.008	0.011
Shot scale	60.4 mixed – 80.0 extreme wide	10.6	4	0.032	0.039
Motion intensity	62.1 medium – 69.9 high	3.4	2	0.180	0.207

Table 11: Correct-overview rate pooled across the six models, by clip property (labels inferred by a language model; § 3 ). Buckets with fewer than 60 verdicts and buckets labelled unclear are omitted. Caption-level

\chi^{2}

treats captions as independent and overstates the evidence. The clip-level test (Kruskal–Wallis on per-clip correct shares) is the appropriate one; ∗ survives Bonferroni correction across the six properties.

Bucket	Correct (%)	Verdicts	Clips	Per model	Model spread (pts)
Content type
domestic lifestyle	54.5	233	40	36–40	31.7
urban vehicle	63.6	231	40	38–39	25.0
nature outdoor	66.9	236	40	38–40	30.0
product object	71.5	207	36	32–36	23.8
people activity	72.6	237	40	39–40	23.5

Table 12: Correct-overview rate by bucket for the two properties with the largest effects, pooled across the six models, all captions. Per model : range of captions per model in the bucket. Model spread : best minus worst model’s correct-overview rate in the bucket. Buckets with fewer than 60 verdicts and unclear labels are omitted.

Figure 8: Number of the six models failing each of the 172 paired clips, against a binomial at the pooled rate. The spread is wider than independence predicts, but the tails are thin: two clips defeated every model and 25 defeated none.

Model	Handheld	Tripod	$\Delta$ [95% CI]	Med./high	Low	$\Delta$ [95% CI]
	Camera platform			Scene complexity
Qwen3.8-27B	52.4	26.3	+26.1 [9.5, 42.9]	54.4	41.2	+13.3 [ $-$ 0.8, 27.6]
Muse Spark 1.2	30.0	14.0	+16.0 [0.9, 30.5]	37.1	20.4	+16.7 [3.8, 29.4]
Kimi K3	40.3	27.1	+13.2 [ $-$ 3.5, 29.6]	34.4	24.0	+10.4 [ $-$ 2.2, 22.9]
Qwen3.8-Max	28.1	24.1	+4.0 [ $-$ 11.2, 19.6]	28.1	26.9	+1.2 [ $-$ 11.6, 13.8]
Gemini 3.1 Pro	46.8	43.6	+3.1 [ $-$ 14.9, 21.4]	41.4	41.6	$-$ 0.2 [ $-$ 14.4, 14.0]

Table 13: Intervention rate (the complement of the correct-overview rate) on difficult against easier footage, per model, over all 1,144 captions.

\Delta

is the penalty in points, with a 95% bootstrap interval over clips. Complexity compares medium or high against low.

Figure 9: Penalty on difficult footage with 95% bootstrap intervals over clips. Blue marks intervals that exclude zero. Models are sorted by handheld penalty.

Figure 10: Omission rate against factual-error rate, one point per model, window detector, all captions. A precision–recall trade-off would place these on a descending line; the correlation is

r=-0.02

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Identifier	Route	Providers	Tokens/s video	Retried	Re-encoded	Invalid kept
Qwen3.8-Max	qwen/qwen3.8-max	OpenRouter	1	739	38	24	5
Muse Spark 1.2	meta/muse-spark-1.2	OpenRouter	1	283	0	0	0
Kimi K3	moonshotai/kimi-k3	OpenRouter	1	2596	55	0	0
Seed 2.1 Turbo	bytedance-seed/seed-2-1-turbo	OpenRouter	1	754	7	0	0
Gemini 3.1 Pro	gemini-3.1-pro-preview	Gemini API	1	211	0	0	0
Qwen3.8-27B	qwen/qwen3.8-27b	OpenRouter	10	775	22	35	0

Appendix

Table 14: Generation details. Providers : distinct serving providers recorded in the responses. Tokens/s video : median input tokens per second of video on first attempts, a proxy for each provider’s frame sampling. Retried : clips re-requested with the schema validator’s message appended. Re-encoded : clips sent as a different file from the one the other models received (the Qwen3.8 endpoints’ 6 MB limit). Invalid kept : outputs reviewed despite never passing validation. All runs used temperature 0 and muted video, between 16 and 27 August 2026.

	Raw			Reviewer- and clip-adjusted
Pair (row $-$ column)	Gap	Bonferroni CI	Separates	Gap	Bonferroni CI	Separates
3.8-Max – Muse	$-$ 0.6	[ $-$ 15.1, 13.4]		+0.9	[ $-$ 13.3, 15.2]
3.8-Max – Kimi	$-$ 1.2	[ $-$ 14.5, 12.2]		$-$ 2.6	[ $-$ 15.1, 10.3]
3.8-Max – Seed	$-$ 2.9	[ $-$ 16.3, 11.0]		$-$ 1.2	[ $-$ 15.0, 12.8]
3.8-Max – Gemini	$-$ 14.0	[ $-$ 28.5, 1.2]		$-$ 13.2	[ $-$ 27.8, 2.3]
3.8-Max – 3.8-27B	$-$ 18.6	[ $-$ 32.6, $-$ 4.7]	yes	$-$ 20.7	[ $-$ 35.6, $-$ 6.6]	yes

Appendix

Table 15: Pairwise gaps in overview intervention rate (row model minus column model, points) on the 172-clip paired subset. Raw: paired bootstrap over clips, 20,000 resamples. Adjusted: additive model with annotator and clip effects refitted in each of 4,000 clip-bootstrap resamples. Intervals are Bonferroni-corrected for 15 comparisons.

Model	$n$	Any correction	$\geq$ 3 words	$\geq$ 6 words	$\geq$ 16 words
Qwen3.8-Max	172	27.3	22.1	16.9	7.0
Muse Spark 1.2	172	27.9	23.8	19.2	7.6
Kimi K3	172	28.5	23.3	19.8	7.6
Seed 2.1 Turbo	172	30.2	24.4	20.3	11.0
Gemini 3.1 Pro	172	41.3	35.5	27.9	15.1
Qwen3.8-27B	172	45.9	37.8	26.7	11.0

Appendix

Table 16: Table 2 on the 172-clip paired subset.

Model	$n$	Factual	Omission	Camera	Inference	Temporal	Segment.
Qwen3.8-Max	172	25.0	16.9	9.9	4.7	0.6	13.4
Muse Spark 1.2	172	36.6	15.1	13.4	4.7	0.6	15.1
Kimi K3	172	33.7	12.2	12.2	4.7	1.2	10.5
Seed 2.1 Turbo	172	32.6	19.2	13.4	5.8	1.7	18.0
Gemini 3.1 Pro	172	30.8	23.8	14.5	3.5	1.7	16.9
Qwen3.8-27B	172	44.2	19.8	12.8	4.1	0.0	7.6

Appendix

Table 17: Table 7 on the 172-clip paired subset.

Video Caption Evaluation

See all 15 papers

Sep 9, 2026

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.

Zizhen Wang, Bo Feng, Zhengfeng Lai +5

Apple

Video Caption Evaluation Video Captioning

Ask PDF

Aug 5, 2026

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.

Mukhtiar Ali, Harsh Dubey, Sugam Mishra +1

South Dakota State University Brookings, SD, USA

Video-Language Model Evaluation Video Captioning Video Caption Evaluation

Ask PDF

Jul 23, 2026

ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning

Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected objects to curb hallucination, but apply only a single, fixed correction pass without prioritizing which objects matter most, leaving semantically significant content omitted. We propose a prominence-aware, iterative post-hoc rectification framework that overcomes both limitations without modifying the underlying captioning model's parameters: a lightweight scoring mechanism ranks detected objects by spatial saliency, temporal persistence, and relational dynamics, and an iterative, prompt-driven refinement loop uses this ranking to progressively inject missing yet contextually relevant objects into the caption over multiple rounds. We validate the framework on MSVD and MSR-VTT using object-grounded automatic metrics, a 110-participant human study, and qualitative comparison against ChatGPT and Gemini; in human evaluation, the framework raises perceived completeness by up to 48% and reduces hallucination by up to 45% relative to a strong pretrained captioning baseline, all without retraining or reference captions. These results position prominence-guided iterative rectification as a lightweight, scalable, and model-agnostic route to more complete and trustworthy video captioning, with direct relevance to accessibility, retrieval, and other multimedia understanding applications.

Debjyoti Das Adhikary, Aritra Hazra, Partha Pratim Chakrabarti

Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur 721302, India

Video Captioning Vision-Language Grounding VLM Hallucination Mitigation

Ask PDF

Abstract

Video Caption Evaluation

Explore similar work

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning