Target-Checked Reliability Score Refinement for Video Question Answering

Guoxiang Ren; Rohitash Chandra

cs.CVSep 9, 2026

Target-Checked Reliability Score Refinement for Video Question Answering

Authors: Guoxiang Ren, Rohitash Chandra

Organizations: Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW Sydney, NSW 2052, Australia

Abstract

Video-language models can answer multiple-choice questions with high confidence yet be wrong. We study whether answer-level reliability scores can be improved under target shift without retraining the models or changing their answers. We collect option-probability lists from three fixed video-language models under four deterministic video samplings and represent cross-view changes and cross-model agreement as a response graph. Using a labeled target pilot, we compare the original score, defined as the probability assigned to the chosen answer, with a histogram-based gradient-boosting (HGB) score trained on the development datasets and a regularized logistic-regression score trained on the target pilot. A candidate replaces the original score only when repeated video-level checks indicate a positive, stable improvement. We develop this rule on public VideoQA benchmarks and Video Hallucination Diagnosis (VHD), a controlled diagnostic dataset for shared high-confidence errors. Ranking quality is measured by the area under the risk-coverage curve (AURC), where lower is better. On a held-out 963-question HERBench split, the method reduces mean AURC across the three models by 16.64% (95% confidence interval (CI), 12.12 to 22.61%); the smallest model-level gain is 11.39%. On a separate held-out 911-question Perception Test split, the mean reduction is 18.87% (95% CI, 15.43 to 22.14%). For InternVL3.5, the target check retains the original scores. Using the same outputs, the method outperforms seven training-free baselines in mean AURC on both datasets. It also improves AUROC, reduces calibration error, and lowers the error rate at 50% coverage by 6.50 and 6.58 percentage points.

Figures & tables

Figure 1: Workflow for target-checked reliability score refinement.

Dataset	Questions	Video groups	Answer options	Model families	Role
NExT-GQA	500	500	5	3	Method development and training
STAR	500	500	4	3	Method development and training
MVBench	1,800	1,723	2 to 4	3	Method development and training
VHD (ours)	801	341	2	3	Method development, training, and diagnostic analysis
TempCompass	410	410	2	3	Method development and training
Video-MME	1,089	363	4	3	Pre-revision evaluation

Table 1: Dataset roles and video-grouping units used in the study.

Method	Mean gain	Worst	Positive	Negative
Logistic score, all graph features	11.61%	$-16.48$ %	12	3
Permuted relation labels	10.03%	$-15.90$ %	12	3
Other models, base view	9.84%	$-13.19$ %	12	3
Pre-revision rule	6.95%	0.00%	7	0
Copied other-model responses	3.73%	$-7.98$ %	10	5
Original score	0.00%	0.00%	0	0

Table 2: Leave-one-dataset-out results on the five development datasets.

Figure 2: VHD consensus diagnostic with 95% source-clip bootstrap intervals.

Evaluation	Method	Settings	Mean	Worst	Learned
Development	Pre-revision	15	6.95%	0.00%	7
Video-MME	Pre-revision	3	10.95%	0.00%	2
EgoSchema	Pre-revision	3	19.81%	0.00%	2
EgoSchema	Lower-cost	3	19.52%	0.00%	2
Video-MME-v2	Pre-revision	3	0.09%	$-1.22$ %	2
Video-MME-v2	Lower-cost	3	0.10%	$-1.34$ %	2

Table 3: Results under protocols fixed before the target-check revision.

Policy	Mean	Worst	Gain $>0$	Original kept	Check
Target pilot only	0.47%	$-0.62$ %	30.0%	66.7%	Fail
Three candidates	0.92%	$-0.65$ %	28.3%	65.0%	Pass
Two candidates, no margin	0.96%	$-1.49$ %	33.3%	65.0%	Fail
Final rule	0.99%	0.00%	33.3%	66.7%	Pass

Table 4: Video-MME-v2 target-check development at the 50% pilot fraction.

Pilot	Mean	Worst	Gain $>0$	Original kept
25%	0.32%	0.00%	13.3%	86.7%
35%	0.33%	0.00%	13.3%	86.7%
50%	0.99%	0.00%	33.3%	66.7%

Table 5: Pilot-fraction sensitivity during Video-MME-v2 development.

Model	Acc.	Original AURC	Refined AURC	Gain	Selected score
InternVL3.5	0.464	0.3277	0.2904	11.39%	target pilot
Qwen3-VL	0.476	0.3727	0.2786	25.26%	target pilot
LLaVA-OV2	0.481	0.3153	0.2734	13.29%	target pilot
Model mean	0.474	0.3385	0.2808	16.64%	–

Table 6: HERBench final-test results (963 questions).

Model	Score used	Videos	Qs.	Pilot gain	Lower bound
InternVL3.5	target pilot	34	1008	12.70%	6.78%
Qwen3-VL	target pilot	34	1008	31.39%	23.20%
LLaVA-OV2	target pilot	34	1008	12.25%	4.87%

Table 7: HERBench target-pilot score decisions.

Figure 3: Held-out results by model and scoring method.

Model	Acc.	Original AURC	Refined AURC	Gain	Selected score
InternVL3.5	0.728	0.1008	0.1008	0.00%	original
Qwen3-VL	0.673	0.1557	0.1119	28.14%	target pilot
LLaVA-OV2	0.616	0.1895	0.1356	28.46%	target pilot
Model mean	0.672	0.1487	0.1161	18.87%	–

Table 8: Perception Test final-test results (911 questions).

Model	Score used	Videos	Qs.	Pilot gain	Lower bound
InternVL3.5	original score	250	977	0.00%	0.00%
Qwen3-VL	target pilot	250	977	27.19%	21.05%
LLaVA-OV2	target pilot	250	977	29.62%	23.91%

Table 9: Perception Test target-pilot score decisions.

Figure 4: Pilot-size sensitivity and model-input ablations on the fixed held-out test splits.

Figure 5: Risk-coverage curves on the two held-out test splits, averaged across the three models.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$v,c$	temporal view and answer option
$p_{v}(c)$	probability assigned to option $c$ in view $v$
$a,b$	unchanged base answer and correct option
$y$	1 when $a=b$ , otherwise 0
$G$	response graph for one question
$z$	numerical measurements calculated from $G$

Appendix

Table 10: Notation used in the method.

Task	Scored rows	Accuracy	Mean gain	Worst
Action Counting	195	0.708	42.97%	-3.55%
Action Sequence	216	0.634	19.74%	8.19%
Appearance: Attributes	45	0.778	54.13%	44.84%
Appearance: Behavior/Interactions	132	0.902	26.97%	16.11%
Appearance: Localization/Trajectory	273	0.795	27.63%	14.62%
False Action Memory	225	0.507	12.47%	-13.48%

Appendix

Table 11: HERBench final-test results by task using condensed task labels.

Area	Scored rows	Accuracy	Mean gain	Worst
abstraction	855	0.561	18.29%	0.00%
memory	114	0.544	15.72%	0.00%
physics	975	0.656	20.37%	0.00%
semantics	789	0.830	22.04%	0.00%

Appendix

Table 12: Perception Test final-test results by area.

Label	Pairs	Agreement / $\kappa$
Question valid	1,155	97.7% / 0.945
Correct option	789	89.2% / 0.785
Language prior	180	89.4% / 0.789
Video evidence	180	89.4% / 0.789
Uncertainty	282	90.1% / 0.796

Appendix

Table 13: Agreement between the two VHD question reviewers.

Task	$N$	All models agree (%)	All agree and are wrong (%)	Wrong when all agree (%, 95% CI)
Visible evidence	333	74.2	23.1	31.2 [25.4, 37.0]
Video conflict	178	65.2	55.6	85.3 [78.6, 91.5]
Hidden evidence	290	77.2	12.8	16.5 [11.8, 21.5]

Appendix

Table 14: VHD consensus diagnostic over three model families with fixed weights.

Panel of Figure 6 — Figure 6: VHD annotation conditions for the same source clip.

Method	Mean	Min.	AUROC	ECE	Brier
Target-checked score	16.64%	11.39%	0.799	0.047	0.180
Pilot, all views	16.64%	11.39%	0.799	0.047	0.180
Development logistic	16.36%	12.18%	0.798	0.126	0.196
Development HGB	14.33%	8.09%	0.779	0.117	0.202
Pilot, base view	14.11%	8.93%	0.783	0.050	0.187
Pilot, confidence	0.27%	-0.00%	0.717	0.049	0.212

Appendix

Table 15: HERBench reliability baselines.

Method	Mean	Min.	AUROC	ECE	Brier
Pilot, all views	19.57%	2.11%	0.845	0.121	0.166
Target-checked score	18.87%	0.00%	0.844	0.074	0.154
Pilot, base view	18.29%	-0.86%	0.843	0.119	0.167
Development HGB	14.98%	-4.52%	0.826	0.086	0.161
Development logistic	14.91%	-3.92%	0.830	0.088	0.164
Original score	0.00%	0.00%	0.767	0.129	0.205

Appendix

Table 16: Perception Test reliability baselines.

Target	Score	Score gain	$\Delta$ (pp)	95% CI
HERBench	Top-two margin	-1.20%	17.84	[13.30, 24.16]
HERBench	Negative entropy	-0.08%	16.73	[12.05, 23.15]
HERBench	Base vote	8.91%	7.73	[4.40, 12.21]
HERBench	Base support	13.12%	3.53	[1.35, 6.74]
HERBench	Own-view stability	-15.15%	31.80	[24.92, 39.75]
HERBench	All-view vote	12.53%	4.11	[1.09, 7.21]

Appendix

Table 17: Training-free confidence and consistency scores.

Target	Baseline	Gain (pp)	95% CI
HERBench	Original score	16.64	[12.09, 22.54]
HERBench	Pilot, confidence	16.37	[11.82, 22.36]
HERBench	Pilot, base view	2.54	[0.34, 5.48]
HERBench	Development logistic	0.28	[-1.01, 1.35]
HERBench	Development HGB	2.32	[0.15, 4.84]
HERBench	Pilot, all views, unchecked	0.00	[0.00, 0.00]

Appendix

Table 18: Paired baseline comparisons on the fixed held-out test splits.

Target	Pilot	Videos	Learned used	Gain	Harmful use
HERBench	10%	7	55.0%	9.43%	1.7%
HERBench	20%	14	73.3%	11.64%	0.0%
HERBench	30%	20	85.0%	13.76%	0.0%
HERBench	40%	27	100.0%	16.14%	0.0%
HERBench	50%	34	100.0%	16.64%	0.0%
Perception	10%	50	56.7%	13.96%	0.0%

Appendix

Table 19: Sensitivity to the number of labeled pilot videos.

Target	Model set	Calls	Frames	Mean	Min.
HERBench	Confidence only	1	16	0.27%	0.00%
HERBench	Base, 3 families	3	48	14.11%	8.93%
HERBench	Base+dense, 3 families	6	120	16.51%	11.81%
HERBench	Own 4 views	4	56	1.68%	-1.00%
HERBench	4 views, 2 families	8	112	12.56%	2.74%
HERBench	4 views, 3 families	12	168	16.64%	11.39%

Appendix

Table 20: Response-set ablation and model-input cost.

Confidence Estimation for VLMs

See all 25 papers

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study
Aug 16, 2026Yogesh KumarHallucination Detection in VLMs Video QA
Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge
Jun 3, 2026Tomoya Miyazawa, Hiroyasu OkunoVLM Evaluation Self-Consistency Decoding
Data Analytics Labo Co.
FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling
Aug 22, 2026Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu +8Reward Modeling Video Generation Evaluation
Shanghai Jiao Tong University · Tencent Youtu Lab · Tongji University

Aug 16, 2026J/K move · Enter open · S save

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study

Yogesh Kumar

Hallucination Detection in VLMs Video QA LLM Answer Verification

Open paper AskPDF arXiv

Video question answering systems built on vision-language models often produce timestamped claims with high confidence even when unsupported by the cited frame. This deceptive hallucination arises because timestamps imply grounding without ensuring correctness, increasing user trust but not accuracy. We introduce a pipeline that closes this loop. A retrieval-augmented language model drafts answers with per-claim timestamp citations, and each cited frame is independently re-examined before being shown to the user. We compare against a plain baseline and ablate three verification designs, evaluated on both Apple Silicon (MLX) and Google Colab (HF Transformers, CUDA). Directly asking the vision model whether a frame supports a claim fails completely (0% catch rate on 40 claims) due to sycophancy. Blind re-captioning plus a general LLM judge improves results but is unstable, oscillating between 0% and 100% flagged depending on prompt phrasing. Replacing that judge with a small natural language inference model yields a stable, interpretable verifier that catches 79% of fabricated claims on adversarial false-premise questions while leaving true claims untouched. We release the full pipeline, evaluation harness, and implementations for both Apple Silicon and Colab. Code is available at https://github.com/yogesh-iitj/grounded-video-qa.

Abstract

Confidence Estimation for VLMs

Explore similar work

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study

Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge

FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling