Counterfactual Sensitivity Is Not Repairability: Auditing Replay Probes for Video Evidence

Rama AlHamidi; Rasul Khanbayov; Erchin Serpedin; Hasan Kurban

cs.CVAug 16, 2026

Counterfactual Sensitivity Is Not Repairability: Auditing Replay Probes for Video Evidence

Authors: Rama AlHamidi, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban

Organizations: Department of Computer and Electrical Engineering, Texas A&M University at Qatar, Doha, Qatar · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar · Department of Computer and Electrical Engineering, Texas A&M University, College Station, TX, USA

Abstract

Tool-using video agents retrieve visual evidence before answering, but the final answer is not forced to depend on what was retrieved. The natural black box test is counterfactual: destroy the semantic content of the frames the agent retrieved and check whether the answer changes, against a matched sham that re-executes the identical pipeline on those same frames. We introduce CARVE, a black-box counterfactual probe that compares answer changes under matched SHAM and DESTROY replays. Across three independent k=3 runs on a frozen VideoExplorer-style agent, DESTROY changes the answer 29.3 percentage points more often than SHAM, yielding a large and reproducible aggregate effect. Question-level scores are less stable, and increasing the replay budget from k=3 to k=10 reduces ties but weakens the original zero-threshold routing policy. At k=3, CARVE selects 538 of 1,258 LVBench questions and improves accuracy by 3.26 points, with higher fallback yield than most matched random subsets. The score shows only a weak association with annotated temporal coverage, so CARVE is best understood as a routing signal rather than a direct grounding classifier. Our implementation is available at https://github.com/KurbanIntelligenceLab/CARVE.

Figures & tables

Figure 1: CARVE estimates evidence dependence through matched counterfactual replays. The agent produces a frozen answer $\hat{a}$ from retrieved evidence. CARVE replays the same question under two matched conditions: SHAM preserves the evidence, while DESTROY removes its visual content. Comparing replay answers with $\hat{a}$ gives change rates $\hat{\delta}_{\mathrm{sham}}$ and $\hat{\delta}_{\mathrm{destroy}}$ . Their difference, $\hat{\Delta}$ , measures answer change attributable to evidence destruction beyond SHAM variability: positive values favor DESTROY, zero indicates no separation, and negative values favor SHAM.

Probe flip rates
Metric	Three-seed mean	Seed SD
SELF flip rate	$0.1767$	$0.0068$
SHAM flip rate	$0.3455$	$0.0042$
DESTROY flip rate	$0.6389$	$0.0027$
Estimated-effect distribution
Mean $\hat{\Delta}$	$0.2934$	$0.0070$

Table 1: Aggregate probe behavior across three finalized replay seeds. Entries are seed-level means and sample standard deviations across three independent

k=3

runs on the same LVBench evaluation universe. Each seed is treated as an independent replication rather than pooled into a larger replay budget.

Figure 2: Increasing replay depth resolves the score but weakens the fixed routing rule. (a) Transition matrix for the 1,249 LVBench questions with valid scores at both $k=3$ and $k=10$ . Most low-budget ties resolve, whereas only a small fraction of original non-ties become tied. (b) Consequences of applying the same $\hat{\Delta}\leq 0$ routing rule at each replay budget. Increasing the budget selects fewer questions and reduces both final accuracy gain and net repairs per fallback call. Routing accuracy is evaluated over the full 1,258-question population; questions without a valid probe score retain the vanilla answer.

Score by evidence coverage and correctness
Gold cov.	Correct	$n$	Mean $\hat{\Delta}$	95% CI
Yes	Yes	275	0.453	[0.403, 0.502]
No	Yes	293	0.325	[0.266, 0.387]
Yes	No	245	0.191	[0.136, 0.244]
No	No	436	0.242	[0.207, 0.278]
Association with evidence coverage

Table 2: The probe has a small aggregate association with temporal coverage, but weak question-level discrimination. Statistics are computed over LVBench questions with valid probe scores; 95% confidence intervals are obtained by bootstrap resampling at the video level. Gold coverage denotes positive temporal overlap between retrieved evidence and the annotated answer-bearing interval. Among correct answers, covered questions receive higher mean scores than uncovered questions, while the association reverses among incorrect answers. Correlations with temporal recall and IoU are small, and the ROC-AUC is close to chance.

Policy	Selected	Repairs	Harms	Yield per 100
$\hat{\Delta}<0$	166	33	19	8.43
$\hat{\Delta}=0$	372	65	38	7.26
$\hat{\Delta}\leq 0$	538	98	57	7.62
Plain random	538	–	–	5.09
Video-balanced random	538	–	–	5.28
Evidence-matched random	538	–	–	5.30

Table 3: CARVE identifies a higher-yield fallback subset than matched random routing. Routing outcomes are computed over the 1,258-question LVBench evaluation universe. Repairs and harms are question counts; yield is net repairs per 100 fallback calls. Random rows report mean yield across matched random subsets containing the same number of questions as the CARVE-selected set.

Method	Acc. (%)	Tools	Tok.	$\Delta$ Acc.	Repairs	Harms	95% CI
LVBench ( $n=1{,}258$ )
Vanilla	$45.47$	$4.21$	32.2k	–	–	–	–
Judge-grnd.	$45.47$	$4.39$	37.6k	$0$	$3$	$3$	[-0.40, +0.40]
Look-more	$45.07$	$4.39$	37.5k	$-0.40$	$2$	$7$	[-0.92, +0.07]
Self-consist.	$43.48$	$4.21$	36.9k	$-1.99$	$32$	$57$	[-3.58, -0.40]
Random deferral	$47.65$	$4.21$	35.8k	$2.18$	–	–	[+0.95, +3.42]*

Table 4: Accuracy and rerouting outcomes. Results are evaluated at the question level on LVBench (

n=1{,}258

) and Video-MME (

n=874

), with paired 95% confidence intervals clustered by video.

\Delta

Acc. is measured relative to Vanilla on the same evaluation universe. For Random deferral, the starred interval is the central 95% range of the matched Monte Carlo random-selection distribution rather than a confidence interval. Repairs and harms count question-level answer changes from incorrect to correct and correct to incorrect, respectively. The probe issues no planner tool calls, so equal tool counts do not imply equal total compute; token totals also exclude some probe rendering.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Run pair	Pearson $r$	Sign agreement	Trigger-set Jaccard
Run 1 vs. Run 2	0.586	0.617	0.198
Run 1 vs. Run 3	0.548	0.617	0.237
Run 2 vs. Run 3	0.547	0.601	0.205

Appendix

Table A1 : Question-level scores are moderately reproducible at

k=3

. All agreement statistics compare paired

\hat{\Delta}

estimates for the same LVBench questions across two independent replay runs. Pearson

r

measures continuous-score agreement; sign agreement measures whether paired scores fall in the same negative, zero, or positive category; trigger-set Jaccard measures overlap between the question sets selected by the strict

\hat{\Delta}<0

rule.

LVBench
Budget	Distribution	$\hat{\Delta}<0$	$\hat{\Delta}=0$	$\hat{\Delta}>0$	Tie excess
$k=3$	Homogeneous prediction	0.117	0.236	0.647	–
$k=3$	Observed ( Run 1 )	0.133	0.298	0.569	+0.062
$k=10$	Homogeneous prediction	0.056	0.071	0.874	–
$k=10$	Observed	0.165	0.123	0.712	+0.052
	Change from $k=3$ to $k=10$	+0.032	$-$ 0.175	+0.143	$-$ 0.010

Appendix

Table A2 : Effect of replay budget on the distribution of estimated score differences. Proportions are computed over questions with a valid

\hat{\Delta}

estimate at the corresponding replay budget. For LVBench, the k=3 observed row corresponds to Run 1 , which is the run subsequently extended to

k=10

; Table 1 instead reports means across all three independent

k=3

runs. Tie excess is the observed question-level tie rate minus the homogeneous finite-replay prediction.

Replay budget	Stratum	$n$	$\hat{\delta}_{s}$ (SHAM)	$\hat{\delta}_{d}$ (DESTROY)
$k=3$	Covered, correct	275	$0.215$ [0.181, 0.250]	$0.668$ [0.625, 0.710]
	Uncovered, correct	293	$0.281$ [0.241, 0.324]	$0.606$ [0.555, 0.658]
	Covered, incorrect	245	$0.420$ [0.376, 0.462]	$0.612$ [0.564, 0.657]
	Uncovered, incorrect	436	$0.424$ [0.391, 0.455]	$0.666$ [0.632, 0.700]
$k=10$	Covered, correct	276	$0.212$ [0.182, 0.242]	$0.644$ [0.605, 0.683]
	Uncovered, correct	294	$0.284$ [0.248, 0.322]	$0.617$ [0.574, 0.660]

Appendix

Table A3 : Component decomposition of the CARVE score by temporal coverage and answer correctness. Entries are question-level mean SHAM and DESTROY flip rates within each LVBench stratum; brackets give 95% bootstrap confidence intervals obtained by resampling videos. Among correct answers, the larger score for covered questions is driven mainly by lower SHAM flip rates. Among incorrect answers, the association reverses because covered questions have lower DESTROY flip rates. The same qualitative pattern appears at both replay budgets.

Metric	$\hat{\Delta}$	Confidence
Accuracy at 100% coverage	42.86%	42.86%
Accuracy at 90% coverage	44.18%	43.47%
Accuracy at 70% coverage	44.81%	42.79%
Partial risk AUC, 70–100%	0.558	0.567
Full-range AURC	0.544	0.521

Appendix

Table A4: Selective-prediction performance depends on the coverage regime. Results are computed over the strict common-valid Video-MME question subset (

n=595

). Coverage is the fraction of questions retained after ranking; reported accuracies are computed over the retained questions at each coverage level. Partial risk AUC summarizes the 70–100% coverage range, while AURC summarizes the full risk–coverage curve.

Source	Source gain	Transfer gain	Rule selected
	(pp)	(pp)	(%)
7B	$+3.51\pm 1.53$	$+2.17\pm 1.33$	96.8
32B	$+1.95\pm 1.21$	$+3.52\pm 1.65$	93.2

Appendix

Table A5: The inclusive deferral rule transfers across answerer backbones. Entries report mean accuracy gain

\pm

standard deviation over 400 independently sampled video-level splits. Source gain evaluates the selected rule on the backbone used for rule selection; transfer gain evaluates that same rule on the other backbone. Rule-selected percentages report the fraction of the 400 splits in which the inclusive

\hat{\Delta}\leq 0

policy is chosen.

Method	Train-free	Black-box	Agentic	Primary signal
VER [ 4 ]	✗	✗	✗	Evidence-grounding reward
DeFacto [ 6 ]	✗	✗	✗	Counterfactual supervision
VideoSEAL [ 5 ]	✗	✗	✓	Temporal overlap and judge score
STEAR [ 11 ]	✓	✗	✗	Internal patch intervention
FACTOR [ 13 ]	✓	✓	✗	Attribute intervention
Self-consistency [ 14 ]	✓	✓	Optional	Answer agreement

Appendix

Table A6 : Comparison of CARVE with grounding, perturbation, and selective-prediction methods. Train-free methods do not update the evaluated model; black-box methods require no access to model internals; agentic methods operate over multi-step tool-use trajectories.

AI Agent Auditing

AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.

See all 262 papers

FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding
Aug 11, 2026Fufangchen Zhao, Jinhu Fu, Jiachen Lei +3Counterfactual Evaluation Video QA
State Key Laboratory of Networking and Switching Technology, BUPT · AMAP, Alibaba Group
Counterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA
Jun 8, 2026Zhou Du, Hamid Krim, Xiao Wu +3Causal Representation Learning Video QA
Graduate School of Informatics, Nagoya University, Nagoya 464-8601, Japan · Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC 27695, USA · School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611730, China +1
REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering
Aug 9, 2026Caijun Yan, Yang Zhou, Meixing Shi +4Video QA Long-Video QA
1Zhejiang University · 2Shanghai AI Laboratory

Aug 11, 2026J/K move · Enter open · S save

FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding

Fufangchen Zhao, Jinhu Fu, Jiachen Lei, Jiahong Wu+2

State Key Laboratory of Networking and Switching Technology, BUPT · AMAP, Alibaba Group

Counterfactual Evaluation Video QA Causal Reasoning in Videos

Open paper AskPDF arXiv

Counterfactual video understanding evaluates whether models grasp physical and commonsense regularities. However, existing multiple-choice question (MCQ) benchmarks inadvertently leak target events through their questions and candidate options. This reduces the core challenge from active discovery to text-guided verification. In this paper, we present FADE, an effective training framework for counterfactual discovery and explanation. Our method is built on an evidence-first, two-stage training paradigm. First, evidence-internalized supervised fine-tuning grounds the model's predictions in decisive visual anomalies. Second, we apply a fading-anchor reinforcement learning strategy that progressively removes textual guidance, compelling the model to independently discover and explain evidence. To rigorously evaluate this capability, we also introduce an effective pipeline that converts existing MCQ datasets into aligned MCQ, open-ended question answering (OQA), and captioning tasks without requiring additional data curation. Our simple approach yields strong results. Using Qwen3-VL-8B as the baseline, FADE achieves state-of-the-art strict paired scores across all three tasks on DualityVidQA-test and IPV-Bench, outperforming GPT-5.6. In specific, when transitioning from constrained MCQs to unconstrained OQA and captioning, our model demonstrates remarkable robustness. Its performance retention is 90.4% and 67.4% on DualityVidQA-test-substantially higher than the 48.1% and 30.7% retained by GPT-5.6. We hope this simple framework can serve as a solid baseline for future research in unconstrained counterfactual video understanding.

Abstract

AI Agent Auditing

Explore similar work

FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding

Counterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA

REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering