cs.CVOct 2, 2026

Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation

Authors: Mahsa Mohammadi, Sareh Rowlands

Organizations: University of Exeter Exeter, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Benchmark scores license claims about the capabilities being evaluated. We audit the inference licensed by an evaluation protocol, rather than the predictive model alone. Using procedural action anticipation as a controlled case study, we study a broader evaluation-validity failure mode: a protocol can remain temporally causal and free of classical target leakage while still supplying a privileged intermediate representation. The failure is not merely optimistic accuracy, but a mismatch between the capability claimed and the construct actually measured. On Breakfast, matched recognizer-history and GT-history regimes score 30.3% and 62.3% (Delta_PH(R_BA) = +32.0, 95% Student-t interval [+24.1, +39.8]). A fixed-checkpoint 2 x 2 intervention isolates a +14.6-point test-time oracle contrast; a matched no-video provenance probe yields a +26.8-point gap; and a boundary-independent query/history control retains a +19.4-point gap. We refer to this discrepancy as a privileged-information gap, defined relative to a specified non-oracle recovery pipeline, and propose a reusable four-condition audit. Across five evaluation settings the gap is heterogeneous; a standard 50 Salads long-term-anticipation re-implementation provides cautious support beyond the audit-specific next-action construction.

Figures & tables

Figure 1: Four-condition audit. Video only measures the primary modality; recognizer history adds recoverable symbolic context; GT history supplies the same semantic variable from an oracle source; and the context-only probe removes the observation-window visual input. Blue denotes non-oracle inputs; red denotes the privileged oracle input.
ConditionAct.@1Act.@5Verb@1
Video only21.7±2.721.7{\scriptstyle\pm 2.7}54.3±2.354.3{\scriptstyle\pm 2.3}36.3±1.336.3{\scriptstyle\pm 1.3}
Recognizer history, boundary-free26.3±0.926.3{\scriptstyle\pm 0.9}——
Recognizer history, boundary-aware30.3±3.130.3{\scriptstyle\pm 3.1}67.7±1.567.7{\scriptstyle\pm 1.5}44.1±1.544.1{\scriptstyle\pm 1.5}
GT transcript (oracle)62.3±2.662.3{\scriptstyle\pm 2.6}89.8±1.789.8{\scriptstyle\pm 1.7}68.1±2.168.1{\scriptstyle\pm 2.1}
Context-only (oracle context)58.8±2.058.8{\scriptstyle\pm 2.0}85.4±1.485.4{\scriptstyle\pm 1.4}63.1±2.863.1{\scriptstyle\pm 2.8}
Table 1: Primary four-condition Breakfast audit (mean ±\pm std over 4 splits). The boundary-aware recognizer defines the primary matched non-oracle regime; the stricter boundary-free row is shown separately because it changes the history-recovery pipeline RR .
ComparisonNon-oracleOracleGap [95% CI]Interpretation
Matched information regimes30.3±3.130.3{\scriptstyle\pm 3.1}62.3±2.662.3{\scriptstyle\pm 2.6}+32.0+32.0 [+24.1,+39.8][+24.1,+39.8]Primary estimator-relative protocol-level audit gap.
Fixed checkpoint30.3±3.130.3{\scriptstyle\pm 3.1}44.9±4.644.9{\scriptstyle\pm 4.6}+14.6+14.6 [+11.0,+18.3][+11.0,+18.3]Test-time oracle substitution with weights fixed.
No-video context32.5±1.732.5{\scriptstyle\pm 1.7}59.2±2.559.2{\scriptstyle\pm 2.5}+26.8+26.8 [+22.3,+31.2][+22.3,+31.2]Separates oracle provenance from context predictability.
Boundary-independent query/history28.1±1.928.1{\scriptstyle\pm 1.9}47.5±1.847.5{\scriptstyle\pm 1.8}+19.4+19.4 [+16.4,+22.4][+16.4,+22.4]Removes annotated boundaries from query scheduling and recovered history.
Table 2: Key Breakfast controls. Each row addresses a different alternative explanation. The no-video provenance probe is separately retrained, so its oracle-context score ( 59.259.2 ) need not equal the main context-only baseline ( 58.858.8 ).
Figure 2: Train–test provenance intervention. Holding the recognizer-trained checkpoint fixed yields a +14.6+14.6 test-time oracle contrast ( [+11.0,+18.3][+11.0,+18.3] ); the off-diagonal cells expose the provenance interaction.
DatasetWindow-rec.BFRBAGT
Breakfast44.4±1.244.4{\scriptstyle\pm 1.2}26.3±0.926.3{\scriptstyle\pm 0.9}30.3±3.130.3{\scriptstyle\pm 3.1}62.362.3
50 Salads74.7±0.774.7{\scriptstyle\pm 0.7}43.2±4.343.2{\scriptstyle\pm 4.3}46.0±3.546.0{\scriptstyle\pm 3.5}53.753.7
GTEA99.6±0.199.6{\scriptstyle\pm 0.1}66.2±2.766.2{\scriptstyle\pm 2.7}70.2±4.270.2{\scriptstyle\pm 4.2}74.274.2
Table 3: Boundary information in history recovery. Window-recognition accuracy measures the past-window labeler itself; BFR and BA are downstream anticipation Top-1 with boundary-free and boundary-aware recovered histories.
SettingRecognizerGTΔPH\Delta_{\mathrm{PH}}95% split-level intervalScope
Breakfast30.3±3.130.3{\scriptstyle\pm 3.1}62.3±2.662.3{\scriptstyle\pm 2.6}+32.0+32.0[+24.1,+39.8][+24.1,+39.8]primary audit
50 Salads46.0±3.546.0{\scriptstyle\pm 3.5}53.7±2.453.7{\scriptstyle\pm 2.4}+7.7+7.7[+5.4,+9.9][+5.4,+9.9]next-action audit setting
GTEA †70.2±4.270.2{\scriptstyle\pm 4.2}74.2±3.874.2{\scriptstyle\pm 3.8}+4.0+4.0[+1.4,+6.6][+1.4,+6.6]diagnostic next-action
Assembly101 †18.9±4.418.9{\scriptstyle\pm 4.4}19.4±3.919.4{\scriptstyle\pm 3.9}+0.4+0.4[−3.4,+4.2][-3.4,+4.2]diagnostic next-action
EPIC-100 †‡14.714.715.015.0+0.3+0.3—diagnostic, n=1n=1
Table 4: Cross-setting scope test. GTEA, Assembly101, and EPIC-100 ( † ) use diagnostic next-action settings rather than the datasets’ standard anticipation benchmarks. EPIC-100 ( ‡ ) is a single-run low-accuracy diagnostic.
 β=.1\beta=.1β=.2\beta=.2β=.3\beta=.3β=.5\beta=.5
ΔPH\Delta_{\mathrm{PH}}+2.0+2.0+5.2+5.2+3.8+3.8+2.4+2.4
95% CI (per video)[−3.6,+7.5][-3.6,+7.5][+0.5,+9.8][+0.5,+9.8][+0.4,+7.1][+0.4,+7.1][+0.3,+4.4][+0.3,+4.4]
95% CI (per split)[−8.3,+12.3][-8.3,+12.3][−4.0,+14.3][-4.0,+14.3][−2.5,+10.0][-2.5,+10.0][−1.5,+6.2][-1.5,+6.2]
Table 5: Standard 50 Salads LTA re-implementation. Point gaps are GT minus recognizer MoC. The supplement reports the corresponding condition means and full protocol details.
Figure 3: Estimator-relative sensitivity on Breakfast. Left: history-dose response. Right: recognizer-history performance as controlled history noise decreases. The second panel directly demonstrates that improving recovery changes the matched non-oracle score and therefore the measured ΔPH(R)\Delta_{\mathrm{PH}}(R) .
Figure 4: Breakfast decomposition. The latest action alone captures a large fraction of the context-only gain; exact ordering adds comparatively little.
EvidenceWhat it addresses
Fixed checkpoint: +14.6+14.6Not only a consequence of training under cleaner history.
Matched regimes: +32.0+32.0Primary estimator-relative audit gap.
No-video provenance: +26.8+26.8Not merely “context is predictive.”
Boundary-independent: +19.4+19.4Annotated query/history boundaries are not sufficient.
50 Salads standard LTASupporting evidence beyond audit-specific next-action construction.
Table 6: Evidence hierarchy. Each control targets a different rival explanation.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

WorkObserved-past sourceRelevance
Uncertainty-aware anticipation Abu Farha and Gall [2019]Clean/GT and recognized observationsSeparates clean and inferred observed actions.
Cycle-consistent anticipation Abu Farha et al. [2020]Visual featuresMotivates direct video-to-future prediction rather than fixed transcripts.
AntGPT Zhao et al. [2024]Symbolic actions derived from observationsUses semantic action sequences as context.
AAG Benavent-Lledo et al. [2025]GT, recognizer-, or VLM-derived historyMakes history provenance a first-order modeling choice.
Appendix
Table 7: Examples of observed-past representations in prior work.
Figure 5: Task definition. The model observes only the procedural prefix available before prediction time. Future frames and the target label are never model inputs.
Figure 6: Operationalization of the estimator-relative privileged-information gap. The matched oracle-history and recovered-history regimes share task, architecture family, targets, visual construction, optimization, and training budget, while history provenance differs.
ConditionAct.@1Act.@5Verb@1
Video only21.7±2.721.7{\scriptstyle\pm 2.7}54.3±2.354.3{\scriptstyle\pm 2.3}36.3±1.336.3{\scriptstyle\pm 1.3}
Recognizer history, boundary-free26.3±0.926.3{\scriptstyle\pm 0.9}——
Recognizer history, boundary-aware30.3±3.130.3{\scriptstyle\pm 3.1}67.7±1.567.7{\scriptstyle\pm 1.5}44.1±1.544.1{\scriptstyle\pm 1.5}
GT transcript (oracle)62.3±2.662.3{\scriptstyle\pm 2.6}89.8±1.789.8{\scriptstyle\pm 1.7}68.1±2.168.1{\scriptstyle\pm 2.1}
Context-only oracle probe58.8±2.058.8{\scriptstyle\pm 2.0}85.4±1.485.4{\scriptstyle\pm 1.4}63.1±2.863.1{\scriptstyle\pm 2.8}
Appendix
Table 8: Breakfast next-action anticipation (4 splits).
Train historyTest: recognizerTest: GT
Recognizer30.3±3.130.3{\scriptstyle\pm 3.1}44.9±4.644.9{\scriptstyle\pm 4.6}
GT oracle23.8±1.623.8{\scriptstyle\pm 1.6}62.3±2.662.3{\scriptstyle\pm 2.6}
Appendix
Table 9: 2×22\times 2 train-history ×\times test-history intervention (Breakfast, 4 splits).
ConditionAct.@1 (%)
Video only21.5±1.221.5{\scriptstyle\pm 1.2}
Recognizer history28.1±1.928.1{\scriptstyle\pm 1.9}
GT history (oracle)47.5±1.847.5{\scriptstyle\pm 1.8}
ΔPH\Delta_{\mathrm{PH}}+19.4±1.9+19.4{\scriptstyle\pm 1.9}
95% Student- tt interval[+16.4,+22.4][+16.4,+22.4]
Appendix
Table 10: Boundary-independent query and history-recovery control (Breakfast, 4 splits). Queries start at feature-frame 64 and repeat every 60 feature frames; the visual window is 32 feature frames, and the boundary-free history recognizer uses 16-feature-frame windows with stride 8.
DatasetWindow-recog.BFRBADropGT
Breakfast44.4±1.244.4{\scriptstyle\pm 1.2}26.3±0.926.3{\scriptstyle\pm 0.9}30.3±3.130.3{\scriptstyle\pm 3.1}−4.0-4.062.362.3
50 Salads74.7±0.774.7{\scriptstyle\pm 0.7}43.2±4.343.2{\scriptstyle\pm 4.3}46.0±3.546.0{\scriptstyle\pm 3.5}−2.8-2.853.753.7
GTEA99.6±0.199.6{\scriptstyle\pm 0.1}66.2±2.766.2{\scriptstyle\pm 2.7}70.2±4.270.2{\scriptstyle\pm 4.2}−4.0-4.074.274.2
Appendix
Table 11: Boundary-free versus boundary-aware history recognition.
History length HH01248
GT history23.851.061.960.060.6
Recognizer history23.827.429.232.231.0
Appendix
Table 12: History-length dose response on Breakfast.
DatasetBackbone#clsVideoRec.(BA)Rec.(BFR)GTΔPHBA\Delta^{BA}_{PH}ΔPHBFR\Delta^{BFR}_{PH}
BreakfastI3D4721.730.3±3.130.3{\scriptstyle\pm 3.1}26.3±0.926.3{\scriptstyle\pm 0.9}62.3±2.662.3{\scriptstyle\pm 2.6}+32.0+32.0+36.0+36.0
50 SaladsI3D1940.746.0±3.546.0{\scriptstyle\pm 3.5}43.2±4.343.2{\scriptstyle\pm 4.3}53.7±2.453.7{\scriptstyle\pm 2.4}+7.7+7.7+10.5+10.5
GTEA †I3D1167.370.2±4.270.2{\scriptstyle\pm 4.2}66.2±2.766.2{\scriptstyle\pm 2.7}74.2±3.874.2{\scriptstyle\pm 3.8}+4.0+4.0+8.0+8.0
Assembly101 †TSM8621.318.9±4.418.9{\scriptstyle\pm 4.4}—19.4±3.919.4{\scriptstyle\pm 3.9}+0.4+0.4—
EPIC-100 †‡TSN33114.414.7—15.0+0.3+0.3—
Appendix
Table 13: Full cross-setting audit. The 50 Salads row is the next-action audit setting; the standard long-term-anticipation re-implementation is reported separately below.
Observed historyβ=.1\beta=.1β=.2\beta=.2β=.3\beta=.3β=.5\beta=.5
Recognizer7.66.24.94.1
GT transcript9.511.38.76.5
ΔPH\Delta_{\mathrm{PH}}+2.0+5.2+3.8+2.4
95% CI (per video)[−3.6,+7.5][-3.6,+7.5][+0.5,+9.8][+0.5,+9.8][+0.4,+7.1][+0.4,+7.1][+0.3,+4.4][+0.3,+4.4]
95% CI (per split)[−8.3,+12.3][-8.3,+12.3][−4.0,+14.3][-4.0,+14.3][−2.5,+10.0][-2.5,+10.0][−1.5,+6.2][-1.5,+6.2]
Appendix
Table 14: 50 Salads under the standard long-term anticipation protocol (observe 20%20\% , predict β\beta ; MoC). Gaps are computed from unrounded paired values and may differ slightly from subtracting displayed rounded means.
Conditionh=1h=1h=2h=2h=3h=3h=4h=4h=5h=5
Video only25.423.921.719.820.6
Recognizer history35.129.524.421.822.0
GT history61.046.638.133.334.0
ΔPH\Delta_{\mathrm{PH}}+25.9+17.1+13.7+11.5+12.1
95% interval[+22.8,+29.1][+22.8,+29.1][+14.9,+19.3][+14.9,+19.3][+11.1,+16.3][+11.1,+16.3][+9.9,+13.0][+9.9,+13.0][+10.3,+13.9][+10.3,+13.9]
Appendix
Table 15: Breakfast multi-horizon anticipation (4 splits).
PredictorBreakfast50 SaladsGTEA
Majority8.4±0.78.4{\scriptstyle\pm 0.7}7.7±0.87.7{\scriptstyle\pm 0.8}26.6±0.426.6{\scriptstyle\pm 0.4}
Bigram P(ai+1∣ai)P(a_{i+1}|a_{i})47.3±2.647.3{\scriptstyle\pm 2.6}47.5±1.847.5{\scriptstyle\pm 1.8}50.6±3.950.6{\scriptstyle\pm 3.9}
Latest action only50.1±0.850.1{\scriptstyle\pm 0.8}60.7±3.660.7{\scriptstyle\pm 3.6}73.8±6.073.8{\scriptstyle\pm 6.0}
Completed-action set58.5±0.958.5{\scriptstyle\pm 0.9}54.5±3.854.5{\scriptstyle\pm 3.8}73.6±3.673.6{\scriptstyle\pm 3.6}
Full ordered history60.6±1.060.6{\scriptstyle\pm 1.0}54.5±3.354.5{\scriptstyle\pm 3.3}73.3±4.573.3{\scriptstyle\pm 4.5}
Gain from latest action+41.7+53.0+47.2
Appendix
Table 16: Decomposition of privileged-history signal (GT history, no observation-window video).
Probe inputAct.@1 (%)
History only59.1±1.859.1{\scriptstyle\pm 1.8}
State only46.5±2.146.5{\scriptstyle\pm 2.1}
History + state59.2±2.259.2{\scriptstyle\pm 2.2}
Appendix
Table 17: Source of context-only probe signal on Breakfast.
Probe contextAct.@1 (%)
Recognizer context32.5±1.732.5{\scriptstyle\pm 1.7}
GT context59.2±2.559.2{\scriptstyle\pm 2.5}
Privileged-context gap+26.8±2.8+26.8{\scriptstyle\pm 2.8}
95% Student- tt interval[+22.3,+31.2][+22.3,+31.2]
Appendix
Table 18: No-video provenance probe on Breakfast (4 splits). This is a separately retrained experiment; its GT score need not equal the main 58.8%58.8\% context-only baseline.
ArchitectureVideoRec.GTΔPH\Delta_{\mathrm{PH}}nnCI/status
Transformer-fusion21.730.362.3+32.04[+24.1,+39.8][+24.1,+39.8]
TCN24.932.259.3+27.14[+24.8,+29.4][+24.8,+29.4]
GRU27.837.061.7+24.84[+20.2,+29.4][+20.2,+29.4]
SSM24.429.457.6+28.22robustness
Causal LM21.632.157.6+25.62robustness
Diffusion18.428.857.8+28.92robustness
Appendix
Table 19: Breakfast-only architecture sensitivity.
Target-class frequency
Binrarestmiddlecommonest
Mean train count102.6192.6411.8
Act.@150.262.577.5
Transition probability P(ai+1∣ai)P(a_{i+1}|a_{i})
Binraremediumfrequent
Mean probability0.0280.3060.782
Appendix
Table 20: Error analysis under privileged history (Breakfast split 1).
 GT historyRecognizer history
Full model63.431.7
Zero video39.0(−24.4)39.0(-24.4)12.1(−19.6)12.1(-19.6)
Zero history14.2(−49.2)14.2(-49.2)9.3(−22.4)9.3(-22.4)
Zero state61.6(−1.8)61.6(-1.8)29.5(−2.2)29.5(-2.2)
grad ×\times input video57.3%79.5%
grad ×\times input history36.0%17.8%
Appendix
Table 21: Input-stream attribution (Breakfast split 1).
More in this topic

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

Explore similar work

CardsList

  1. What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals

    Aug 18, 2026Xi Qin, Jizhou TongBenchmark AuditingBenchmark Validity

    Wuhan University

  2. Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation

    Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM EvaluationLLM Auditing

    School of Statistics and Data Science, Southwestern University of Finance and Economics

  3. Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

    Sep 24, 2026Shuzhi Gong, Fengze Sun, Yuansan LiuTemporal Video GroundingVideo-Language Model Evaluation

    The University of Melbourne