Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

Dan Ben-Ami; Kobi Cohen; Chaim Baskin

cs.CVOct 6, 2026

Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

Authors: Dan Ben-Ami, Kobi Cohen, Chaim Baskin

Organizations: INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel

Abstract

Streaming video-language models must decide not only what to answer, but whether the evidence needed for the current question has arrived. Existing systems learn that decision as a separate trigger; we ask whether an unmodified model already computes it. We show that frozen VideoLLMs carry a linearly readable evidence-readiness signal, labelled from timestamped evidence rather than from model output. It decodes in all seven models of a shared byte-identical evaluation (AUROC 0.733-0.905 under the strictest not-ready sampling, where a fitted clock is near chance), and a probe fitted without any of a benchmark family's footage still reads that family. It is question-conditioned: on byte-identical windows, changing only the question reverses the readout on 66.1% of pairs, while every question-blind control is at chance by construction. The model can answer incorrectly and still encode readiness: AUROC remains 0.722 among wrong answers. Readiness also beats uncertainty estimators and their supervised combination on latency-matched answer selection, and tracks independent human judgments more closely than confidence. Released streaming triggers are also linear readouts, yet a trained trigger read on its own base model's activations is approximately orthogonal to readiness and decodes it far less accurately than a probe. We turn the readout into Readiness Gating, an answer-timing policy that improves accuracy by up to +9.75 pp at matched video duration with negligible computational overhead. How much it gains varies with the accuracy headroom the task makes available: across 26 configurations the gain tracks that headroom, and an intervention that moves it over identical pixels moves the gain with it.

Figures & tables

Figure 1: A model can be confident and unready at the same time. Asked an object’s colour before it enters the stream, the model answers confidently from prior alone; a linear readout of its frozen residual stream reports that the evidence has not arrived. (Schematic; measured in section 4 .)

source	geometry	label	units	shuffle control	confidence	visual layer- $0$	readiness AUROC
HD-EPIC †	composite	cue present	520 prefixes	$0.524$	$0.845$	$0.500$	$\mathbf{0.985}$ $[.965,.999]$
HD-EPIC †	matched pair	availability	4,800 windows	$0.495$	$0.636$	$0.495$	$\mathbf{0.827}$ $[.812,.840]$
NExT-GQA	quad	containment	1,208 quads	$0.498$	$0.588$	$0.500$	$\mathbf{0.727}$ $[.710,.745]$
NExT-GQA	prefix	onset	5,580 windows	$0.502$	$0.580$	$0.544$	$\mathbf{0.733}$ $[.715,.749]$
ST-Evidence	prefix	onset	6,546 windows	$0.484$	$0.589$	$0.552$	$\mathbf{0.722}$ $[.706,.739]$
ST-Evidence-Gen	prefix	onset	4,346 prefixes	$0.578$	$0.622$	$0.651$	$\mathbf{0.827}$ $[.813,.840]$

Table 1: Core representation results. Labels come from evidence timestamps, not model correctness. Rows marked † pose questions we construct rather than questions the source ships (Appendix D.2 ). The column is nine separate tests, not one scale: rows differ in geometry and strictness, so each AUROC is read against the three columns beside it rather than against the rows above it (Appendix A ). Read that way the verdict is uniform, with readiness above all three columns on every row by

0.133

0.220

. Only the shuffle is a control, and it is measured on every row rather than

0.500

by construction; the other two are competing readouts that may legitimately carry signal. Visual layer-

0

reads raw patch embeddings before any decoder layer, against a language-position layer-

0

readout forced to

0.500

everywhere ( section 3 ); it is forced by the design on the crossed quads, and on the controlled composites is quoted under a pair-preserving grouping ( subsection C.4 ).

measurement	Oops!	ActivityNet-Captions
readiness AUROC on evaluated windows	$0.831$	$0.753$
elapsed-time AUROC on the same windows	$0.519$	$0.436$
selection accuracy, answering at end of video	$48.7\%$	$35.8\%$
selection accuracy, confidence	$60.6\%$	$54.0\%$
selection accuracy, Readiness Gating (ours)	$\mathbf{67.6\%}$	$\mathbf{64.7\%}$
gating advantage over supervised combination	$+4.5$ pp $[+0.3,+8.7]$	$+8.2$ pp $[+4.6,+11.7]$

Table 2: Uncertainty baselines do not recover readiness on the two tested high-conversion settings. All arms answer at an identical deadline; the middle block gives the accuracies the advantages come from (paired margins

+7.1

and

+10.7

pp over confidence). The two AUROCs are read on the windows the policy chooses among, not the Table 1 pools of the same names. Every estimator loses on both, and adding a supervised combination back to gating gives no detectable gain.

Figure 2: Readiness accrues with depth, and it is not the model’s own confidence. Seven models, four language-backbone families, on one frozen Oops! manifest of

1{,}204

windows over

301

videos with not-ready rows drawn from both sides of the evidence. (a) Held-out AUROC at the final language position against fraction of decoder depth, cluster-bootstrap bands over videos; layer

0

is the raw input embedding, exactly

0.500

by construction, and every model peaks between

0.5

and

0.8

of depth rather than at the output. (b) Readiness at each model’s best depth, rows ordered by parameter count, against the model’s own answer probability (

0.514

–

0.651

), the measured layer-

0

visual readout (

0.579

–

0.683

) and the label-shuffle null, all on the same rows. Parameter count does not order readiness: the

2

B model reads above four larger ones.

	HERBench shot-ID		ActivityNet, same-action
monitor	readiness AUROC	selection acc.	readiness AUROC	selection acc.
answer at end (no selection)	—	$0.251$	—	$0.358$
evidence oracle	—	$0.897$	—	$0.703$
Readiness Gating (ours)	$\mathbf{0.926}$	$\mathbf{0.763}$	$\mathbf{0.753}$	$\mathbf{0.647}$
confidence	$0.749$	$0.495$	$0.594$	$0.540$
elapsed time	$0.507$	—	$0.436$	—

Table 3: Released streaming triggers, scored against evidence-readiness labels on two substrates. These are the policy ladders,

6{,}368

and

16{,}109

windows, not the Table 1 probing pools of the same names ( Appendix I ). At a common deadline every arm emits the answer cached from its own highest-scoring window, so selection is latency-matched by construction. Our probe is fitted on each dataset’s training folds and evaluated on held-out folds; the released triggers are applied without fitting. Removing that asymmetry does not change the ordering: on MMDuet’s own activations a probe reads

0.843

where its trained head reads

0.607

( subsection 4.5 ). The comparison tests transfer to readiness labels, not which complete streaming system performs best. Dispider is a secondary baseline and decides once per

16

s, too coarse for the ActivityNet ladder, so those cells are empty; every rival selection arm is significantly worse than plain confidence (

-0.107

-0.219

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

geometry	shape	frames	label	what it is for
growing prefix	$[0,t]$ , evidence-relative ends	$32$	onset, $\mathrm{seen}/K$	graded accumulation; earliest evidence
recent window	$[t-W,t]$ , fixed span, $50\%$ overlap	$32$	containment	first evidence-agnostic ladder
containment window	$16$ s span, free $2$ s stride	$32$ @ $2$ fps	containment	the reference format; model axis, utility
short window	$2$ s span, $0.5$ s stride	$8$ @ $4$ fps	containment	point-onset footage
matched pair	$2$ rows per item, all else fixed	$32$ @ $2$ fps	availability	availability vs. correctness; causal work
crossed quad	2 questions $\times$ 2 windows	$32$ @ $2$ fps	containment	question-conditioning; visual controls at $0.5$

Appendix

Table A.1: Every streaming geometry, its shape, its frame budget, and its label. The containment window (bold) is the primary geometry: constant span and an evidence-agnostic checkpoint grid. Sampled frame count is constant within each geometry, so no readout can substitute “how many frames” for “which frames”. Reported AUROCs are not comparable across rows: the rows differ both in how strict the label is and in how the not-ready rows are drawn (Appendix A.3 ), so each result in section 4 names the row it belongs to.

label	definition	used for
containment	coverage contains the evidence hull; null on partial	probe fitting
deployed	the same, partial folded into not-ready	policy evaluation
onset	any delivered frame inside an evidence segment	threshold sensitivity

Appendix

Table A.2: Three labels from one cache. Rows with partial coverage are excluded from binary probe fitting, because their state is genuinely intermediate; a deployed policy may not drop a row, so it treats them as not ready.

scored against the benchmark’s evidence label	NExT-GQA	ActivityNet-Cap.	Oops!
AUROC, readiness probe	$0.671$ $[.607,.733]$	$0.752$ $[.693,.804]$	$0.791$ $[.712,.860]$
AUROC, independent rater verdict	$0.778$ $[.731,.822]$	$0.681$ $[.628,.734]$	$0.722$ $[.655,.787]$
rater accuracy	$0.760$	$0.697$	$0.733$
rater $\kappa$	$0.527$	$0.364$	$0.385$
rater sensitivity / specificity	$0.687/0.868$	$0.600/0.761$	$0.700/0.744$
screens ( $n$ )	$300$	$300$	$240$

Appendix

Table B.1: A person shown the delivered frames recovers the benchmark’s own evidence label about as well as the probe does. Intervals overlap on all three evaluations, with the raters ahead on NExT-GQA and behind on the other two. The

240

on Oops! is because

60

of its

300

screens sit at partial coverage and carry no binary label there; the rater-verdict comparisons in Table B.2 use all

300

scored against the raters’ readiness verdict	NExT-GQA	ActivityNet-Cap.	Oops!
AUROC, readiness probe	$\mathbf{0.871}$ $[.832,.909]$	$\mathbf{0.944}$ $[.916,.967]$	$\mathbf{0.923}$ $[.891,.949]$
AUROC, the model’s own confidence	$0.643$	$0.691$	$0.674$
AUROC, an external LLM asked directly	$0.629$	$0.594$	$0.576$
model accuracy, rater-ready screens	$81.7\%$	$71.6\%$	$62.6\%$
model accuracy, rater-not-ready screens	$65.2\%$	$34.6\%$	$34.6\%$
screens ( $n$ ), probe and confidence	$300$	$300$	$300$

Appendix

Table B.2: The readout tracks a human readiness verdict; the model’s own confidence and an external LLM asked the question directly do not. All three rows are continuous scores on the same screens. The lower block shows the raters’ verdict is not a private opinion: the frozen model answers correctly

16.5

36.9

and

28.0

pp more often on the screens they called ready.

rater answer accuracy	pre (far)	pre (near)	partial	first ready	post	overall
NExT-GQA	$0.700$	$0.717$	$0.867$	$0.917$	$0.950$	$0.830$
ActivityNet-Cap.	$0.433$	$0.450$	$0.883$	$0.833$	$0.483$	$0.617$
Oops!	$0.733$	$0.933$	$0.950$	$0.983$	$0.883$	$0.897$

Appendix

Table B.3: The windows sit where the design says they do. Accuracy rises into the evidence-containing windows on all three evaluations. It stays high in the post window on NExT-GQA, where the delivered frames still contain the evidence, and falls on the other two, where by that window the evidence has left the span. This reproduces the non-monotonicity of the labels themselves.

evaluation	held-out unit	fitting and scoring
controlled-composite headline	cue object	Nested within-pool fitting; evaluation on held-out objects.
controlled-composite visual null	pair-preserving group	Both questions on identical frames share a fold; distinct from the headline protocol.
natural within-benchmark probes, including the model axis	video	Video-grouped inner selection and outer evaluation; matched pairs and crossed quads stay together.
held-out-family transfer	footage family	Selection on source families only; the target family’s footage is excluded from fitting.
external transfer	disjoint source and target videos	Source scaler, layer and probe are frozen on the target; a target-fitted probe is a separate comparison.

Appendix

Table C.1: Which groups are held out, and where fitting occurs. The geometry, label and scored row counts for the within-benchmark representation results are listed together in Table E.1 . Table D.1 inventories the source and builder pools, whose counts can differ from the scored subset. Transfer details are in Appendices F and G .

evaluation	probe	final	vision	option	shuffle	clock	conf.	resid. drop	$\Delta$ over conf.
		layer $0$
NExT-GQA	$0.733$	$0.500$	$0.544$	$0.500$	$0.502$	$0.633$	$0.573$	$0.0001$	$+0.106$
ST-Evidence-MCQ	$0.722$	$0.500$	$0.552$	$0.500$	$0.484$	$0.628$	$0.589$	$-0.0001$	$+0.088$
ST-Evidence-Gen	$0.827$	$0.500$	$0.651$	$0.507$	$0.578$	$0.722$	$0.622$	$0.0001$	$+0.180$
the streaming cohort	$0.808$	$0.500$	$0.602$	$0.485$	$0.529$	$0.737$	$0.626$	$0.0000$	$+0.164$
HERBench shot-ID	$0.918$	$0.500$	$0.534$	$0.500$	$0.498$	$0.505$	$0.738$	$0.0000$	$+0.155$

Appendix

Table C.2: The dissociation battery. One row per evaluation, each read against its own comparisons; values are not comparable down a column, because the rows differ in geometry and in label strictness (Appendix A.1 ). The three layer-

0

cells are different objects (Appendix C.3 ): at the final language position, and at the option tokens wherever the option count is constant, layer

0

is the same embedded state on every row and is forced to

0.500

, whereas the vision cell is a baseline that can carry real signal. The two option cells that depart from

0.500

belong to the pools that mix two- and four-option items, where that cell becomes an indicator of the option count rather than a control. A dash marks a cell not available on that pool; such a cell is never assumed to be

0.500

. For shot ordering, the activation cache was not retained. The clock column names a different pool in each row: it is high wherever evidence onset is strongly time-correlated, which is why those rows carry the qualification stated in Appendix E.1 and are not read as readiness results on their own. “Conf.”, “resid. drop” and the last column, which gives the AUROC gain of the readiness score over a confidence-only baseline, all come from the pooled confidence-separation estimator on the same rows ( Table E.3 ), which is why the confidence figure here can differ in the third decimal from the fitted baseline in Table 1 . Every row uses the pixel-cleaned pool that section 4 reports on. The indented row is the same graded shot-boundary prefix ladder as the row above it, restricted to the streaming eligibility cohort; it is a growing prefix, which is why its clock sits with the other prefixes.

pool	footage	evidence timestamps	items	rows	source groups	readiness	model acc.
controlled composites, crossed	HD-EPIC egocentric, P01–P09	cue presence by construction, 17-object library	260 (520 prefixes)	1,040	31 videos, 90 clips	0.5 by constr.	–
matched present / absent windows	HD-EPIC egocentric, P01–P09	official fine-grained action segments	2,400 pairs	4,800	142 videos	0.5 by constr.	–
fine-grained action VQA	HD-EPIC egocentric	the benchmark’s own items	1,200	–	86 videos	–	–
NExT-GQA prefixes	NExT-QA	human temporal grounding	1,000	5,695	473 videos	3,600 / 2,095	0.729 five-opt., 0.695 six-opt.
NExT-GQA crossed quads	NExT-QA	human temporal grounding	1,208 quads (1,196 intact)	4,808	277 videos	0.5 by constr.	–
ST-Evidence-MCQ prefixes	NExT-QA	independent human evidence	1,298	6,756	348 videos	4,400 / 2,356	0.704

Appendix

Table D.1: Source and builder pools, evidence annotations, and sizes. “Rows” counts delivered checkpoints, not items. Readiness gives ready / not-ready counts where recorded and marks the balanced matched and crossed designs. Source groups describe the listed pool, not necessarily the split or bootstrap key. A dash marks a value not recorded. Pixel filtering and its pool-specific effects are detailed in Appendix D.5 ; scored representation pools are identified in Table E.1 .

evaluation	questions	source of the items
HD-EPIC controlled composites	constructed	synthetic cues composited onto real footage
HD-EPIC matched present / absent	constructed	from the official fine-grained action segments
HD-EPIC fine-grained action VQA	shipped	the benchmark’s own items, read verbatim
NExT-GQA	shipped	NExT-QA’s questions with human grounding
ST-Evidence-MCQ	shipped	the benchmark’s own QA and evidence
ST-Evidence-Gen	shipped	the benchmark’s own items, two item filters applied

Appendix

Table D.2: Task provenance. “Constructed” means we authored the question, the answer and the distractors from the source’s annotation; the footage and the evidence timestamps are always the source’s. Readiness labels are derived from source timestamps throughout.

model	HuggingFace id	vision tower	language backbone	dec. lay.	how frames are fed	probed layers
Qwen3-VL-8B-Instruct	Qwen/Qwen3-VL-8B-Instruct	Qwen-ViT	Qwen3-8B	36	native video path, video metadata	0, 14, 18, 23, 29, 32
Qwen3-VL-2B-Instruct	Qwen/Qwen3-VL-2B-Instruct	Qwen-ViT	Qwen3-2B	–	native video path	0, 11, 14, 18, 22, 25
Qwen3-VL-32B-Instruct	Qwen/Qwen3-VL-32B-Instruct	Qwen-ViT	Qwen3-32B	–	native video path	0, 26, 32, 42, 51, 58
InternVL3.5-8B [ Wang et al., 2025b ]	OpenGVLab/InternVL3_5-8B-HF	InternViT	Qwen3-8B	36	multi-image, one $448$ px tile per frame	0, 14, 18, 23, 29, 32
LLaVA-OneVision-7B [ Li et al., 2025 ]	llava-hf/llava-onevision-qwen2-7b-ov-hf	SigLIP [ Zhai et al., 2023 ]	Qwen2-7B	28	native video path, no metadata	by depth fraction
LLaVA-OneVision-1.5-8B [ An et al., 2025 ]	lmms-lab/LLaVA-OneVision-1.5-8B-Instruct	–	Qwen3-8B	–	–	0, 14, 18, 23, 29, 32

Appendix

Table D.3: Every model evaluated. Layers are probed at fractions of each model’s own depth rather than at transcribed indices, because decoder depth differs across architectures; the indices shown are the resolved ones where they are recorded, and a dash marks a value not recorded. Qwen3-VL-8B-Instruct [ Qwen Team, 2025 ] is the default everywhere unless a result names another.

geometry	frames	rate	span	stride	decode resolution
controlled composites	16	2 fps	fixed 8 s	–	448 px long side
growing prefixes	32	–	$[0,t]$	evidence-relative ends	448 px long side
containment windows	32	2 fps	16 s	free 2 s	–
short windows	8	4 fps	2 s	0.5 s	–
sparse bursts	16	2–3 fps	3–6 bursts of 1–2 s	one block	–

Appendix

Table D.4: Frame budgets per geometry. Frame count is constant across every row of a geometry. Prefixes sample uniformly across

[0,t]

, except on the segment-aligned pools, where sampling guarantees at least two frames per revealed evidence segment. A dash marks a value not recorded.

pool	dropped	of	rate
NExT-GQA prefixes	115 prefixes (21 items entirely)	5,695	2.0%
ST-Evidence-MCQ prefixes	210 prefixes	6,756	3.1%
ST-Evidence-Gen prefixes	5 prefixes	4,346	0.1%
matched present / absent windows	0	4,800	0%
model-axis short windows	0	–	0%
Oops! short windows	–	–	0.3%

Appendix

Table D.5: Pixel quality-control drop counts. The filter is model-independent and runs before any outcome exists. Only the last row has a rate large enough to change what a pool can support.

pool	geometry	label	rows / clusters	probe AUROC	shuffle	L $0$ @vision	clock
controlled composites	controlled composite	cue presence	1,040 / 17 obj.	$0.985$ $[.965,.999]$	$0.524$	$0.500$	–
matched windows	matched pair	availability	4,800 / 142	$0.827$ $[.812,.840]$	$0.495$	$0.495$	$0.506$
the same, InternVL3.5-8B	matched pair	availability	4,800 / 142	$0.815$ $[.802,.829]$	$0.504$	$0.510$	$0.506$
NExT-GQA prefixes	growing prefix	onset	5,580 / 458	$0.733$ $[.715,.749]$	$0.502$	$0.544$	$0.633$
ST-Evidence-MCQ prefixes	growing prefix	onset	6,546 / 338	$0.722$ $[.706,.739]$	$0.484$	$0.552$	$0.628$
ST-Evidence-Gen prefixes	growing prefix	$\mathrm{seen}/K$	4,346 / 1,529	$0.827$ $[.813,.840]$	$0.578$	$0.651$	$0.722$

Appendix

Table E.1: Every decodability cell with its own comparisons. All AUROCs are nested, cluster-held-out, and read at the final prompt token. The column is not a ranking: the rows differ in geometry ( Table A.1 ), in label strictness ( Table A.2 ) and in where the not-ready rows are drawn from (Appendix A.3 ), so each cell is interpretable only against the comparisons printed beside it. The shuffle is the cluster-grouped label shuffle measured on that pool, not assumed to be

0.500

; on the graded prefixes it is

0.578

, a base-rate artefact, so that row reads “

0.827

against

{\sim}0.58

”. The clock is a fitted elapsed-time baseline on the same rows: it is informative by construction on a growing prefix, exactly

0.500

on a balanced crossed pool, and

0.956

on the shot-ordering prefixes , where evidence onset is strongly time-correlated and the readiness reading is therefore not creditable on its own. Layer

0

at the final language position is exactly

0.500

in every row by construction; the visual layer-

0

baseline, which can carry real signal, is the adjacent column. It is forced to

0.500

on the crossed quads by the design and is quoted on the controlled composites under a pair-preserving grouping (Appendix C.4 ); the dash on the shot-ordering prefixes marks a cell that is not measurable, its activation cache not having been retained, rather than one assumed to be

0.500

, and the composited stimuli carry no elapsed-time column at all. Row counts identify the reported decoding pools; pixel quality control is documented in Appendix D.5 , and grouping in Table C.1 . Oops! is quoted in the both-sided negative regime; the before-only regime reads

0.919

but carries a clock at

0.950

		paired drop on byte-identical frames, $624$ pairs
model	language backbone	identical question	paraphrase	distinct question	register-matched
Qwen3-VL-2B	Qwen3	$+0.196$	$+0.448$	$\mathbf{+0.594}$	$+0.477$
Qwen3-VL-8B	Qwen3	$+0.080$	$+0.153$	$\mathbf{+0.778}$	$+0.732$
Qwen3-VL-32B	Qwen3	$+0.300$	$+0.579$	$\mathbf{+0.418}$	$+0.680$
LLaVA-OV-1.5-8B	Qwen3	$+0.305$	$+0.435$	$\mathbf{+0.708}$	$+0.431$
GLM-4.1V-9B	GLM-4	$+0.003$	$+0.140$	$\mathbf{+0.269}$	$+0.458$

Appendix

Table E.2: Replacing the question collapses the readout in every model tested. Each cell is the mean paired drop in readiness score between a row and the same row re-asked with a donor question, on the same

624

evidence-carrying pairs and byte-identical frames. Distance increases left to right: another item’s identical question, a paraphrase from a different corpus, and a specific question from a pool carrying one question per item, with a register-matched build of that last donor in the final column. Under a genuinely distinct question every model drops by

+0.27

+0.78

, on

64

–

98\%

of pairs. Register is not what drives it: Pixtral’s raw and register-matched drops agree to four decimals (

0.3252

0.3253

) and Idefics3’s register-matched drop is the larger of its two, so the two builds should be read as a range rather than a point.

pool	readiness AUROC	residualized on conf.	drop	confidence alone	increment
controlled composites, factorial	$0.9317$	$0.9317$	$\phantom{-}0.0000$	$0.534$	$+0.397$
controlled composites, crossed	$0.9920$	$0.9920$	$\phantom{-}0.0000$	$0.845$	$+0.147$
matched windows	$0.7701$	$0.7702$	$-0.0001$	$0.636$	$+0.135$
the same, InternVL3.5-8B	$0.7506$	$0.7506$	$\phantom{-}0.0000$	$0.567$	$+0.184$
NExT-GQA prefixes	$0.6793$	$0.6792$	$+0.0001$	$0.573$	$+0.106$
ST-Evidence-MCQ prefixes	$0.6761$	$0.6762$	$-0.0001$	$0.589$	$+0.088$

Appendix

Table E.3: Confidence explains none of the readout. Each row residualizes the readiness score on the model’s output confidence over the same examples and re-scores; the drop is raw minus residualized, and the largest movement in either direction is

0.0002

. These cells use a pooled estimator, and the ActivityNet row is the containment-label build, so the levels are not comparable with Table E.1 and only the drop and the increment are read here. The increment is the AUROC gain of the activation score over a confidence-only baseline on the same rows.

		Oops! windows		ActivityNet
signal	what it is	acc.	probe $-$ this	acc.	probe $-$ this
sc_agree	share of $K$ sampled answers, each with a rationale, agreeing with the mode	$0.417$	$+0.260$ $[.205,.314]$	$0.393$	$+0.253$ $[.208,.301]$
sc_neg_entropy	negative entropy of that sampled answer distribution	$0.417$	$+0.260$ $[.205,.314]$	$0.393$	$+0.253$ $[.208,.301]$
se_neg	semantic entropy: $K$ open-ended answers clustered by bidirectional entailment	$0.478$	$+0.199$ $[.144,.253]$	$0.413$	$+0.233$ $[.191,.276]$
se_neg_lennorm	the same, length-normalized	$0.487$	$+0.189$ $[.135,.244]$	$0.427$	$+0.220$ $[.177,.264]$
se_neg_loose	the same, one-direction clustering	$0.446$	$+0.231$ $[.176,.288]$	$0.423$	$+0.223$ $[.180,.267]$

Appendix

Table E.4: Every alternative signal, and the probe’s margin over it. Selection accuracy is measured at an identical deadline, so the comparison is latency-matched by construction; the probe reads

0.676

and

0.647

on these substrates and answering only at the end reads

0.487

and

0.358

. Intervals are video-cluster bootstraps of the paired difference. Every alternative loses on both substrates; adding the supervised head back to the probe moves selection by

+0.003

and

-0.002

, both intervals spanning zero. Signal names match 1(b) .

$K$	rows	ready	not ready	AUROC
$1$	2,309	1,877	432	$0.852$
$2$	1,121	667	454	$0.787$
$3$	379	174	205	$0.716$
$4$	211	81	130	$0.699$
$5$	156	51	105	$0.729$
$6$	75	24	51	$0.555$

Appendix

Table E.5: Per-

K

strata, each with its minority-class count. Read on the pixel quality-controlled build of the graded prefixes, which reads

0.831

[0.817,0.844]

overall. The minority class is in bold and falls from

432

rows at

K=1

24

K=6

; the

K=6

cell rests on

24

positives and is printed for completeness rather than read as a decline. Strata above

K=6

are not tabulated for the same reason. Within each

K

the mean score rises monotonically with the number of segments seen for

K=1

–

5

, which is

96\%

of the rows, with within-

K

Spearman correlations of

0.489

0.521

0.505

0.599

and

0.614

backbone	model	regime	AUROC	$95\%$ CI	L $0$ @final	L $0$ @vision	shuffle	clock track
Qwen3	Qwen3-VL-2B	before	$0.8934$	$[0.8727,\,0.913]$	$0.500$	$0.8194$	$0.4920$	$0.1039$
		near	$0.8930$	$[0.8722,\,0.9126]$	$0.500$	$0.8085$	$0.5078$
		after	$0.8648$	$[0.8364,\,0.8923]$	$0.500$	$0.6974$	$0.4986$
		both	$0.8495$	$[0.8277,\,0.8705]$	$0.500$	$0.6828$	$0.5021$
Qwen3	Qwen3-VL-8B	before	$0.9190$	$[0.9022,\,0.9356]$	$0.500$	$0.7848$	$0.4925$	$0.1065$
		near	$0.9106$	$[0.8925,\,0.9275]$	$0.500$	$0.7834$	$0.5006$

Appendix

Table F.1: The full model axis: seven models

\times

four negative-sampling regimes. Readiness AUROC is nested cross-validated with video-disjoint folds; the interval is a video-cluster bootstrap. The final-language-position layer-

0

column is

0.500

in all

28

cells and is forced to be: at layer

0

a fixed text position holds the same embedded token on every row, so the readout is degenerate before any frame is consulted. The visual layer-

0

column can carry real signal because it reads raw patch embeddings. It spans

0.5790

–

0.8194

, and the language-position readout clears it in every cell, by at least

0.0740

. The within-video shuffle stays inside

0.4669

–

0.5206

throughout. The fitted elapsed-time clock is a property of the regime rather than the model:

0.9497

( before ),

0.8663

( near ),

0.9069

( after ),

0.5271

( both ). Clock tracking is one number per model, on the before

\rightarrow

both contrast.

Panel of Figure 24 — Figure F.1: Readiness is neither a stopwatch nor an uncertainty estimate. (a) Readiness AUROC against the AUROC of a fitted elapsed-time clock, one point per model $\times$ negative regime. Colours identify Qwen3-VL-2B (blue), Qwen3-VL-8B (red), Qwen3-VL-32B (green), LLaVA-OV-1.5-8B (orange), GLM-4.1V-9B (violet), Idefics3-8B (brown) and Pixtral-12B (teal). Before-only sampling makes the clock ( $0.950$ ) stronger than every probe. With both-sided sampling the clock falls to $0.527$ , while all seven probes remain informative: their AUROCs change much less than the clock’s. (b) Eleven modern uncertainty estimators, plus a supervised head fitted over all of them ( composite ), compared on fixed-deadline answer selection. Points are Readiness Gating’s margin over each baseline with a cluster-bootstrap interval, using earliest-window tie-breaking; Appendix E.4 reports the favourable-tie-break comparison. Every estimator loses on both substrates. The last row compares the combined head ( composite+probe ) with Readiness Gating alone: adding the baseline scores yields no detectable selection gain, with both intervals spanning zero.

held-out family	transfer AUROC	$95\%$ CI	own AUROC	retention	L $0$ @vision	shuffle	clock
NExT-GQA	$0.7073$	$[0.6893,\,0.7261]$	$0.7464$	$0.9476$	$0.5139$	$0.5006$	$0.4586$
ST-Evidence-MCQ	$0.7153$	$[0.6904,\,0.7425]$	$0.7448$	$0.9604$	$0.5310$	$0.4976$	$0.4548$
ST-Evidence-Gen	$0.7501$	$[0.7211,\,0.7808]$	$0.7692$	$0.9752$	$0.5389$	$0.5208$	$0.5446$
ActivityNet	$0.6525$	$[0.6335,\,0.6745]$	$0.7637$	$0.8544$	$0.4974$	$0.5040$	$0.4942$
Oops!	$0.7610$	$[0.7382,\,0.7843]$	$0.8740$	$0.8707$	$0.5507$	$0.5022$	$0.5278$
mean retention, footage-group hold-out				$\mathbf{0.9217}$

Appendix

Table F.2: One probe, fitted without a family’s footage, read on that family. Rows are the footage-group hold-out, in which NExT-GQA and ST-Evidence-MCQ are removed together because they share

221

videos. Retention is transfer AUROC over the family’s own fitted AUROC, the comparable quantity given that the family ceilings themselves run

0.7448

–

0.8740

. Holding out benchmarks singly raises the mean to

0.9491

, and the whole of that difference is the two NExT-QA families, whose retention rises by

0.0575

and

0.0799

once their shared footage is allowed back into the training union. Every cell sits fully inside the source union’s delivered frame-density envelope.

rank	aligned (median)	random-rotation null	row-shuffled alignment	pairs beating own null
$4$	$0.5135$	$0.4932$	$0.5109$	$32/42$
$8$	$0.6733$	$0.4822$	$0.4801$	$41/42$
$16$	$0.6938$	$0.5175$	$0.5114$	$42/42$
$32$	$0.6935$	$0.4940$	$0.5237$	$42/42$
$64$	$0.6925$	$0.5060$	$0.4925$	$42/42$
$\mathbf{128}$	$\mathbf{0.7134}$	$\mathbf{0.4963}$	$0.5009$	$\mathbf{42/42}$

Appendix

Table F.3: The cross-model rotation, with both controls read at the same rank. Medians over the

42

ordered model pairs on the frozen Oops! manifest. Rank

128

is the primary rank; reading an aligned value against a null taken from a different rank is exactly the error these columns exist to prevent. In un-aligned transfer, the source probe is applied to the target’s activations with no map at all. It reads

0.4668

over the

14

ordered pairs whose two models share

d_{\text{model}}

, i.e. at chance. Recovering the basis therefore needs paired activations on shared inputs, and this is not a shippable probe.

cell	base rate	carried-over	$J$ at $k=8$	Platt at $k=8$	oracle threshold	constant predictor
OVO-Bench (window)	$0.1306$	$0.1377$	$0.6955$	$\mathbf{0.8648}$	$0.8681$	$0.8694$
Fine-grained Action	$0.2733$	$0.5764$	$0.6456$	$\mathbf{0.7277}$	$0.7311$	$0.7267$
NExT-GQA ladder	$0.1968$	$0.6744$	$0.6515$	$\mathbf{0.8006}$	$0.8062$	$0.8032$
ActivityNet ladder	$0.1850$	$0.4715$	$0.5982$	$\mathbf{0.8099}$	$0.8151$	$0.8150$

Appendix

Table G.1: Accuracy under a threshold refitted on eight labelled target videos. Direction, layer, and scaler are frozen throughout; only the threshold moves. The two refitting arms differ in objective: our intercept-only variant of Platt scaling [ Platt, 2000 ] moves the crossing point, whereas Youden’s

J

[ Youden, 1950 ] maximizes TPR

-

FPR, which is not an accuracy objective and loses by roughly

0.17

on the same eight labels (

0.6955

[0.6083,0.7854]

against

0.8648

[0.8473,0.8791]

on the first row). The last column is the accuracy of always predicting the majority class, and it is the number that bounds how much this repair can be worth.

Panel of Figure 28 — Figure G.1: The ranking transfers; the operating point does not. Reliability of the carried-over probe on the external benchmark over $3{,}913$ scored rows, against the diagonal. The probe concentrates almost all mass above $0.9$ , placing $3{,}093$ of $3{,}913$ rows there despite the pool’s far lower positive rate. At the carried-over threshold of $0.5$ , almost every row is therefore called ready. The curve is nonetheless monotone at the top end, which is why the ordering survives while the calibration does not (ECE $0.808$ ).

Panel of Figure 29 — Figure H.1: Changing distractors moves headroom and policy gain together. (a) ActivityNet-Captions conditions hold items, videos, windows, labels, frames and answer positions fixed, changing only the distractors. The attainable band moves from $17.5$ to $34.5$ pp and $\Delta^{*}$ from $+1.10$ to $+4.66$ pp; error bars are the reported confidence intervals. (b) The same two conditions (red) enter the $26$ -configuration comparison over ten substrates, each configuration entering once on its full item pool. The plotted quantity is the seconds-matched $\Delta^{*}$ of gated stopping, not the selection edge quoted in subsection 4.4 , which is why some configurations sit below zero. The fitted slope is $+0.148$ pp per band pp. Three high-band configurations carry much of the pooled relationship; the remaining $23$ give $r^{2}=0.479$ . These are configuration-level observations, not $26$ independent dataset replications. The band tracks gain but does not determine it.

system	head	shape	base model
MMDuet	relevance , informative	$2\times 3584$ ea.	LLaVA-OneVision (Qwen2-7B)
Dispider	silent	$1\times 1536$	Qwen2
VideoLLM-online	streaming-EOS logit	unembedding row	Llama-3-8B

Appendix

Table I.1: The released triggers are linear residual-stream readouts, the same kind of object as our probe. MMDuet ships as a

0.13

GB adapter and publishes a

2

fps rate; Dispider’s head is embedded in its checkpoint’s root shards and decides once per

16

s, and its sign convention is that a high score means respond . VideoLLM-online was obtained but never run, so no number is asserted for it anywhere in this paper.

quantity	Qwen3-VL-8B	InternVL3.5-8B
commitment excess over a magnitude-matched control	$+0.3924$ $[+.331,+.454]$	$+0.3495$ $[+.306,+.395]$
deferral-to-answer flips, treatment vs. control	$101$ vs. $3$	$91$ vs. $7$
exceeds all $35$ alternative directions	yes, $p_{\max T}=0.0278$	yes, same $p$
dose–response Spearman over four doses	$1.0$	$0.8$
answer-content equivalence test	passes	passes
replicates on evidence-present-but-wrong	yes	yes

Appendix

Table J.1: Adding the readiness direction changes answer commitment. Nothing was re-selected for the second model: it ran through the locked cell chosen on the first. The second model’s dose curve is not fully monotone, which is the one qualitative difference between them.

Explore similar work

Sep 29, 2026cs.CV

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.

Xiang Hu, Jiazuo Yu, Lu Zhang +2

IIAU Lab, Dalian University of Technology

Streaming Video Understanding Relevance

Ask PDF

Feb 11, 2026cs.CV

Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models

Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM failures follow two pathways: some visual signals are never reliably encoded, while others are encoded but overridden by model priors. We introduce REVEAL, a diagnostic stress-test benchmark for quantifying when and why VidLMs under-use visual evidence. REVEAL contains five controlled probes: camera-motion sensitivity, cross-frame integration, video sycophancy, language-only shortcuts, and temporal expectation bias. Together, they test whether models encode basic video signals, combine evidence across frames, and preserve visual evidence against user assertions, language cues, and learned event expectations. Across 12 VidLMs we find systematic failures along both pathways, with most models falling below chance on the binary and six-way probes that humans solve at 78--100% accuracy. Under assertive prompts, a model's output distribution becomes nearly invariant to whether it is shown a real video or random noise, making visual evidence effectively causally inert. We further carry out mechanistic probes to identify where these failures arise in the model pipeline and why visual evidence is lost. REVEAL provides a scalable, human-verified framework for moving beyond aggregate scores toward structured, reproducible evaluation of multimodal reliability.

Sethuraman T, Savya Khosla, Aditi Tiwari +12

University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6

Video-Language Models Visual Evidence Stress Testing Prior Knowledge Integration Sycophancy

Ask PDF

Apr 2, 2026cs.CV

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question-answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of cross-verification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data are publicly released.

Jiahao Meng, Yue Tan, Qi Xu +9

PKU · WHU · CASIA +2

Long Video Question Answering

Ask PDF