Supervising the Visible State: Separating Representation from View Selection in Streaming Video

Yihao Hu; Pan Wang; Yitong Lin; Zhihao Wen; Deyuan Liu; Hang Wang; Mingda Wan; Yiwen Zhao

cs.CVOct 26, 2026

Supervising the Visible State: Separating Representation from View Selection in Streaming Video

Authors: Yihao Hu, Pan Wang, Yitong Lin, Zhihao Wen, Deyuan Liu, Hang Wang, Mingda Wan, Yiwen Zhao,

Organizations: Westlake University · USTC · ZJU · SMU ·

Abstract

An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.

Figures & tables

Figure 1: Context-centric streaming versus supervising the visible state. Top: recent-frame models see little history and history-centric models leave temporal composition implicit; both inherit answer-level supervision, whereas we supervise the visible state and acquire causal views on demand. Bottom: counting movie inserts, the recent view misses earlier ones and historical sampling skips the first; retrieving it restores the count. This example does not separate the two levers.

Figure 2: Overview. Phase 1, ViSS (Section 4.1 ): probes of event identity, order, accumulation, and relation change (illustrated), together with sufficiency, train a shared representation in one epoch; recent-4 is the dominant training input and the evaluation interface (longer views: Appendix B.3 ). Phase 2, TVS (Section 4.2 ): the checkpoint is frozen; the event view is acquired when the recent margin is low, the history view when the two disagree, and a calibrated margin rule keeps one proposal.

Proprietary MLLMs
		Real-Time Visual Perception							Backward Tracing				Forward Active Responding
Model	Frames	OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	REC	SSR	CRR	Avg.	Overall
Human	–	93.96	92.57	94.83	92.70	91.09	94.02	93.20	92.59	93.02	91.37	92.33	95.48	89.67	93.56	92.90	92.81
Gemini 1.5 Pro	1 fps	85.91	66.97	79.31	58.43	63.37	61.96	69.32	58.59	76.35	52.69	62.54	35.53	74.24	61.67	57.15	63.00
GPT-4o	64	69.80	64.22	71.55	51.12	70.30	59.78	64.46	57.91	75.68	48.66	60.75	27.58	73.21	59.40	53.40	59.54
Open-source offline MLLMs

Table 1: Full-split OVO-Bench accuracy by official task and group. Overall is the official task-then-group macro; task names are listed in Appendix C.4 . Published rows keep their own protocols (Appendix C.4 ) and serve as context; the Ours rows share one backbone, scorer, and recent-4 starting view and carry the paper’s claims. Pale blue highlights the learned representation; darker blue denotes the complete adaptive system. For TVS, Frames is the per-call maximum.

Proprietary MLLMs
Model	Frames	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	ALL
Human	–	89.47	92.00	93.60	91.47	95.65	92.52	88.00	88.75	89.74	91.30	91.46
Gemini 1.5 Pro	1 fps	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69
GPT-4o	64	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28
Claude 3.5 Sonnet	20	80.49	77.34	82.02	81.73	72.33	75.39	61.11	61.79	69.32	43.09	72.44
Open-source offline MLLMs

Table 2: Full-split StreamingBench real-time accuracy by official task. Column abbreviations follow the benchmark taxonomy. Published rows keep their own protocols and serve as context; the Ours rows share one backbone, scorer, and recent-4 starting view. Pale blue highlights the learned representation; darker blue denotes the complete adaptive system. For TVS, Frames is the per-call maximum, not the union across calls.

Figure 3: What supervision and view selection each change, on one Qwen3-VL-8B checkpoint. (a) OVO-Bench task accuracy change from ViSS under fixed recent-4 evidence (

\Delta_{\mathrm{rep}}

) and from TVS with the checkpoint frozen (

\Delta_{\mathrm{access}}

). (b) Gain over recent-4 versus processed frames per query (filled: recent-4, TVS, oracle; open: fixed views, max-confidence). (c) Selection-error attribution conditioned on oracle coverage: mostly a correct answer that was held and then replaced.

Supervision	SB	OVO
Base	80.52	58.97
Strict-matched generic	84.97	63.48
ViSS w/o sufficiency	84.89	63.52
ViSS	84.77	65.21

Table 3: Controlled analysis of state supervision and view selection. (a) fixes recent-4 access; (b) fixes the ViSS checkpoint.

Figure 4: One query per lever. Left: same four recent frames, two checkpoints (control checkpoint: B, ViSS: A). Right: one frozen ViSS checkpoint, three causal views; margins in parentheses, full trace in Figure 5 .

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

View	Recent support	Historical support	Frames
recent $x_{e,0}$	4 frames from the latest 4 seconds	none	4
event $x_{e,1}$	the same recent 4	up to 16 ranked frames, standard resolution	20
compact history $x_{e,2}$	8 frames from the latest 8 seconds	up to 24 ranked frames, reduced resolution	32

Appendix

Table 4: Complementary causal views. Counts are per-call maxima; short prefixes may supply fewer frames. Recent support uses the standard image path, while compact historical images have maximum edge 224 pixels.

Component	Setting
Historical ranking	$16\times 16$ RGB descriptors; novelty, trajectory curvature, and recency with weights $4{:}9{:}2$
Candidate construction	up to 64 prefix-grid positions, plus reserved recent IDs after filtering
Calibration	300 queries from 300 source-video-disjoint nonbenchmark videos
Acquisition threshold	$\tau_{a}=0.9976896$
Relative-margin tolerance	$\tau_{c}=-0.5059$
Inference budget	at most 3 calls; at most 32 frames in any one call

Appendix

Table 5: Fixed causal-view and search configuration used on both benchmarks.

Family	Visible-state target	Rows
source-aligned anchors	general causal video understanding and capability retention	3,594
repetition-counting probes	accumulated event count over an ordered visual prefix	2,172
prefix-32 state probes	event state, temporal order, and accumulation over longer prefixes	1,012
sufficiency anchors	whether the available causal evidence supports an answer	116
capability probes	backward state retrieval, action sequence, and counting	783
Total		7,677

Appendix

Table 6: Supervision families in the sealed 7,677-example mixture.

Source and input	Supervision question	Direct target
RepCount annotations; 16 frames	The frames are chronological causal observations. Count how many complete situp repetitions have fully finished by this point. An incomplete motion does not count. Reply with one integer only.	2
Teacher capability probe; 12 frames	What gesture did the woman make with her right hand before she began speaking with her hand open and palm facing up?	D : She pointed her index finger upwards.

Appendix

Table 7: Actual training-record examples. Question text and targets are preserved; common chat instructions and multiple-choice distractors are omitted. Frame counts describe the serialized input, not the recent-4 evaluation view.

Setting	Value
Model and input
Initialization	Qwen3-VL-8B-Instruct; fresh run from public weights
Trainable modules	language model and multimodal projector; vision tower frozen
Input limits	8,192 tokens; 200,704 pixels per image
Loss	assistant targets only; serialized example weights
Data and optimization

Appendix

Table 8: Reproducibility settings for the sealed ViSS training run.

Training attribute	Strict-matched generic	ViSS
Rows	7,677	7,677
Visual inputs and exposure	identical	identical
Weights, stages, and order	identical	identical
Optimization, initialization, seed	identical	identical
Question–target formulation	generic temporal QA	visible-state probes
StreamingBench	84.97	84.77

Appendix

Table 9: Strict-matched supervision control under recent-4 evaluation.

Seed pair	StreamingBench	OVO-Bench	HLD
1	2,120 / 2,115	63.48 / 65.21	125 / 140
2	2,117 / 2,113	63.21 / 64.90	124 / 138
3	2,116 / 2,112	63.63 / 65.37	126 / 140
Mean score	84.88 / 84.70	63.44 / 65.16	67.20 / 74.91
Mean paired difference	$-0.17$	$+1.72$	$+7.71$

Appendix

Table 10: Three paired training seeds, with Generic / ViSS in each cell. Seed rows give correct counts for StreamingBench (

N=2{,}495

) and HLD (

N=186

), and official macro scores for OVO. Mean scores are percentages; mean paired differences are percentage points computed before rounding.

Configuration	Rows	SB	OVO	CT	REC	HLD
full ViSS	7,677	84.77	65.21	44.04	21.20	75.27
without sufficiency	7,561	84.89	63.52	46.63	20.63	70.97

Appendix

Table 11: Sufficiency sensitivity under recent-4 evaluation.

Fixed one-view controls
	StreamingBench			OVO-Bench
Access policy	Acc.	Calls/q	Frames/q	Macro	Calls/q	Frames/q
recent only	84.77	1.00	4.00	65.21	1.00	4.00
event only	85.69	1.00	19.51	67.67	1.00	19.79
history only	86.45	1.00	29.97	68.84	1.00	30.54
static history-17	85.69	1.00	17.00	–	–	–

Appendix

Table 12: Accuracy–cost trade-off under the frozen ViSS representation.

Group	Task	Base	ViSS	$+TVS{}$	Oracle
RT	Optical character recognition (OCR)	95.30	95.30	96.64	97.32
RT	Attribute recognition (ATR)	82.76	84.48	82.76	87.93
RT	Object recognition (OJR)	82.61	83.15	79.89	87.50
RT	Action recognition (ACR)	80.73	82.57	80.73	87.16
RT	Future prediction (FPD)	73.27	75.25	80.20	83.17
RT	Spatial understanding (STU)	73.60	72.47	67.42	78.09

Appendix

Table 13: Task-wise OVO accuracy across representation and access stages. Task names follow Niu et al. [33] ; RT, Bwd, and Fwd denote real-time, backward, and forward groups. All four stages use the official scoring rule.

Diagnostic	Interpretation	StreamingBench	OVO-Bench
View recoverability	A broader causal view contains a correct proposal	43.7% (166/380)	37.1% (447/1,206)
Policy recovery	TVS returns the correct answer when one is available	63.3% (105/166)	71.1% (318/447)
Recent-answer preservation	TVS retains an initially correct recent proposal	96.9% (2,049/2,115)	92.3% (1,688/1,829)

Appendix

Table 14: Conditional diagnostics of view availability and proposal selection. Counts underlying each rate are shown in parentheses.

Failure mechanism	Trace-level criterion	StreamingBench	OVO-Bench
Acquisition miss	No correct proposal is computed along the deployed path	32 (25.2%)	53 (19.6%)
Acceptance miss	A correct proposal is computed but never retained	10 (7.9%)	23 (8.5%)
Replacement error	A retained correct proposal is subsequently replaced	85 (66.9%)	194 (71.9%)
Total	Errors containing at least one correct replayed proposal	127	270

Appendix

Table 15: Attribution of the remaining query-level gap to the three-view oracle. Percentages are conditioned on oracle-gap cases.

Figure 5: Qualitative case study of causal view acquisition and proposal commitment. Recent-4 omits the event in which the cat takes the teddy bear and proposes D. The event view exposes the decisive evidence and proposes A; the compact-history view independently supports A, which the acquire–commit policy preserves.

Acquired view	Frames	Proposal	Margin	Retained
recent	4	D	0.5084	D
event	20	B	0.1228	B
compact history	29	C	0.2416	C

Appendix

Table 16: A recorded replacement error. The recent answer is correct, but both subsequent proposals pass the calibrated relative-margin criterion. Margins are rounded for display.

Figure 6: Corrective historical evidence is not accessed. Four actual frames per view are displayed; complete inputs contain 4 recent, 20 event, and 6 history frames. Earlier frames show all three paintings, while recent-4 loses the left neighbour. All proposals use the same frozen ViSS checkpoint; only recent-4 is executed by TVS.

Figure 7: All-view failure with ambiguity about what to count. Four actual frames per view are displayed; complete inputs contain 4 recent, 20 event, and 21 history frames. The same frozen ViSS checkpoint is used throughout. Exact traces for both cases are provided in supplementary/viss_failure_cases.json ; no new proposals are generated.

Figure 8: Accumulating movie insertions. At 242 seconds, the recent-frame control predicts two insertions and the historical-search control predicts three. Historical sampling omits the first insertion block. Our causal-access run exposes this missing evidence and returns four, matching the reference. Each insertion block may contain several consecutive film excerpts.

Figure 9: Following the current lesson topic. The observed lesson moves from perimeter to area and area formulas. Both controls predict rectangle perimeter (A), while our run predicts the annotated triangle-area option (D). The displayed prefix establishes the current area topic; no future frames are exposed to the model.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding
Aug 17, 2026Keming Wu, Baoyi Wang, Kaichen Zhang +7Streaming Video Understanding Streaming Inference
Tsinghua University · Zhejiang University · The University of Hong Kong +3
Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Apr 20, 2026Kecheng Zhang, Zongxin Yang, Mingfei Han +6Streaming Video Understanding Efficient VLM Inference
University of Science and Technology of China · Harvard University · Department of CV, MBZUAI +3
Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning
Sep 29, 2026Ziheng Huang, Yicheng Bao, Xueheng Li +8Streaming Video Understanding Video QA
East China Normal University · University of Science and Technology of China · Xiamen University

Aug 17, 2026J/K move · Enter open · S save

StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding

Keming Wu, Baoyi Wang, Kaichen Zhang, Xiang An+6

Tsinghua University · Zhejiang University · The University of Hong Kong+3

Streaming Video Understanding Streaming Inference Temporal Video Understanding Knowledge Distillation On-Policy Distillation

Open paper AskPDF arXiv

Streaming video understanding demands direct responses from the causally observed prefix of an unfolding video. Existing systems add inference-time memory, retrieval, and compression, yet a training-free sliding-window baseline already matches them. We therefore fix a memory-free recent-window protocol and ask how far post-training alone can go. Reinforcement learning with verifiable rewards fits this regime poorly, encouraging long ``think-then-answer'' generations, while on-policy distillation (OPD) supplies dense token-level teacher supervision on student trajectories but is stable only when both models train in thinking mode. These observations lead to StreamOPD, a recipe combining verifiable streaming-video data, thinking-mode OPD, and instruct-mode deployment. It raises StreamingBench from

77.9\%

83.9\%

---within

0.3

points of the 9B teacher---and improves OVO-Bench excluding its hallucination-detection subtask (HLD) by

9.1

points under unchanged inference. As a teacher-privilege extension, Spatio-Temporal CueGate (ST-CueGate) aggregates cue-versus-no-cue teacher likelihood ratios into a group-relative response score that reweights OPD. It reaches

71.9\%

on OVO-Bench (excluding HLD) and

64.9\%

on Video-MME, and is the only variant that stays above the base model on all four benchmarks. Replacing the teacher with a frozen copy of the student's initial policy---on-policy self-distillation---retains most of these gains and lifts HLD to

57.0\%

, above both the untrained student and the 9B teacher, so abstention loss is not intrinsic to the recipe. We provide a transparent and reproducible reference for open-source streaming-video research.

Abstract

Streaming Video Understanding

Explore similar work

StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding

Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions

Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning