StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding

Keming Wu; Baoyi Wang; Kaichen Zhang; Xiang An; Zuhao Yang; Sudong Wang; Haowei Zhu; Tingxuan Huang

cs.CVAug 17, 2026

StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding

Authors: Keming Wu, Baoyi Wang, Kaichen Zhang, Xiang An, Zuhao Yang, Sudong Wang, Haowei Zhu, Tingxuan Huang,

Organizations: Tsinghua University · Zhejiang University · The University of Hong Kong · LMMs-Lab ·

Abstract

Streaming video understanding demands direct responses from the causally observed prefix of an unfolding video. Existing systems add inference-time memory, retrieval, and compression, yet a training-free sliding-window baseline already matches them. We therefore fix a memory-free recent-window protocol and ask how far post-training alone can go. Reinforcement learning with verifiable rewards fits this regime poorly, encouraging long ``think-then-answer'' generations, while on-policy distillation (OPD) supplies dense token-level teacher supervision on student trajectories but is stable only when both models train in thinking mode. These observations lead to StreamOPD, a recipe combining verifiable streaming-video data, thinking-mode OPD, and instruct-mode deployment. It raises StreamingBench from $77.9\%$ to $83.9\%$ ---within $0.3$ points of the 9B teacher---and improves OVO-Bench excluding its hallucination-detection subtask (HLD) by $9.1$ points under unchanged inference. As a teacher-privilege extension, Spatio-Temporal CueGate (ST-CueGate) aggregates cue-versus-no-cue teacher likelihood ratios into a group-relative response score that reweights OPD. It reaches $71.9\%$ on OVO-Bench (excluding HLD) and $64.9\%$ on Video-MME, and is the only variant that stays above the base model on all four benchmarks. Replacing the teacher with a frozen copy of the student's initial policy---on-policy self-distillation---retains most of these gains and lifts HLD to $57.0\%$ , above both the untrained student and the 9B teacher, so abstention loss is not intrinsic to the recipe. We provide a transparent and reproducible reference for open-source streaming-video research.

Figures & tables

Figure 1: Teacher/student training-mode pairings. All models are evaluated in instruct mode under the fixed recent-window protocol. Among the tested pairings, only both-thinking training converges in our runs and reaches $83.9\%$ on StreamingBench.

Figure 2: StreamOPD pipeline with ST-CueGate. (a) During thinking-mode training, the student generates an on-policy response $y$ from a video sampled with the standard training pipeline. The cue-augmented teacher forward provides the distillation target $\tau^{+}$ , while the no-cue forward provides a nested reference $\tau^{-}$ on the same frames and response. Their token-wise conditional log-likelihood ratio is aggregated into a response-level weight $w$ that gates the OPD advantage. (b) During instruct-mode inference, the student uses the recent-window protocol—frames only, with no cue, memory module, or generated thinking trace.

Figure 3: Distribution and concentration of the cue-conditional log-likelihood ratio over 300,866 student-response tokens. (a) Most token scores are near zero, while the remaining distribution is heavy-tailed and includes both positive and negative shifts. (b) The top $20\%$ of positive-score tokens account for $82\%$ of the total positive mass.

Figure 4: Prompt formats for ST-CueGate. The student receives the original video question, while the teacher additionally receives an automatically screened spatio-temporal cue. ST-CueGate also scores the same response under the original no-cue prompt.

Model	#Frames	Streaming Bench	OVO-Bench
			Real-Time Visual Perception							Backward Tracing				Avg.
			OCR	ACR	ATR	STU	FPD	OJR	Avg.	EPM	ASI	HLD	Avg.	Avg.
Human	–	91.46	94.0	92.6	94.8	92.7	91.1	94.0	93.2	92.6	93.0	91.4	92.3	92.77
Offline Video LLMs
Qwen2.5-VL-7B ( Bai et al. 2025 )	1 fps	73.31	67.8	55.1	67.2	42.1	66.3	60.9	59.9	51.5	58.8	23.7	44.7	52.28
LLaVA-OneVision-7B ( Li et al. 2024 )	32	71.12	66.4	57.8	73.3	53.4	71.3	62.0	64.0	54.2	55.4	21.5	43.7	53.85

Table 1: Main results on OVO-Bench ( Niu et al. 2025 ) and StreamingBench ( Lin et al. 2026 ) . We report OVO-Bench per-task accuracy under its Real-Time and Backward tracks; “Avg.” is their mean and “–” denotes unreported results. All StreamOPD models are evaluated in instruct mode with a recent-4-frame window. ST-CueGate (OPD) uses the 9B teacher, while ST-CueGate (OPSD) ( Zhao et al. 2026 ) uses the frozen 4B initial policy;

\dagger

denotes Qwen2.5-VL-7B + HERMES (4K tokens).

Method	Grounded	StreamingBench	OVO-Bench (excl. HLD)	Video-MME	LongVideoBench	Avg.
Qwen3.5-4B (student)	–	77.87	59.94	64.22	57.74	64.94
OPD ( $n{=}1$ )	–	83.91	69.02	63.33	59.84	69.03
OPD ( $n{=}4$ )	–	83.83	70.09	64.07	61.26	69.81
ExOPD ( $n{=}1$ )	✗	84.29	68.38	63.78	60.36	69.20
ViCuR (cue-only, $n{=}1$ )	✓	84.27	67.98	60.56	54.00	66.70
ViCuR (cue-only, $n{=}4$ )	✓	84.49	67.86	64.11	60.73	69.30

Table 2: Ablation of teacher conditioning and cue gating on the 25k data pool. The first row is the untrained student. “ViCuR (cue-only)” retains only ViCuR’s teacher-side visual cue, without its student recovery module ( Tian et al. 2026 ) ; our video implementation of V-Zero ( Sun et al. 2026 ) uses a temporally shuffled video as the negative evidence view. Grounded denotes a teacher-side signal derived from the same underlying training clip; ∗ denotes nested cue-removal gating. The OVO-Bench column excludes HLD; raw HLD and the full macro are reported separately in Table 1 . Avg. is the unweighted mean over the four benchmarks.

Gate Setting	Streaming Bench	OVO-Bench (excl. HLD)	Video- MME	LongVideo Bench	Avg.
$n{=}1$ , batch, seq.	84.23	68.93	65.11	59.69	69.49
$n{=}4$ , uid, seq.	84.55	71.93	64.85	61.41	70.69
$n{=}4$ , uid, token	84.23	68.72	63.52	59.99	69.12
$n{=}8$ , uid, seq.	84.11	68.81	64.41	60.66	69.50

Table 3: Ablation of ST-CueGate rollout grouping and gate granularity on the 25k data pool. “seq.” denotes one response-level gate; “token” denotes token-wise gates. Avg. is the unweighted mean over the four benchmarks.

$\alpha_{\mathrm{g}}$	Streaming Bench	OVO-Bench (excl. HLD)	Video- MME	LongVideo Bench	Avg.
0.25	84.19	69.09	64.19	60.73	69.55
0.50	84.55	71.93	64.85	61.41	70.69
1.00	84.39	68.97	63.22	59.76	69.09

Table 4: Ablation of the ST-CueGate strength

\alpha_{\mathrm{g}}

with fixed

n{=}4

, UID sibling normalization,

\gamma{=}1

, and gate range

[0,2]

. Avg. is the unweighted mean over the four benchmarks.

Teacher	Streaming Bench	OVO-Bench (excl. HLD)	Video- MME	LongVideo Bench	Avg.
Qwen3.5-9B	84.55	71.93	64.85	61.41	70.69
Qwen3.5-27B	83.59	67.52	61.04	54.38	66.63

Table 5: Ablation of teacher model size for ST-CueGate with a fixed Qwen3.5-4B student. Avg. is the unweighted mean over the four benchmarks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Student / teacher	Qwen3.5-4B / Qwen3.5-9B
Training / evaluation mode	thinking / instruct
Training visual input	default 2 fps; no explicit frame cap
Training data	25,118 cueinstruct samples
Epochs	4
Random seeds	42, 43, 44

Appendix

Table 6: Training configuration for the main ST-CueGate experiment.

Axis	Values tried	Used
Gate strength $\alpha_{\mathrm{g}}$	$0.25$ , $0.5$ , $1.0$	$0.5$
Rollouts per prompt $n$	$1$ , $4$ , $8$	$4$
Gate granularity	sequence, token	sequence
Comparison group	minibatch, prompt UID	prompt UID
Teacher	Qwen3.5-9B, Qwen3.5-27B, frozen 4B copy	Qwen3.5-9B
Learning rate	$1\times 10^{-6}$ , $2\times 10^{-7}$	$1\times 10^{-6}$

Appendix

Table 7: Values explored per hyperparameter and the setting used in the reported runs. Gate range

[w_{\min},w_{\max}]{=}[0,2]

\gamma{=}1

, and four epochs were fixed throughout.

Benchmark	Visual Input	Decoding	Reported Metric
StreamingBench	recent 4 frames at 1 fps	instruct, greedy	answer accuracy
OVO-Bench	recent 4 frames at 1 fps	instruct, greedy	backward+realtime macro, with/without HLD
Video-MME	standard lmms-eval, max 32 frames	instruct, greedy	overall accuracy
LongVideoBench	standard lmms-eval, max 32 frames	instruct, greedy	overall accuracy

Appendix

Table 8: Evaluation protocols. The recent-window constraint applies to the streaming benchmarks; Video-MME and LongVideoBench use their standard general-video protocols.

Metric	Recent-4	Dense-32	$\Delta$
StreamingBench	84.19	81.15	$-3.04$
OVO incl. HLD	68.29	63.06	$-5.23$
OVO excl. HLD	70.48	69.03	$-1.45$
Backward Avg. incl. HLD	54.47	54.51	$+0.04$
Backward Avg. excl. HLD	58.85	66.45	$\mathbf{+7.60}$
Realtime Avg.	82.11	71.61	$-10.50$

Appendix

Table 9: Context-policy comparison on the same fixed ST-CueGate model. Dense-32 uniformly samples the visible prefix, while Recent-4 retains only the latest four frames. EPM: Episodic Memory; ASI: Action Sequence Identification; HLD: Hallucination Detection.

Format	Answer	Question	Accepted cue
Multiple choice	C	As the camera focuses on the man in the tank top, what specific exercise is he performing?	Around 0:04, when the man in the tank top is shown, look at the movement of his arms and the cable machine he is using.
Multiple choice	D	As the man finishes sniffing the collar of his striped shirt, what is he about to do next?	Around 0:09, as he lowers his hand from his collar, look at his face and the direction he turns his head.
Binary	Yes	As the man wraps his wrists before lifting, are the straps he is using black and purple?	Around the segment where the man is preparing to lift, look closely at the straps on his wrists.
Binary	Yes	As the person is shown laughing, are their eyes closed?	Around the segment from 6.5 to 7.5 seconds, look at the person’s face.
Counting	0	As the woman in the blue dress is laughing, how many lollipops are visible?	Around the time the woman in the blue dress is laughing, look at the hand in the foreground.
Counting	2	Just before the scene transitions, how many men are giving a thumbs-up gesture?	Around the moment the second text overlay appears, look at the hands of both men standing in the frame.

Appendix

Table 10: Representative cues drawn from the accepted training pool, two per answer format. The answer column is shown here for inspection only; it is never part of the teacher prompt.

$n$	$\alpha_{\mathrm{g}}$	Attainable $w$ range	Clip reachable
4	0.25	$[0.57,\,1.43]$	no
4	0.50	$[0.13,\,1.87]$	no
4	1.00	$[-0.73,\,2.73]$	yes
8	0.50	$[-0.32,\,2.32]$	yes

Appendix

Table 11: Attainable gate values before clipping to

[w_{\min},w_{\max}]{=}[0,2]

, from the bound

|\widetilde{g}|\leq\sqrt{n-1}

. The reported configuration (

n{=}4

\alpha_{\mathrm{g}}{=}0.5

) cannot reach either boundary.

Figure 5: Data construction pipeline. Public video instruction data are cast into verifiable QA, pass-rate filtered by $8$ -sample student/teacher agreement, and annotated offline by a stronger VLM with an automatically screened spatio-temporal visual cue. The cue is wrapped in an explicit instruction frame and appended only to the teacher prompt.

Dataset	Size	Composition
8.3k filtered	8,343	20k after pass-rate filter
20k full	20,031	rlvr $\cup$ cot_rlvr
cot_verifiable	23,884	verifiable CoT pool
25k merged	25,118	8.3k $\cup$ cot_verifiable
Cue injected	24,257	automatically accepted teacher cue
Cue fallback	861	original teacher prompt

Appendix

Table 12: Training-data construction. The 25k merged set is the default training configuration.

Figure 6: Evolution of the cue-conditional log-likelihood ratio across student checkpoints. (a) The cue-versus-no-cue $\Delta_{t}$ distributions remain similar from the base model through steps 700 and 2100. (b) Positive-score concentration is also stable: the top $20\%$ of positive- $\Delta$ tokens carry approximately $80$ – $82\%$ of the total positive mass at every stage.

Figure 7: Pure RL drifts from the deployed response format. (a) GRPO outputs grow from concise answers into long rationales, while OPD remains concise. (b) GRPO becomes sensitive to whether the evaluator reads the first or last answer, whereas the concise OPD model is parser-invariant. This gap diagnoses response-format drift under sparse reward.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 124 papers

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding
Sep 29, 2026Yuwei Miao, Xuesheng Zhang, Wenhao Zou +5Streaming Video Understanding Self-Distillation
Baidu · University of Chinese Academy of Sciences · China University of Petroleum, Beijing
Thinking in Streaming Video
Mar 13, 2026Zikang Liu, Longteng Guo, Handong Li +7Streaming Video Understanding Streaming Inference
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
StreamScout: Learning When to Look Deeper for Streaming Video Understanding
Aug 31, 2026Ce Zhang, Jing Bi, Jinxi He +9Adaptive Inference Streaming Video Understanding
Carnegie Mellon University · TikTok · University of Rochester +2

Sep 29, 2026J/K move · Enter open · S save

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

Yuwei Miao, Xuesheng Zhang, Wenhao Zou, Jixia Zhang+4

Baidu · University of Chinese Academy of Sciences · China University of Petroleum, Beijing

Streaming Video Understanding Self-Distillation On-Policy Distillation Temporal Video Understanding

Open paper AskPDF arXiv

Real-time video understanding requires incrementally maintaining a memory of streaming content, and optimizing this requires dense process signals. On-Policy Self-Distillation (OPSD), which lets one model serve as both teacher and student with the teacher receiving additional privileged information such as the question and ground-truth (GT) answer, can supply such token-level signals. However, applying it directly to streaming video raises two problems. (1) The student cannot be optimized end-to-end, where memory is written before the question arrives, yet the teacher scores it with the question-and-GT privilege, misaligning their preferences. (2) Effective-entity memory collapses, where the question-and-GT privilege makes the teacher favor only question-relevant entities, and token-mean averaging over a memory renders its signal invariant to how many entities that memory covers, both driving memory against the streaming need for diversity. To address these issues, we propose Event-Grounded Self-Distillation (EGSD), which characterizes streaming memory as an incremental update over verifiable Events (key visual entities, actions, and details) and targets the two problems on this basis. For problem (1), we adapt the OPSD signal into a multiplicative weight combined with the outcome reward; for problem (2), we re-weight the teacher with Events as privileged information to counter its question-relevance bias, and add an entity-coverage reward to supply the coverage preference the token-mean teacher lacks. Extensive experiments on mainstream online and offline benchmarks show EGSD achieves strong performance, reaching 79.8% on StreamingBench and 73.4% on the OVO-Bench Real-Time track, while memory analysis shows effective-entity recall rises 17.4% at only 6.8% more memory length.

Abstract

Streaming Video Understanding

Explore similar work

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

Thinking in Streaming Video

StreamScout: Learning When to Look Deeper for Streaming Video Understanding