Organizations: Fudan University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory · Stanford University · Peking University · The Hong Kong University of Science and Technology · The University of Tokyo · The University of Hong Kong · The Chinese University of Hong Kong
Streaming video understanding requires models to repeatedly decide whether to respond or continue observing. Recently, Group Relative Policy Optimization (GRPO) has been applied to this setting, with existing approaches emphasizing reward design while retaining a single trajectory-level advantage across turns. Such shared credit can reinforce redundant replies after earlier successes and discourage useful follow-ups after earlier failures. We propose TurnWise, a turn-level GRPO framework that aligns learning with individual streaming decisions. TurnWise addresses the mismatch between trajectory-level feedback and turn-level interaction, promoting informative responses and reducing redundancy. Specifically, turn-level credit assignment uses silence-completed prefix scoring to construct group-relative advantages from each decision's remaining return, separating its learning signal from earlier rewards. Turn-mean policy optimization averages token losses within each turn and then across turns, giving response and waiting decisions equal weight regardless of output length. These complementary designs improve streaming policy learning without a learned critic or additional rollouts. Extensive experiments show our method outperforms prior RL approaches for streaming video understanding.
Figures & tables
Figure 1: Shared advantages can reinforce redundant replies. GRPO assigns one trajectory-level advantage to all turns, so later repetitions may inherit positive credit from earlier successes. TurnWise subtracts group-relative accrued reward before each decision, allowing later turns to receive credit according to their remaining return. The dialogue is illustrative. The advantage plot uses recorded TurnWise advantages and the corresponding GRPO counterfactual.
Figure 2: Overview of TurnWise. Given streaming video input x , user queries, dialogue history, and an observation plan, the policy samples a group of trajectories {τi}i=1G . TurnWise converts trajectory rewards Ri into group-relative remaining-return advantages A^i,t through silence-completed prefix scoring. Turn-mean aggregation then gives every response-or-wait decision equal weight in the trajectory loss Li .
Method
ProactiveVQA
OVO-Timing
StreamingBench
PAUC ↑
Timing F1 ↑
RealTime Acc. ↑
Proactive Acc. ↑
Avg. ↑
Ego
TV
VAD
Web
Overall
CRR
SSR
REC
Avg.
Qwen2.5-VL-3B
Baseline
9.32
27.85
8.98
42.45
22.15
0.00
0.53
3.46
1.33
69.42
22.40
45.91
+ SFT
26.05
27.60
26.30
35.00
28.74
0.00
0.95
0.76
0.57
74.68
22.80
48.74
+ MMDuet2
44.10
39.70
28.70
45.90
39.60
13.86
17.56
21.76
17.72
74.88
20.40
47.64
Table 1: TurnWise achieves the best aggregate scores on ProactiveVQA, OVO-Timing, and StreamingBench with both backbones. Within each backbone, the best and second-best results are bold and underlined , respectively. Δ denotes TurnWise minus the strongest non-TurnWise result in each aggregate column.
Figure 3: TurnWise achieves higher late-stage reward with fewer redundant replies relative to credit-only optimization. Panel a shows reward, panel b shows reply-normalized repetition penalty magnitude, and panel c shows non-silent decision fraction. Faint traces are batch means. Solid curves are trailing 20-update averages. Gray shading marks credit warmup.
Table 5
Figure 4: Turn-level credit assigns negative advantages to 59.83% of repetitive replies with positive global advantages (432 of 722). Both allocations use the same sampled groups from Qwen2.5-VL-3B training, using complete trajectories with full credit correction at α=1 . Panel a shows advantage density over 84,184 decisions. Turn advantages are clipped to ±5 as in training. The dashed diagonal marks unchanged allocation. Panel b shows advantage signs for all 2,972 Judge-labeled repetitive replies in this cohort.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
Streaming video requires handling dynamic information density under strict latency budgets. Yet, existing methods typically employ static strategies, such as fixed memory compression or reliance on a single model, forcing a trade-off: fast models fail on complex queries, while always-on heavy models violate real-time constraints and overcomplicate simple queries. Rather than fixing these decisions upfront, we propose R3-Streaming (Remember, Respond, Reason), which formulates streaming video understanding as a cascaded control problem: for each query, the system compresses memory, judges response readiness, and routes computation sequentially, so that each downstream decision builds on progressively refined information states. To optimize this pipeline, we introduce an age-aware forgetting policy for memory compression, as aggressively compressing historical frames can yield substantial performance gains. For compute routing, we propose TB-GRPO, a target-balanced reinforcement learning objective that routes hard queries to a stronger model while preventing mode collapse. Extensive evaluations demonstrate that R3-Streaming achieves state-of-the-art results among streaming MLLMs, reaching 57.92 on OVO-Bench and 76.36 on StreamingBench, while reducing visual token usage by 95 to 96 percent.
Jinming Liu, Jianguo Huang, Zhaoyang Jia +7
1Shanghai Jiao Tong University · 2Eastern Institute of Technology, Ningbo, China · 3Microsoft Research Asia
Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.
Ziheng Huang, Yicheng Bao, Xueheng Li +8
East China Normal University · University of Science and Technology of China · Xiamen University
Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.
Ce Zhang, Jing Bi, Jinxi He +9
Carnegie Mellon University · TikTok · University of Rochester +2