cs.CVMar 12, 2026
SaveThink While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China
Abstract
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/
Figures & tables
Figure 1: Overview of Think While Watching. (a) Interleaved baseline. Video perception and answer generation are executed sequentially, which can cause memory erosion , where early memory is forgotten, and a serialization bottleneck , where generation stalls further input ingestion. (b) Think While Watching (ours). The video frames are processed in segments (SEG 1 to SEG 4) to build a continuous segment-level memory . During streaming, questions are answered online by retrieving implicitly relevant memories while continuing to watch. (c) Latency comparison. A schematic timeline showing that interleaved processing accumulates queueing delay, while our decoupled design parallelizes segment processing and answering to reduce latency.
Figure 2: Training components of Think While Watching. (a) segment-level streaming attention mask and streaming positional encoding: example input stream with generated outputs . Green indicates the source prefix available at time step , which is allowed to attend to. Red masks all future segments to prevent information leakage. For positional encoding, we use separate position indices for the input and output streams. (b) Three-stage training strategy: single-round CoT for streaming input adaptation, multi-round CoT for multi-turn interaction, and long-range capability training for long-term memory, uncertainty handling, and distractor learning.
| Stage | Source | Instances | Rounds | Avg. rounds | Frames | Min (s) | Avg (s) | Max (s) |
|---|---|---|---|---|---|---|---|---|
| 1 | VideoChatOnline-IT | 5,160 | 5,160 | 1.00 | 8.18 | 79.40 | 3550.10 | |
| 2 | VideoChatOnline-IT | 2,752 | 8,513 | 3.09 | 2.00 | 400.92 | 3443.97 | |
| 3 | YouTube | 1,500 | 6,000 | 4.00 | 100-300+ | 600.12 | 1697.30 | 3595.03 |
Table 1: Dataset statistics across three training stages. Stages 1&2 are built from VideoChatOnline-IT short videos, while Stage 3 contains long-range multi-round dialogues from YouTube. Video duration is reported in seconds (min/avg/max).
| Performance | Efficiency | ||||||||
| Regime | Method | SQA | OmniSource | Realtime | Proactive | Overall | Avg Tokens | Token Reduce | |
| Closed-source models | |||||||||
| Offline | Gemini 1.5 Pro [ 41 ] | 54.80 | 67.80 | 77.39 | 45.10 | 70.26 | - | - | - |
| GPT-4o [ 42 ] | 32.80 | 50.95 | 74.54 | 56.86 | 64.10 | - | - | - | |
| Open-source models | |||||||||
| Online | Flash-VStream-7B [ 32 ] | 26.80 | 26.00 | 23.23 | 1.96 | 24.04 | - | - | - |
Table 2: StreamingBench results with accuracy. Columns left of the double bar report performance , higher is better, while columns on the right report efficiency for open-source models only. is computed against the same backbone Thinking baseline. Avg Frames is 148.35 for the single-turn protocol and 62.58 for the multi-turn protocol.
| Performance | Efficiency | |||||||
| Regime | Method | Backward | Realtime | Forward | Overall | Avg Tokens | Token Reduce | |
| Closed-source models | ||||||||
| Offline | Gemini 1.5 Pro [ 41 ] | 69.32 | 62.54 | 57.15 | 63.00 | - | - | - |
| GPT-4o [ 42 ] | 64.46 | 60.75 | 53.40 | 59.54 | - | - | - | |
| Open-source models | ||||||||
| Online | Flash-VStream-7B [ 32 ] | 27.38 | 28.37 | 45.09 | 33.61 | - | - | - |
Table 3: OVO-Bench results . Avg Frames is 63.23 for the single-turn protocol and 25.47 for the multi-turn protocol.
| Video-MME | LV-Bench | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Short | Medium | Long | Overall | ER | EU | KIR | TG | Rea | Sum | Overall |
| Thinking | 77.78 | 66.67 | 62.22 | 68.89 | 54.22 | 51.62 | 62.07 | 46.82 | 55.50 | 40.35 | 53.47 |
| Instruct | 78.22 | 66.78 | 62.89 | 69.30 | 56.15 | 53.78 | 65.86 | 50.45 | 60.00 | 43.86 | 56.19 |
| TWW | 78.89 | 67.11 | 64.00 | 70.00 | 58.07 | 54.24 | 66.21 | 51.36 | 62.00 | 47.37 | 57.39 |
| TWW | 79.00 | 67.11 | 62.33 | 69.48 | 56.44 | 54.39 | 66.55 | 50.91 | 61.50 | 45.61 | 56.81 |
| TWW | 83.22 | 70.11 | 66.89 | 73.41 | 57.33 | 54.70 | 67.59 | 51.36 | 62.50 | 52.63 | 57.68 |
Table 4: Offline video understanding results on Video-MME and LV-Bench.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
|---|---|
| GPUs | 8 NVIDIA RTX A6000 (48GB) |
| Model | Qwen3-VL-2/4/8B-Instruct |
| Training method | Full-parameter SFT |
| Precision | bf16 |
| Global batch size | 128 |
| Optimizer | AdamW |
Table A1: Implementation details and training hyperparameters.
Figure A1: Decoder-induced ingestion backlog under interleaved streaming. As utilization increases, interleaved decoding pauses can amplify the catch-up delay and enter a backlog explosion regime, while our decoupled design substantially reduces decoder-induced backlog growth.
Figure A2: Dataset composition
| Subset | Samples | Proportion (%) |
|---|---|---|
| HiREST | 352 | 6.8 |
| DiDeMo | 4023 | 78.0 |
| QuerYD | 785 | 15.2 |
| Total | 5,160 | 100.0 |
Table A2: Stage 1 detailed statistics.
| Subset | Samples | Segments | Questions | Avg. Segs. | Avg. Qs |
|---|---|---|---|---|---|
| AVA | 610 | 6,136 | 1,902 | 10.06 | 3.12 |
| COIN | 501 | 2,793 | 1,109 | 5.57 | 2.21 |
| GOT-10K | 999 | 2,830 | 2,820 | 2.83 | 2.82 |
| HiREST-Step | 102 | 320 | 422 | 3.14 | 4.14 |
| ViTT | 346 | 1,090 | 1,436 | 3.15 | 4.15 |
| YouCook2 | 194 | 630 | 824 | 3.25 | 4.25 |
Table A3: Stage 2 detailed statistics. Segments counts the number of segment units after stream segmentation, and Questions counts the number of question turns in the synthesized dialogues. Avg. Segs. and Avg. Qs. denote the average numbers of segments and questions per sample.
| Category | Samples | Segments | Questions | Avg. Segs. | Avg. Qs |
|---|---|---|---|---|---|
| Tutorial | 487 | 5,631 | 2,017 | 11.56 | 4.14 |
| Lecture | 521 | 4,532 | 1,893 | 8.70 | 3.63 |
| Longform | 492 | 5,536 | 2,090 | 11.25 | 4.25 |
| Total | 1,500 | 15,699 | 6,000 | 10.47 | 4.00 |
Table A4: Stage 3 statistics by category.
| ID | Constraint |
|---|---|
| A | Strict one-to-one alignment: exactly one output chunk per input unit, preserving order. |
| B | No empty segment: each segment must update/maintain grounded state based on visual evidence. |
| C | No future information: segment reasoning uses only current and past units; QA uses evidence up to its timestamp. |
| D | No answer leakage: do not reveal reference answers in reasoning; copy the answer only in the final Answer field. |
| E | Video-grounded only: rely on provided frames/timestamps/metadata; avoid external assumptions. |
| F | Streaming quality: emphasize boundary cues and conservative state updates across segments. |
Table A5: Core constraints for pseudo streaming CoT generation.
| Query keyword | Video type |
|---|---|
| Tutorial | |
| sourdough bread tutorial | Complete bread-making workflow |
| furniture restoration | Furniture restoration project |
| oil painting tutorial | Oil painting step-by-step tutorial |
| car repair tutorial complete | Full car repair walkthrough |
| sewing tutorial complete | Complete sewing tutorial |
Table A6: Representative YouTube search queries used for Stage 3 video retrieval.
Figure A3: Dataset-derived pseudo streaming CoT example under the single-round protocol. A question is asked once after the observed video prefix. The model first writes a segment-level memory note from the incoming frames, identifying the tutorial title card, the cleaning material, the faucet, and the rinsing action around the keg. When queried at 49.1s, it answers using only the accumulated evidence so far and correctly concludes that the event Clean a Beer Keg is still ongoing.
Figure A4: Dataset-derived pseudo streaming CoT example under the multi-round protocol. The stream follows a crochet tutorial. Memory notes from Segments 1 to 8 trace the progression from the introduction to close-up hand actions and the gradual formation of the crochet pattern. After the first event-listing question, the model reports only the completed events observed so far. As additional segments arrive, it updates the memory with later stitching and finishing steps. When the same question is asked again at 417.6s, the answer includes only the newly completed events that were not reported earlier, which illustrates incremental reasoning across multiple turns over a continuous stream.
Figure A5: Real multi-round streaming example. Left: Segment-level memory notes summarize the contestant’s appearance, location, and actions across seven streamed segments. Middle: Representative frames show the progression of a card routine from the main stage to the judges’ desk and finally to a red-lit table. Right: Five temporally ordered questions are interleaved with the stream. Questions Q2 to Q5 depend on earlier turns, requiring the model to resolve the previously mentioned person, identify the current interaction partner, recall the recent fist bump, update the contestant’s outfit after the scene transition, and track the object held in hand at the end. This example illustrates how persistent segment-level memory supports consistent reasoning across multiple turns over a changing stream.
Figure A6: Residual failure cases in multi-turn streaming reasoning. (a) Long-range recall failure. The model fails to recover an early fine-grained identity cue across turns. As illustrated in the top row, the green box marks the correct person referred to in the first question, whereas the red box marks the person incorrectly retrieved by the model in the later question, indicating that the coarse event context is retained but the specific identity association is lost. (b) Premature commitment under incomplete evidence. The middle row shows that, at the queried moment, the decisive visual evidence for whether the red-jersey player is taking a corner kick has not yet become sufficiently clear. Instead of deferring its response until the event is visually confirmed, the model commits to a specific answer too early. (c) Distractor-induced memory contamination. The bottom row shows repeated alternation between the speaker and inserted movie clips. Because these inserted clips appear interleaved with the main explanation, the model is distracted by the intervening visual switches and produces an incorrect insertion count. These failures are residual edge cases of the three long-horizon behaviors explicitly targeted in Stage 3 training, namely long-term evidence retention, uncertainty handling, and distractor robustness.