Organizations: National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, P. R. China · Institute of Artificial Intelligence (TeleAI), China Telecom, 31 Jinrong Ave., Xicheng District, Beijing 100033, P. R. China · University of Science and Technology Beijing, Beijing 100083, P. R. China
Multimodal Large Language Models (MLLMs) have shown strong capabilities in image understanding, motivating recent efforts to extend them to video reasoning. However, existing Video LLMs struggle in online streaming scenarios, where long temporal context must be preserved under strict memory constraints. We propose WAT (Watching Before Thinking), a two-stage framework for online video reasoning. WAT separates processing into a query-independent watching stage and a query-triggered thinking stage. The watching stage builds a hierarchical memory system with a Short-Term Memory (STM) that buffers recent frames and a fixed-capacity Long-Term Memory (LTM) that maintains a diverse summary of historical content using a redundancy-aware eviction policy. In the thinking stage, a context-aware retrieval mechanism combines the query with the current STM context to retrieve relevant historical frames from the LTM for cross-temporal reasoning. To support training for online video tasks, we introduce WAT-85K, a dataset containing streaming-style annotations emphasizing real-time perception, backward tracing, and forecasting. Experiments show that WAT achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench, outperforming existing open-source online Video LLMs while operating at real-time frame rates.
Figures & tables
Fig. 1: Illustrations of different Video LLMs: (a) Vanilla offline processing, which encodes the uniformly sampled frames at once; (b) Vanilla online processing, which encodes frames sequentially into the space-limited memory; while (c) Our WAT introduces a watching-before-thinking pipeline to enhance online comprehension through hierarchical memory and explicit query-based retrieval reasoning.
Fig. 2: Overview of the proposed WAT framework, which follows a two-stage pipeline for online streaming video reasoning. Given a continuous video stream V and a textual query Q , WAT decouples perception and reasoning into a Watching stage and a Thinking stage. In the Watching stage, WAT maintains a hierarchical memory system consisting of a Short-Term Memory (STM, MS ) and a Long-Term Memory (LTM, ML ). The STM serves as a high-fidelity sliding window, implemented as a First-In-First-Out (FIFO) queue to preserve recent frames and capture fine-grained temporal dynamics. In parallel, the LTM stores a compact yet semantically diverse set of historical features, managed by a redundancy-aware eviction policy that removes over-represented entries while protecting recent content. In the Thinking stage, triggered by a query Q , WAT performs context-aware retrieval by first fusing the query with the current STM content to form a conditioned query representation, which is then used to retrieve the top- K most relevant historical features F∗ from the LTM. Finally, the retrieved features F∗ , together with the short-term memory MS and the query Q , are concatenated into a unified multimodal input sequence and fed into a Multimodal Large Language Model (MLLM) to perform coherent cross-temporal reasoning and generate the final response. The retrieval process is trained with an auxiliary Retrieval Alignment Contrastive Learning (RACL) objective. This contrastive loss aligns the relevant features F∗ with the query Q (positive) against random visual evidence FR∗ (negative).
Fig. 3: Case study of WAT on a sample video from OVO-Bench demonstrates its online reasoning capabilities. At 59s, when a user poses the query “What does the man do before?”, WAT proactively infers the preceding actions and correctly outputs the past activity, showing its ability to leverage hierarchical memory for temporal context. Furthermore, at 184s, when presented with a follow-up question about the future, “What will the man do next?”, the system similarly produces an accurate prediction, highlighting WAT’s capacity for both backward tracing and forward forecasting in continuous video streams.
Fig. 4: Visualization comparisons of retrieved visual content and corresponding responses between our method and other state-of-the-art (SOTA) approaches. In the top example, WAT exhibits strong capabilities in long-term temporal reasoning, effectively capturing dependencies across distant frames for online video understanding. This highlights WAT’s advantage in modeling extended temporal contexts, which is crucial for accurate understanding of complex, continuous video streams. The bottom example further demonstrates WAT’s robust performance on conventional (short-form) VideoQA tasks, showing precise alignment between relevant visual segments and generated responses.
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/
Lu Wang, Zhuoran Jin, Yupu Hao +4
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China
Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.
Yiran Guan, Liang Yin, Dingkang Liang +5
Huazhong University of Science and Technology, Wuhan, China · MiLM Plus, Xiaomi Inc., Beijing, China
Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.
Ziheng Huang, Yicheng Bao, Xueheng Li +8
East China Normal University · University of Science and Technology of China · Xiamen University