Streaming Video Understanding

Overview

Streaming video understanding asks how models can interpret frames as they arrive, answer questions about both recent and past events, and sometimes act or speak proactively. Common approaches use causal processing with bounded visual or semantic memory, compressing redundant frames and retrieving relevant evidence when a query arrives; others train models to reason incrementally or decide when they have enough evidence to respond. Benchmarks increasingly test long-range recall, fast events, multi-turn interaction, response timing, and whether a model stays silent when appropriate. A central tension is balancing accurate, grounded memory and timely responses against limits on computation, latency, and storage, while offline video performance often fails to predict real-time reliability.

Latest papers

CardsList

147 papers20 papers in the last four weeks, down 20% on the four weeks before. 0.2% of all new papers.

  1. ALIVE: Making Streaming Video Models Interactive

    Nov 16, 2026Jinzhao Li, Yuan Xu, Jinxuan Wu +13Streaming Video UnderstandingProactive Video Understanding

  2. Beyond Answering: Learning to Speak First and Act in Streaming Video

    Nov 12, 2026Shuhan Li, Huankang Guan, Xintong Guo +15Streaming Video UnderstandingProactive Video Understanding

  3. FuGIBench: Evaluating Reliable Preparation in Streaming Video

    Nov 12, 2026Changheng Cai, Zhichao Huang, Song Ren +8Proactive AssistanceProactive Video Understanding

  4. Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

    Nov 11, 2026Yehua Huang, Penglei Sun, Xiang Li +8Streaming Video UnderstandingTemporal Reasoning in VLMs

  5. TimeTraveler: Temporal Strategy Planning with Time Dictionary for Streaming Video Understanding

    Nov 7, 2026Minkuk Kim, Junho Kim, Seong Tae KimStreaming VQAStreaming Video Understanding

  6. Turns Matter: Advancing Streaming Video RL via Turn-Level Credit

    Nov 4, 2026Shuai Dong, Kanghui Tian, Linghan Chen +20Streaming Video UnderstandingRL for Video Reasoning

  7. SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

    Nov 1, 2026Ruipeng Ma, Qing Xu, Jie Liang +1Streaming Video UnderstandingSurgical Video Understanding

  8. StreamDuo: A Live Commentator and a Research Agent for Unseen Games

    Oct 31, 2026Zixian Gao, Penghao Luo, Xiangcheng Cao +5Streaming Video UnderstandingLLM Agent Training

  9. TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs

    Oct 30, 2026Haotian Yang, Yong Chen, Cheng Peng +4Evidence-Grounded VQAStreaming Video Understanding

  10. Hierarchical Step Decomposition for Training-free Online Video Step Grounding

    Oct 29, 2026Long Zhou, Luca Zanella, Paolo Rota +2Temporal Video GroundingStreaming Video Understanding

  11. AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

    Oct 28, 2026Lu Wang, Zhuoran Jin, Longxiang Wang +8Streaming Video UnderstandingVideo Question Answering

  12. StreamVibe: Proactive Video Agents for Evolving User Intent

    Oct 26, 2026Chengzhi Liu, Yuxiang Lai, Yuzhe Yang +13Vision-Language AgentsMultimodal Intent Recognition

  13. SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding

    Oct 23, 2026Xueyun Tian, Bingbing Xu, Jiaming Liu +6Proactive Video UnderstandingStreaming Video Understanding

  14. DirCast: Live Esports Directing and Commentary with a Single Streaming Video-Language Model

    Oct 22, 2026Zixian Gao, Lixin Xiu, Penghao Luo +5Video-Language ModelsStreaming Video Understanding

  15. ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

    Oct 21, 2026Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun +6Agent MemoryMemory Agent Benchmarks

  16. Cache Locally, Recompute Selectively for Streaming Video Understanding

    Oct 21, 2026Junhyeok Lee, Sangyoon Lee, Jaeho LeeStreaming Video UnderstandingStreaming Inference

  17. PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding

    Oct 20, 2026Siqi Guo, Yuxia Chen, Wenbo Zhu +4Streaming VQAStreaming Video Understanding

  18. StreamAhead: Real-Time Streaming Video Question Answering via Adaptive Precomputation

    Oct 18, 2026Haoyang Fan, Dhruv Parikh, Rajgopal Kannan +1Streaming VQAVideo Question Answering

  19. Memory as Weights: Internalizing Long-Term History for Streaming Videos

    Oct 15, 2026Zhuoming Liu, Jinhong Lin, Lin Zhang +2Memory-Augmented VLMsMemory-Augmented Video Understanding

  20. From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

    Oct 15, 2026Boyu Cui, Junhao Luo, Zhihao Wang +5Proactive Video UnderstandingVideo-Language Model Evaluation

  21. DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

    Oct 14, 2026Boyuan Sun, Deshui Miao, Wenzhao Gao +8Streaming Video UnderstandingVideo Question Answering

  22. FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

    Oct 13, 2026Haonan Hu, Shuhao Chen, Weisen Jiang +3Query-Aware Video Frame SelectionLong-Video Understanding

  23. SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding

    Oct 10, 2026Senbin Xu, Zijun Zhang, Xinhua Wu +7Streaming VQAStreaming Video Understanding