Memory-Augmented Video Understanding

Overview

Memory-augmented video understanding studies how models can retain useful evidence from long videos or live streams when their context and compute are limited. Approaches include bounded frame caches and semantic compression, hierarchical event- or entity-based memories, and retrieval systems that can revisit relevant clips; some methods also update recurrent states or model weights over time. These techniques support tasks such as streaming question answering, long-video and egocentric reasoning, and tracking objects or events across time, which are tested in both real-time and offline benchmarks. A central challenge is balancing compact, efficient memory with reliable retrieval of fine details, especially when later questions depend on evidence that seemed unimportant when first observed.

Latest papers

CardsList

96 papers15 papers in the last four weeks, down 32% on the four weeks before. 0.1% of all new papers.

  1. Beyond Answering: Learning to Speak First and Act in Streaming Video

    Nov 12, 2026Shuhan Li, Huankang Guan, Xintong Guo +15Streaming Video UnderstandingProactive Video Understanding

  2. Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

    Nov 11, 2026Yehua Huang, Penglei Sun, Xiang Li +8Streaming Video UnderstandingTemporal Reasoning in VLMs

  3. Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery

    Nov 7, 2026Zhiran Zhang, Hu Wang, Yichen Wu +1Surgical Video UnderstandingMemory-Augmented Video Understanding

  4. TimeTraveler: Temporal Strategy Planning with Time Dictionary for Streaming Video Understanding

    Nov 7, 2026Minkuk Kim, Junho Kim, Seong Tae KimStreaming VQAStreaming Video Understanding

  5. Sekai: Incremental Event Memory for Generalized Reasoning Video Segmentation

    Nov 4, 2026Zhengtao Liu, Jinrong Xu, Shiu-Hong Kao +2Temporal Video UnderstandingReferring Video Object Segmentation

  6. SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

    Nov 1, 2026Ruipeng Ma, Qing Xu, Jie Liang +1Streaming Video UnderstandingSurgical Video Understanding

  7. StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering

    Oct 31, 2026Zhuchen Shao, Bo Chen, Klara NahrstedtStreaming VQAVideo Question Answering

  8. TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs

    Oct 30, 2026Haotian Yang, Yong Chen, Cheng Peng +4Evidence-Grounded VQAStreaming Video Understanding

  9. DUGO: Dual-Scale Gaussian Memory for Online Temporal Action Localization in Egocentric Procedural Videos

    Oct 30, 2026Mohammed Aman Bhuiyan, Md. Sazzad Hossain Adib, Samiul Basir Bhuiyan +4Temporal Action LocalizationTemporal Video Understanding

  10. StreamVibe: Proactive Video Agents for Evolving User Intent

    Oct 26, 2026Chengzhi Liu, Yuxiang Lai, Yuzhe Yang +13Vision-Language AgentsMultimodal Intent Recognition

  11. 4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

    Oct 23, 2026Yanhao Jia, Yi Zhu, Yiran Zhong +7Long-Horizon Agent EvaluationTemporal Reasoning in VLMs

  12. Organizing Temporal Evidence in Hyperbolic Space for Video Moment Retrieval

    Oct 23, 2026Mengnan Liu, Sanping Zhou, Kun Xia +3Video Moment RetrievalTemporal Video Grounding

  13. Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering

    Oct 22, 2026Raffaele Cali, Giuseppe Lando, Rosario Forte +1Egocentric VQAVideo Question Answering

  14. SENRI: Understanding the Space-Event Nexus in Real-Life Videos

    Oct 18, 2026Kehan Lan, Kaining Ying, Henghui Ding +1Egocentric Video UnderstandingSpatiotemporal Reasoning

  15. Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

    Oct 17, 2026Kerui Chen, Jinglu Wang, Yun Wang +2Memory-Augmented Video UnderstandingMemory-Augmented VLMs

  16. Memory Internalization via Test-Time Training for Long-Video Understanding

    Oct 17, 2026Haonan Zhao, Jiaxuan Chen, Qingyang Liu +4Memory-Augmented Video UnderstandingLong-Video QA

  17. PACE-Mem: Prefix-Aware Complementary Evidence for Multimodal Agents

    Oct 17, 2026Shidu Ren, Rong-cheng Tu, Hang Zhou +1Memory-Augmented AgentsMemory-Augmented Video Understanding

  18. Memory as Weights: Internalizing Long-Term History for Streaming Videos

    Oct 15, 2026Zhuoming Liu, Jinhong Lin, Lin Zhang +2Memory-Augmented VLMsMemory-Augmented Video Understanding

  19. DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

    Oct 14, 2026Boyuan Sun, Deshui Miao, Wenzhao Gao +8Streaming Video UnderstandingVideo Question Answering

  20. PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary

    Oct 12, 2026Xinlong Chen, Weihong Lin, Yuanxing Zhang +9Video CaptioningMemory-Augmented Video Understanding

  21. SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding

    Oct 10, 2026Senbin Xu, Zijun Zhang, Xinhua Wu +7Streaming VQAStreaming Video Understanding

  22. GLIMPSE: Global-Local Indexing with Multi-Precision Spectral Encoding for Explorable Video World Models

    Oct 9, 2026Zhufeng Xu, Henglei Lv, Bailin Deng +5Video World ModelsMemory-Augmented Video Understanding

  23. VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding

    Oct 8, 2026Runquan Gui, Hanzhu Chen, Zehao Wang +3Long-Video QAVideo Question Answering

  24. Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

    Oct 7, 2026Shravan Chaudhari, William Paul, Suchi Saria +2Memory-Augmented Video UnderstandingEgocentric Video Understanding

  25. VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

    Oct 7, 2026Yongchao Xu, Bowen Ye, Jiefeng Gan +5Long-Video UnderstandingMemory-Augmented Video Understanding

  26. VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

    Oct 5, 2026Yucheng Liu, Yufei Yin, Mingxiao Feng +3Long-Video UnderstandingMemory-Augmented Video Understanding

  27. ReMem: Streaming Video Understanding With Long Context Retention

    Oct 5, 2026Li Yiheng, He Xu, Wang Shaobo +2Memory-Augmented VLMsStreaming Video Understanding