SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selection
Authors: Dohwan Ko, Ji Soo Lee, Pierce Chuang, Debojeet Chatterjee, Ashish Shenoy, Yichao Lu, Seungwhan Moon, Xin Luna Dong, Vikas Bhardwaj, Hyunwoo J. Kim
Organizations: Korea University, Seoul, Republic of Korea · Meta AI · KAIST, Daejeon, Republic of Korea
Most keyframe selection studies focus on offline settings, assuming access to the full video and query in advance. In contrast, real-world streaming scenarios require online frame selection under unknown video duration, without access to either the query or future frames during selection. To address this, we introduce Streaming Video Memory (SVMem), a compact and representative memory of previously observed content, updated continuously as the video stream unfolds. Building on this setting, we propose the Streaming Video Memory Agent (SVMemAgent), which dynamically maintains a memory by deciding at each timestep whether to replace an existing memory frame with the incoming frame or discard it. SVMemAgent is trained using Group Relative Policy Optimization (GRPO) with task-driven rewards derived from diverse question-answer pairs, implicitly exposing the policy to a distribution of queries during training so that SVMem retains generally informative frames at inference, when queries are unavailable. Experiments on both online and offline video benchmarks show that SVMemAgent consistently outperforms online frame selection baselines and achieves competitive performance with offline methods that assume access to the full video and query. Through task-driven rewards, SVMemAgent learns an emergent keyframe selection policy that prefers frames containing textual information, which may benefit downstream VideoQA tasks.
Figures & tables
Figure 1: (a) Offline frame selection methods simply retrieve query-relevant frames with access to the full video, whereas (b) SVMemAgent sequentially updates SVMem without access to the query or future frames, deciding whether to discard the current frame or use it to replace an existing memory slot.
Figure 2: Illustration of SVMemAgent and GRPO training with task-driven rewards. SVMemAgent integrates (a) a semantic branch designed for content-aware memory updates and (b) a temporal branch for maintaining uniform temporal coverage. (c) During training, SVMemAgent is optimized using GRPO with task-driven rewards to learn a general prior over informative frames without requiring query information at inference time.
Table 3
Figure 3: (a) Performance across different memory budgets, measured by the number of stored frames; (b) performance across different query arrival times; and (c) performance across varying levels of query novelty. Query novelty is computed as the mean cosine distance between a test-query embedding and its 10 nearest training-query embeddings, obtained using Sentence-BERT [ 19 ] . Higher values indicate that the query is more out-of-distribution relative to the training queries.
Video frame selection chooses a compact set of frames that preserves important events, context, or task-specific evidence while reducing processing costs.
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Generative AI, The Hong Kong Polytechnic University, Hong Kong, China · Tencent Inc., China +1