cs.CVOct 17, 2026

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

Authors: Kerui Chen, Jinglu Wang, Yun Wang, Xinjiang Wang, Yan Lu

Organizations: College of Artificial Intelligence, Zhejiang University. · Microsoft Research Asia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Long videos often exceed the context capacity of multimodal large language models (MLLMs), while agentic search repeats costly video processing and grounding for each query. Reusable memory can amortize this cost, but must preserve information for questions unknown during construction. We propose Video-EMem, an evolvable video memory harness built on time-anchored perceptual observations. This representation combines compact free-form text with sparse images to preserve semantic content and complementary visual details across temporal scales, supporting temporal navigation without a fixed semantic schema. A Writer encodes each video once by consolidating overlapping multiscale observations, and a Reader retrieves and organizes evidence solely from the resulting memory. With the underlying models frozen, we evolve their workflows and operators using distinct feedback: query-independent source fidelity and coverage for the Writer, and QA failures in evidence localization and utilization for the Reader. Experiments support this separation for generalization beyond development queries, with our evolution strategy improving average accuracy by 7.2% over the unevolved harness. While reducing storage by about 99.86% on average relative to raw videos, Video-EMem improves Qwen3-VL-30B by an average of 2.6% across three long-video QA benchmarks over direct input of 2,048 frames. Using the same o3 answer model as the state-of-the-art agentic method, it improves average accuracy by 2.5% with lower per-query latency.

Figures & tables

Figure 1: The need for compact, reusable video memory. (a) Answering multiple questions about the same video incurs repeated processing costs: VLMs repeatedly process long visual contexts and also often omit details due to context limits, while agentic methods revisit the raw video through multiple retrieval and reasoning rounds. (b) To reduce query and storage costs, we adopt a write-once, query-many paradigm, encoding the video once into light-weight memory that preserves information for unknown future questions and supports efficient access across queries.
Figure 2: Method overview. (a) We present Video-EMem, an evolvable memory harness with a Writer that constructs compact, query-independent video memory and a Reader that retrieves and organizes evidence for question answering. Our memory consists of time-anchored observations combining multiscale textual descriptions with sparse complementary images. We first evolve the Writer (b) using source-preservation feedback without queries, then evolve the Reader (c) from QA failures attributed to evidence localization and utilization.
WriterReaderLVBenchVideo-MME-LLongVideoBench-LAvg.
✗✗54.254.260.360.354.454.456.356.3
✓✗60.3_{{\color[rgb]{1,0,0}+6.1}}64.7_{{\color[rgb]{1,0,0}+4.4}}59.2_{{\color[rgb]{1,0,0}+4.8}}61.4_{{\color[rgb]{1,0,0}+5.1}}
✗✓56.4_{{\color[rgb]{1,0,0}+2.2}}61.5_{{\color[rgb]{1,0,0}+1.2}}55.6_{{\color[rgb]{1,0,0}+1.2}}57.8_{{\color[rgb]{1,0,0}+1.5}}
Joint QA Evolution57.5_{{\color[rgb]{1,0,0}+3.3}}62.5_{{\color[rgb]{1,0,0}+2.2}}55.7_{{\color[rgb]{1,0,0}+1.3}}58.6_{{\color[rgb]{1,0,0}+2.3}}
✓✓\mathbf{63.1}_{{\color[rgb]{1,0,0}+8.9}}\mathbf{66.4}_{{\color[rgb]{1,0,0}+6.1}}\mathbf{61.1}_{{\color[rgb]{1,0,0}+6.7}}\mathbf{63.5}_{{\color[rgb]{1,0,0}+7.2}}
Table 2: Effect of evolution strategies. ✓/✗ indicate whether evolution is performed.
Figure 3: Query-relevant evidence (white stars) cover only a limited portion of source information (blue and green dots).
Figure 4: Effect of text-memory budgets. Increasing the budget from 5K to 20K tokens substantially improves accuracy, while expanding it to 40K yields only marginal gains. This saturation may partly reflect visual details omitted from source captions, motivating complementary images.
Figure 5: Residual images preserve visual details omitted from textual memory. In this example, the textual observations describe the cluttered scene but omit the marked number. The retained image preserves these details, enabling the correct answer.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Trajectory of Reader evolution.
Figure 7: Trajectory of Writer evolution.
More in this topic

Memory-Augmented Video Understanding

Memory-augmented video understanding helps models retain and retrieve useful visual history for reasoning over long videos and live streams.

See all 85 papers

Explore similar work

CardsList
Sep 30, 2026

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6--12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7--5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems.

Guangzhi Xiong, Xinyuan Zhang, Xiao Yang +14

Meta Reality Labs · University of Virginia · Work done at Meta +1

Multimodal MemoryReinforcement LearningEgocentric Video QAAgent Memory Management
AskPDF
Sep 1, 2026

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).

Yijun Chen, Yaqi Zheng, Yanya Li +11

Zhejiang University · South China University of Technology · Lenovo Group Limited

Multimodal RAGEfficient Multimodal InferenceMultimodal MemoryMemory-Augmented Video UnderstandingLong-Video QA
AskPDF
May 15, 2026

Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrieval

Long video understanding requires more than large context windows. It also needs a memory mechanism that decides what visual evidence to retain, keeps it searchable over long horizons, and grounds later reasoning in recoverable observations rather than compressed latent state alone. We propose Visual Agentic Memory (VAM), a training-free framework with three components. Online Indexing supports selective evidence retention under streaming constraints. Hierarchical Memory organises retained evidence in a Parallel Representation that aligns temporal context with spatial observations. Agentic Retrieval searches, inspects, and verifies candidate evidence before producing a grounded answer. On OVO-Bench, VAM achieves the highest RT+BT average (68.41) across all reported baselines, improving over end-to-end use of the same underlying MLLM (Gemini 3 Flash, 67.46). On the month-scale split of MM-Lifelong train@month (105.6 hours over 51 days), VAM reaches 17.11%, second only to ReMA with GPT-5 (17.62%). These results suggest that long-horizon video understanding benefits from treating visual memory as an explicit, inspectable, and queryable substrate. Code is available at https://github.com/yiliu-li/Visual-Agentic-Memory.

Aiden Yiliu Li, Nels Numan, Anthony Steed

University College London

Streaming Video UnderstandingAgentic RetrievalLong-Video UnderstandingMemory-Augmented Video UnderstandingMemory-Augmented Agents
AskPDF