cs.CVNov 19, 2026

EvoMem: Self-Evolving Skills for Time-Aware Long Video Memory Agents

Authors: Yutong Guan, Meixing Shi, Haoran Sun, Bing Sun, Yuxiang Cai, Yankai Jiang

Organizations: Zhejiang University · Shanghai Artiffcial Intelligence Laboratory · Research Center, National Certiffcation Technology (Hangzhou) Co., Ltd

Abstract

Although memory-augmented architectures have expanded the ability of large vision-language models (LVLMs) to process long videos, existing retrieval processes still largely rely on global semantic matching and lack state-driven temporal control, often returning fragmented or redundant evidence and failing to maintain a coherent evidence chain. To address this challenge, we propose EvoMem, a time-aware long-video memory agent guided by self-evolving retrieval skills. EvoMem constructs a hierarchical time-indexed memory, where different types of evidence are aligned onto a unified video timeline; during retrieval, temporal cues from acquired evidence are further converted into explicit signals that constrain subsequent memory access. To address recurring procedural errors in long-video question answering, such as unclear retrieval targets, improper memory access paths, or premature stopping, EvoMem introduces retrieval skills to provide process-level guidance, enabling the agent to dynamically adjust retrieval content, memory operations, and target temporal ranges according to the current evidence state, while deciding when to stop. Furthermore, inspired by reinforcement learning, we formulate skill evolution as a reward-guided policy improvement process. Each candidate skill is treated as an external policy patch, and rewards are assigned according to answer-correctness state transitions on the validation set. In this way, effective skills are accepted and low-quality skills are rejected without updating model parameters. Experiments on four long-video question answering benchmarks show that EvoMem significantly outperforms existing baselines, achieving an average accuracy of 60.1% and an average improvement of 8.6 percentage points over the previous best methods, demonstrating its effectiveness for long-video reasoning.

Figures & tables

Explore similar work

CardsList
  1. Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

    Oct 17, 2026Kerui Chen, Jinglu Wang, Yun Wang +2Memory-Augmented Video UnderstandingMemory-Augmented VLMs

  2. VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

    Oct 7, 2026Yongchao Xu, Bowen Ye, Jiefeng Gan +5Long-Video UnderstandingMemory-Augmented Video Understanding

  3. VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

    Oct 5, 2026Yucheng Liu, Yufei Yin, Mingxiao Feng +3Long-Video UnderstandingMemory-Augmented Video Understanding