cs.CVNov 1, 2026

Learning-to-Memorize: Dynamic Context Management for Long-Horizon Autoregressive Video Generation

Authors: Haowei Zhu, Qijie Wang, Jia Li, Xing Wang, Tianyu Zhao, Jiexi Wang, Xurui Peng, Xinglong Wu, 

Organizations: Tsinghua University · ByteDance Seed

Abstract

Autoregressive video generation has achieved impressive results on short clips, yet generating long videos remains challenging due to error accumulation over extended horizons, where each predicted frame depends on potentially imperfect previous frames. Existing approaches typically rely on static context management strategies, such as sliding-window caches that discard early frames or fixed sink frames that permanently preserve initial content. These designs either lose critical long-range context or anchor generation to outdated information, which can lead to degraded fidelity, motion stagnation, and reduced diversity. In this paper, we propose Learning to Memorize (L2M), a learned context memory management framework that dynamically retains, evicts, and evolves historical frames according to predicted importance. The framework consists of an importance prediction router that estimates the relevance of each historical frame, degradation-aware training that improves robustness to noisy or corrupted context, and a dynamic memory initialization and evolution mechanism that updates long-term memory using exponential moving average (EMA) importance scores. This allows high-quality frames to replace stale anchors and enables memory to adapt naturally to scene evolution. Trained with a teacher-student distillation objective and sparsity regularization, L2M makes efficient use of a fixed KV-cache budget while preserving the most informative contextual information. Extensive experiments demonstrate that L2M improves long-horizon video generation quality and temporal consistency over existing baselines, providing a learned strategy for context memory management in long autoregressive video generation.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation

    Mar 22, 2026Zengqun Zhao, Yanzuo Lu, Ziquan Liu +3Long-Horizon Video GenerationAutoregressive Video Generation

  2. DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation

    May 20, 2026Bo Ye, Xinyu Cui, Jian Zhao +2Long-Horizon Video GenerationMemory-Augmented Video Generation