Autoregressive video generation has achieved impressive results on short clips, yet generating long videos remains challenging due to error accumulation over extended horizons, where each predicted frame depends on potentially imperfect previous frames. Existing approaches typically rely on static context management strategies, such as sliding-window caches that discard early frames or fixed sink frames that permanently preserve initial content. These designs either lose critical long-range context or anchor generation to outdated information, which can lead to degraded fidelity, motion stagnation, and reduced diversity. In this paper, we propose Learning to Memorize (L2M), a learned context memory management framework that dynamically retains, evicts, and evolves historical frames according to predicted importance. The framework consists of an importance prediction router that estimates the relevance of each historical frame, degradation-aware training that improves robustness to noisy or corrupted context, and a dynamic memory initialization and evolution mechanism that updates long-term memory using exponential moving average (EMA) importance scores. This allows high-quality frames to replace stale anchors and enables memory to adapt naturally to scene evolution. Trained with a teacher-student distillation objective and sparsity regularization, L2M makes efficient use of a fixed KV-cache budget while preserving the most informative contextual information. Extensive experiments demonstrate that L2M improves long-horizon video generation quality and temporal consistency over existing baselines, providing a learned strategy for context memory management in long autoregressive video generation.
Figures & tables
Figure 1: Overview of L2M. (a) Training phase : Given the current chunk feature fˉb and historical latent frame features {fˉi}i<(b−1)Bf+1 , the importance-aware frame router predicts importance scores for each historical latent frame. Binary frame selection masks mi∈{0,1} are obtained via Gumbel-Softmax discretization. The pruned context is fed to the student network, which is supervised by the teacher operating on the full context. Degradation-aware training augments a subset of historical frames with degradations, encouraging the router to assign them lower retention probabilities. (b) Inference phase : The router predicts frame importance scores, and low-scoring latent frames are evicted to meet the cache budget. High-scoring frames are dynamically selected as long-term memory and periodically updated using maintained EMA scores, continuously promoting informative historical frames.
Model
Aesthetic Quality
Background Consistency
Dynamic Degree
Imaging Quality
Motion Smoothness
Subject Consistency
Temporal Flickering
Quality Score
Overall Consistency
30 seconds
CausVid
61.80
90.70
54.69
66.98
98.06
87.69
96.80
79.18
22.92
Rolling Forcing
59.51
91.58
35.16
69.93
98.43
89.23
96.94
78.50
23.69
Self Forcing
60.28
90.81
44.53
68.73
97.97
87.08
97.00
78.39
24.10
Deep Forcing
61.17
91.90
59.38
68.08
97.70
87.56
96.12
79.37
24.55
L2M
62.69
93.48
57.81
69.54
98.17
90.99
96.71
81.15
24.78
Table 1: Quantitative comparison on long video generation. We evaluate our method against open-source autoregressive video diffusion baselines on 30-second and 60-second videos using multiple quality metrics from VBench-Long Huang et al. (2024) . All metrics follow the convention that higher values indicate better performance. The best results are in bold and the second best are underlined .
Figure 2: Qualitative comparison with baseline methods. Our method demonstrates superior visual quality and stronger overall consistency.
Figure 3: Visualization of layer-wise min-max normalized predicted importance scores.
Router
Degradation
Dynamic Sink Init.
Dynamic Sink Evolving
Quality Score ↑
Overall Cons. ↑
78.39
24.10
✓
80.32
24.44
✓
✓
80.56
24.52
✓
✓
✓
80.88
24.70
✓
✓
✓
✓
81.15
24.78
Table 2: Ablation study on individual components of our method. Results are evaluated on 30-second videos.
Figure 4: Qualitative analysis of dynamic long-term memory evolution. After memory updates, newly emerged informative frames are incorporated into sink memory, enabling better motion tracking and reducing long-term generation artifacts.
Method
0–5 s
5–10 s
10–15 s
15–20 s
20–25 s
25–30 s
Δ↓
Overall ↑
CausVid
24.62
23.27
22.87
22.35
22.32
22.10
2.52
22.92
Rolling Forcing
25.19
24.51
23.92
23.08
22.91
22.50
2.69
23.69
Self Forcing
25.28
25.28
24.70
24.52
23.11
21.79
3.49
24.11
Deep Forcing
25.28
24.74
24.47
24.39
24.24
24.20
1.08
24.55
L2M
25.28
24.79
24.72
24.71
24.65
24.54
0.74
24.78
Table 3: ViCLIP score on consecutive 5-second segments of 30-second videos. Overall Consistency is the average of the six segment scores. Δ is the drop from the first segment to the last. Higher is better for segment and overall scores; lower is better for Δ . Best results are in bold .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Aesthetic Quality
Background Consistency
Dynamic Degree
Imaging Quality
Motion Smoothness
Subject Consistency
Temporal Flickering
Quality Score
Overall Consistency
CausVid
61.13
88.46
45.31
66.20
98.15
83.73
97.08
77.22
21.60
Rolling Forcing
57.86
89.87
36.72
69.47
98.31
85.03
96.80
77.06
22.59
Self Forcing
51.14
84.00
28.91
63.09
98.10
74.33
97.52
71.48
17.04
Deep Forcing
61.48
93.06
44.53
68.88
98.13
89.56
97.00
79.59
24.33
L2M
61.88
92.93
53.12
69.56
98.14
89.72
96.71
80.30
24.47
Appendix
Table 4: Quantitative comparison on 120-second long video generation. We evaluate our method against open-source autoregressive video diffusion baselines using multiple quality metrics from VBench-Long Huang et al. [2024] . All metrics follow the convention that higher values indicate better performance. The best results are in bold and the second best are underlined .
Model
Aesthetic
Background
Dynamic
Imaging
Motion
Subject
Temporal
Quality
Overall
Quality
Consistency
Degree
Quality
Smoothness
Consistency
Flickering
Score
Consistency
CausVid
60.67
87.78
44.53
66.00
98.21
82.54
97.24
76.80
21.12
Rolling Forcing
57.40
89.48
36.72
69.36
98.27
84.11
96.76
76.68
22.40
Self Forcing
44.89
81.27
24.22
58.75
97.92
68.82
97.46
67.81
13.76
Deep Forcing
59.70
89.80
42.19
66.84
97.68
83.82
96.40
76.62
23.33
LongLive
61.23
92.39
40.62
69.38
98.31
89.04
97.58
79.43
24.03
Appendix
Table 5: Quantitative comparison on 240-second long video generation. Higher is better for all metrics. The best results are in bold , and the second-best results are underlined .
Baseline
Scene
Motion
Subject
Overall
Consistency
Quality
Consistency
Quality
CausVid
90.7
100.0
90.7
96.0
Rolling Forcing
80.8
76.2
84.5
78.8
Self Forcing
93.0
97.6
93.0
93.0
Deep Forcing
94.0
85.0
96.0
94.0
Appendix
Table 6: Human evaluation using two-alternative forced choice (2AFC). Each entry denotes the percentage of votes favoring L2M over the corresponding baseline. Higher is better. The best results are in bold , and the second-best results are underlined .
Model
Perceptual Quality ↑
CausVid
70.76
Rolling Forcing
75.53
Self Forcing
70.44
Deep Forcing
73.81
L2M
77.37
Appendix
Table 7: Quantitative comparison on LoCoT2V-Bench. We report Perceptual Quality (PQ) for 30-second videos. Higher is better. The best result is in bold , and the second-best result is underlined .
Method
Quality Score ↑
Overall Consistency ↑
LongLive
80.73
24.58
LongLive + L2M
81.20
24.84
Appendix
Table 8: Integration of L2M into LongLive. Results are evaluated on 30-second videos using 128 MovieGen prompts. Higher is better for both metrics.
Method
Memory (GB)
Inference Time (s)
Baseline
36.05
78.55
Ours
36.32
79.21
Appendix
Table 9: Inference cost comparison on 30-second video generation.
Figure 14Figure 15
Cache Budget ( K )
Quality Score ↑
Overall Consistency ↑
18
80.98
24.72
21
81.15
24.78
24
81.05
24.79
27
81.10
24.83
30
81.02
24.84
Appendix
Table 10: Comparison of cache budget K . Higher is better for both metrics. The best results are in bold , and the second-best results are underlined .
Training Setting
Quality Score ↑
Overall Consistency ↑
No degradation ( p=0,β=0 )
80.66
24.68
Default ( p=0.2,β=0.1 )
81.15
24.78
Stronger degradation
80.92
24.73
Appendix
Table 11: Comparison of degradation strength. Higher is better for both metrics. The best results are in bold .
Figure 9: Visualization of attention weights across layers. Query-averaged attention maps illustrate how each key frame is attended during the generation of the final chunk.
Figure 10: Consistency of importance scores across diffusion timesteps. The stable importance patterns observed at different timesteps (1000, 750, 500, 250, and the final clean KV cache) indicate that predictions made by the Router at the initial timestep ( t=1000 ) remain reliable throughout the entire denoising process.
Context Pruning
Quality Score
Overall Consistency
L2M
81.15
24.78
Uniform pruning (10%)
80.90
24.72
Uniform pruning (20%)
80.68
24.67
Appendix
Table 12: Results of additional context pruning at later timesteps.
Figure 12: Failure cases of L2M. In the first example, rapid camera movement and highly dynamic scene changes cause abrupt content inconsistency, where a new house artifact appears after the waves sweep across the scene. In the second example, an incorrectly generated two-headed zebra appears in several initial frames and remains in subsequent generations due to delayed memory eviction.
Autoregressive video diffusion has recently emerged as a promising paradigm for long-video generation, enabling causal synthesis beyond the temporal limits of bidirectional models. Existing forcing-based training strategies reduce exposure bias by conditioning models on their own predictions during rollout, yet minute-scale generation remains challenging due to progressive temporal degradation and constrained motion evolution. In this work, we study the role of temporal KV memory during long-horizon autoregressive inference. Our analysis shows that simply retaining more historical frames does not consistently improve generation quality; instead, both the quantity and temporal placement of memory strongly affect motion dynamics. These findings suggest that temporal memory should be treated as structured context rather than a homogeneous chronological buffer. Motivated by this observation, we introduce Relax Forcing, a training-free memory mechanism for autoregressive video diffusion. Relax Forcing decomposes temporal context into three functional components: Sink frames that provide global stability, Tail frames that preserve short-term continuity, and dynamically selected History frames that supply mid-range motion structure. History frames are selected using a relaxation-based criterion that encourages alignment with global anchors while suppressing redundancy with recent context. This role-aware sparse memory design mitigates error accumulation during long-horizon rollout while preserving motion evolution and reducing attention overhead. Experiments on VBench-Long show that Relax Forcing improves long-video generation, achieving stronger motion dynamics and higher overall scores than existing autoregressive baselines. These results indicate that structured temporal memory is an effective and complementary direction for scalable long-video generation.
Zengqun Zhao, Yanzuo Lu, Ziquan Liu +3
Queen Mary University of London · Imperial College London · Huawei R&D UK
Autoregressive long video generation often adopts bounded-memory streaming for efficiency, typically combining local windows for short-term continuity with static early-frame sinks as long-range anchors. However, this fixed allocation keeps early frames cached even when the current visual state has substantially diverged from them, while discarding potentially more relevant intermediate history. As a result, the retained long-range context may become less adaptive and bias generation toward outdated cues; in severe cases, RoPE-induced phase re-alignment can homogenize inter-head attention and cause sink collapse, where content regresses toward sink frames. We propose DySink, a retrieval-based framework that maintains a compact memory bank and selects visually relevant historical frames as dynamic frame sinks. DySink couples adaptive retrieval with a sink anomaly gate, which detects excessive inter-head consensus over retrieved context and suppresses collapse-prone context. Experiments on minute-long videos show that DySink consistently improves temporal quality over strong baselines while also achieving higher dynamic degree, enabling coherent and more natural long-horizon visual evolution. The code and model weights are released at https://github.com/yebo0216best/DySink.
Bo Ye, Xinyu Cui, Jian Zhao +2
School of Computer Science and Engineering, Southeast University · Key Lab. of Computer Network and Information Integration, Southeast University · Southeast University +3
Autoregressive (AR) video generation degrades over long horizons due to an overlooked train-inference discrepancy we term KV eviction mismatch: models train on short clips where all context frames reside in the KV cache, but at inference, memory constraints force distant frames to be evicted from the KV cache - removing context the model was conditioned on. Rather than simulating eviction via context truncation - which discards temporal information the model still needs and degrades motion coherence - we keep the context but while progressively reducing the influence of distant frames, making their eventual eviction negligible. To guide this design, we introduce the positional response R(Δ,tdenoise), a perturbation-based sensitivity measure revealing that context influence decays steeply with temporal distance and varies systematically across denoising steps. Motivated by this analysis, we propose Recency Forcing, which applies a non-positive, timestep-dependent bias, termed Temporal Response Bias (TRB), on pre-softmax attention logits derived directly from R, closing the train-inference gap without modifying context length or training objectives. We further introduce Biased Attention Reparameterization (BAR), an exact reformulation that moves the bias outside the softmax, making TRB a standard FlashAttention call at zero overhead. Recency Forcing operates in both training-free mode and training-based mode. Experiments on VBench and VBench-Long demonstrate state-of-the-art long-horizon generation quality at no additional inference cost.