Autoregressive video generation has achieved impressive results on short clips, yet generating long videos remains challenging due to error accumulation over extended horizons, where each predicted frame depends on potentially imperfect previous frames. Existing approaches typically rely on static context management strategies, such as sliding-window caches that discard early frames or fixed sink frames that permanently preserve initial content. These designs either lose critical long-range context or anchor generation to outdated information, which can lead to degraded fidelity, motion stagnation, and reduced diversity. In this paper, we propose Learning to Memorize (L2M), a learned context memory management framework that dynamically retains, evicts, and evolves historical frames according to predicted importance. The framework consists of an importance prediction router that estimates the relevance of each historical frame, degradation-aware training that improves robustness to noisy or corrupted context, and a dynamic memory initialization and evolution mechanism that updates long-term memory using exponential moving average (EMA) importance scores. This allows high-quality frames to replace stale anchors and enables memory to adapt naturally to scene evolution. Trained with a teacher-student distillation objective and sparsity regularization, L2M makes efficient use of a fixed KV-cache budget while preserving the most informative contextual information. Extensive experiments demonstrate that L2M improves long-horizon video generation quality and temporal consistency over existing baselines, providing a learned strategy for context memory management in long autoregressive video generation.
Figures & tables
Figure 1: Overview of L2M. (a) Training phase : Given the current chunk feature fˉb and historical latent frame features {fˉi}i<(b−1)Bf+1 , the importance-aware frame router predicts importance scores for each historical latent frame. Binary frame selection masks mi∈{0,1} are obtained via Gumbel-Softmax discretization. The pruned context is fed to the student network, which is supervised by the teacher operating on the full context. Degradation-aware training augments a subset of historical frames with degradations, encouraging the router to assign them lower retention probabilities. (b) Inference phase : The router predicts frame importance scores, and low-scoring latent frames are evicted to meet the cache budget. High-scoring frames are dynamically selected as long-term memory and periodically updated using maintained EMA scores, continuously promoting informative historical frames.
Model
Aesthetic Quality
Background Consistency
Dynamic Degree
Imaging Quality
Motion Smoothness
Subject Consistency
Temporal Flickering
Quality Score
Overall Consistency
30 seconds
CausVid
61.80
90.70
54.69
66.98
98.06
87.69
96.80
79.18
22.92
Rolling Forcing
59.51
91.58
35.16
69.93
98.43
89.23
96.94
78.50
23.69
Self Forcing
60.28
90.81
44.53
68.73
97.97
87.08
97.00
78.39
24.10
Deep Forcing
61.17
91.90
59.38
68.08
97.70
87.56
96.12
79.37
24.55
L2M
62.69
93.48
57.81
69.54
98.17
90.99
96.71
81.15
24.78
Table 1: Quantitative comparison on long video generation. We evaluate our method against open-source autoregressive video diffusion baselines on 30-second and 60-second videos using multiple quality metrics from VBench-Long Huang et al. (2024) . All metrics follow the convention that higher values indicate better performance. The best results are in bold and the second best are underlined .
Figure 2: Qualitative comparison with baseline methods. Our method demonstrates superior visual quality and stronger overall consistency.
Figure 3: Visualization of layer-wise min-max normalized predicted importance scores.
Router
Degradation
Dynamic Sink Init.
Dynamic Sink Evolving
Quality Score ↑
Overall Cons. ↑
78.39
24.10
✓
80.32
24.44
✓
✓
80.56
24.52
✓
✓
✓
80.88
24.70
✓
✓
✓
✓
81.15
24.78
Table 2: Ablation study on individual components of our method. Results are evaluated on 30-second videos.
Figure 4: Qualitative analysis of dynamic long-term memory evolution. After memory updates, newly emerged informative frames are incorporated into sink memory, enabling better motion tracking and reducing long-term generation artifacts.
Method
0–5 s
5–10 s
10–15 s
15–20 s
20–25 s
25–30 s
Δ↓
Overall ↑
CausVid
24.62
23.27
22.87
22.35
22.32
22.10
2.52
22.92
Rolling Forcing
25.19
24.51
23.92
23.08
22.91
22.50
2.69
23.69
Self Forcing
25.28
25.28
24.70
24.52
23.11
21.79
3.49
24.11
Deep Forcing
25.28
24.74
24.47
24.39
24.24
24.20
1.08
24.55
L2M
25.28
24.79
24.72
24.71
24.65
24.54
0.74
24.78
Table 3: ViCLIP score on consecutive 5-second segments of 30-second videos. Overall Consistency is the average of the six segment scores. Δ is the drop from the first segment to the last. Higher is better for segment and overall scores; lower is better for Δ . Best results are in bold .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Aesthetic Quality
Background Consistency
Dynamic Degree
Imaging Quality
Motion Smoothness
Subject Consistency
Temporal Flickering
Quality Score
Overall Consistency
CausVid
61.13
88.46
45.31
66.20
98.15
83.73
97.08
77.22
21.60
Rolling Forcing
57.86
89.87
36.72
69.47
98.31
85.03
96.80
77.06
22.59
Self Forcing
51.14
84.00
28.91
63.09
98.10
74.33
97.52
71.48
17.04
Deep Forcing
61.48
93.06
44.53
68.88
98.13
89.56
97.00
79.59
24.33
L2M
61.88
92.93
53.12
69.56
98.14
89.72
96.71
80.30
24.47
Appendix
Table 4: Quantitative comparison on 120-second long video generation. We evaluate our method against open-source autoregressive video diffusion baselines using multiple quality metrics from VBench-Long Huang et al. [2024] . All metrics follow the convention that higher values indicate better performance. The best results are in bold and the second best are underlined .
Model
Aesthetic
Background
Dynamic
Imaging
Motion
Subject
Temporal
Quality
Overall
Quality
Consistency
Degree
Quality
Smoothness
Consistency
Flickering
Score
Consistency
CausVid
60.67
87.78
44.53
66.00
98.21
82.54
97.24
76.80
21.12
Rolling Forcing
57.40
89.48
36.72
69.36
98.27
84.11
96.76
76.68
22.40
Self Forcing
44.89
81.27
24.22
58.75
97.92
68.82
97.46
67.81
13.76
Deep Forcing
59.70
89.80
42.19
66.84
97.68
83.82
96.40
76.62
23.33
LongLive
61.23
92.39
40.62
69.38
98.31
89.04
97.58
79.43
24.03
Appendix
Table 5: Quantitative comparison on 240-second long video generation. Higher is better for all metrics. The best results are in bold , and the second-best results are underlined .
Baseline
Scene
Motion
Subject
Overall
Consistency
Quality
Consistency
Quality
CausVid
90.7
100.0
90.7
96.0
Rolling Forcing
80.8
76.2
84.5
78.8
Self Forcing
93.0
97.6
93.0
93.0
Deep Forcing
94.0
85.0
96.0
94.0
Appendix
Table 6: Human evaluation using two-alternative forced choice (2AFC). Each entry denotes the percentage of votes favoring L2M over the corresponding baseline. Higher is better. The best results are in bold , and the second-best results are underlined .
Model
Perceptual Quality ↑
CausVid
70.76
Rolling Forcing
75.53
Self Forcing
70.44
Deep Forcing
73.81
L2M
77.37
Appendix
Table 7: Quantitative comparison on LoCoT2V-Bench. We report Perceptual Quality (PQ) for 30-second videos. Higher is better. The best result is in bold , and the second-best result is underlined .
Method
Quality Score ↑
Overall Consistency ↑
LongLive
80.73
24.58
LongLive + L2M
81.20
24.84
Appendix
Table 8: Integration of L2M into LongLive. Results are evaluated on 30-second videos using 128 MovieGen prompts. Higher is better for both metrics.
Method
Memory (GB)
Inference Time (s)
Baseline
36.05
78.55
Ours
36.32
79.21
Appendix
Table 9: Inference cost comparison on 30-second video generation.
Figure 14Figure 15
Cache Budget ( K )
Quality Score ↑
Overall Consistency ↑
18
80.98
24.72
21
81.15
24.78
24
81.05
24.79
27
81.10
24.83
30
81.02
24.84
Appendix
Table 10: Comparison of cache budget K . Higher is better for both metrics. The best results are in bold , and the second-best results are underlined .
Training Setting
Quality Score ↑
Overall Consistency ↑
No degradation ( p=0,β=0 )
80.66
24.68
Default ( p=0.2,β=0.1 )
81.15
24.78
Stronger degradation
80.92
24.73
Appendix
Table 11: Comparison of degradation strength. Higher is better for both metrics. The best results are in bold .
Figure 9: Visualization of attention weights across layers. Query-averaged attention maps illustrate how each key frame is attended during the generation of the final chunk.
Figure 10: Consistency of importance scores across diffusion timesteps. The stable importance patterns observed at different timesteps (1000, 750, 500, 250, and the final clean KV cache) indicate that predictions made by the Router at the initial timestep ( t=1000 ) remain reliable throughout the entire denoising process.
Context Pruning
Quality Score
Overall Consistency
L2M
81.15
24.78
Uniform pruning (10%)
80.90
24.72
Uniform pruning (20%)
80.68
24.67
Appendix
Table 12: Results of additional context pruning at later timesteps.
Figure 12: Failure cases of L2M. In the first example, rapid camera movement and highly dynamic scene changes cause abrupt content inconsistency, where a new house artifact appears after the waves sweep across the scene. In the second example, an incorrectly generated two-headed zebra appears in several initial frames and remains in subsequent generations due to delayed memory eviction.
School of Computer Science and Engineering, Southeast University · Key Lab. of Computer Network and Information Integration, Southeast University · Southeast University +3