Autoregressive video diffusion models enable streaming generation through sliding-window attention, but each generated block is conditioned on previously generated content, causing appearance and motion errors to propagate recursively over time. Historical key-value (KV) memory preserves earlier subject and scene states and helps maintain long-horizon consistency. However, retaining every generated state creates a historical archive that grows continuously with the rollout, while recurrent states repeatedly add redundant coverage. To address this problem, we propose DensityKV, a training-free historical KV bank management strategy. DensityKV maintains a separate token-level KV bank for each attention head and measures local redundancy among the post-RoPE keys that directly parameterize attention routing using Soft-Riesz density. By constraining neighborhood-density growth after states enter the bank, DensityKV limits repeated historical accumulation while preserving coherent states from each completed generation block. Experiments across three autoregressive video generation backbones and multiple generation lengths show that, at the same upper bound on historical KV capacity, DensityKV improves long-horizon consistency and generation stability while keeping persistent historical storage bounded independently of rollout length.
Figures & tables
Figure 2: Overview of historical-memory construction and use. Left: sliding-window generation discards states outside the active context; LongLive-RAG stores discarded history and retrieves selected blocks; DensityKV instead maintains a bounded token-level history that is attended together with the local window. Right: at every Transformer layer, clean historical K/V states update independent per-head banks. Admission and rejection are determined by Soft-Riesz density over post-RoPE keys, while each value remains exactly paired with its original key. Current queries access the retained history through the backbone’s native attention.
Figure 3: Qualitative comparison on two matched backbone–prompt pairs, each generated for 120 seconds. The upper group uses LongLive on MovieGenBench #097, the wig-and-sunglasses transformation; the lower group uses Causal-Forcing on MovieGenBench #012, the disco-dancing kangaroo.
Time
Backbone
Method
Subject ↑
Background ↑
Smoothness ↑
Dynamic ↑
Aesthetic ↑
Imaging ↑
Avg. Rank ↓
30s
Self-Forcing ( 2025 )
Native
96.21
95.39
98.39
52.03
56.69
63.31
4.00
+∞ -RoPE ( 2025 )
97.32
96.38
98.59
46.82
56.78
63.93
2.83
+ Deep Forcing ( 2025 )
97.04
96.02
98.57
38.85
56.44
61.91
4.50
+ LongLive-RAG ( 2026 )
97.57
96.56
98.76
42.24
57.17
65.43
2.17
+ DensityKV
97.79
96.72
98.72
44.79
57.68
66.93
1.50
LongLive ( 2025 )
Native
97.35
96.15
98.70
44.74
59.38
68.15
3.33
Table 1: Comparison across backbones. Bold/underline mark best/second-best values; Avg. Rank is averaged over the six metrics.
(a) Parameter sensitivity
Parameter
Setting
Subject ↑
Background ↑
Smoothness ↑
Dynamic ↑
Aesthetic ↑
Imaging ↑
Avg. Rank ↓
τ
1.5
97.03
96.29
98.26
52.57
57.97
68.70
2.17
2.0∗
97.19
96.34
98.33
51.94
57.74
68.73
1.83
3.0
97.20
96.33
98.35
44.44
57.83
65.50
2.00
M
6,240
97.25
96.31
98.34
49.38
57.61
67.06
2.17
9,360 ∗
97.19
96.34
98.33
51.94
57.74
68.73
1.67
Table 2: Parameter sensitivity and policy ablations. Bold/underline mark best/second-best displayed values within each ablation group.
Figure 4: Persistent temporal-state storage versus generation length. LongLive-RAG grows linearly, whereas Deep Forcing and DensityKV remain bounded.
Figure 5: Single-metric wins under the matched LongLive-RAG protocol at 30, 60, and 120 seconds.
Figure 6: Extended qualitative comparison. Causal-Forcing on MovieGenBench prompts #028, #048, #095, #109, and #110. Rows: Native, LongLive-RAG, and post-RoPE K-only DensityKV. Columns: 1, 10, 20, 30, 60, and 120 seconds from matched trajectories.
Figure 7: Self-Forcing on MovieGenBench prompts #104, #068, #040, #009, and #004. Rows: Native, LongLive-RAG, and DensityKV. Columns: 1, 10, 20, 30, 60, and 120 seconds.
Figure 8: LongLive on MovieGenBench prompts #006, #007, #010, #032, and #063. Rows: Native, LongLive-RAG, and DensityKV. Columns: 1, 10, 20, 30, 60, and 120 seconds.
Figure 9: Admission analysis for the Causal-Forcing panda-guitar diagnostic (seed 13). (a) Orange intensity gives the number of layer–head banks that admit each spatial token, on a logarithmic scale. (b) Admission counts aggregated by layer and source time. (c) Fractions of possible layer–head entries and unique spatial positions admitted from each source frame. The gray region in (b)–(c) marks initial bank construction from source frames 1–6, whose timestamps span 0–1.25 seconds at 4 fps.
Autoregressive (AR) video diffusion models enable streaming generation of long videos but suffer from significant memory overhead due to the rapidly growing KV cache. Existing KV cache quantization methods are primarily designed for LLMs and transfer poorly to video models, where activations exhibit different statistics and strong spatiotemporal redundancy. We observe that the video KV cache is largely predictable over time, with a clear asymmetry. Keys are highly predictable from their temporal history, while values depend primarily on the current key and need a short history of preceding values. Based on these observations, we propose \method, a training-free predictive KV cache quantization framework for AR video generation. \method predicts keys and values according to their distinct dependencies and stores only the residuals at low precision. Both predictors are lightweight, calibrated once offline by linear regression on a small set of BF16 samples, and remain fixed across prompts and generation runs. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B show that \method consistently achieves higher generation quality than existing KV cache quantization methods at comparable or higher compression ratios, reaching up to 6.71× KV cache compression with the best overall quality--efficiency tradeoff. Our code will be released on GitHub.
Jiahui Hu, Weile Mo, Peiliang Cai +5
Shanghai Jiao Tong University · South China University of Technology · Alibaba Group
Recent advances in autoregressive video diffusion have enabled sequential and streaming video generation. However, long-horizon generation requires increasingly large KV caches, making efficient compression without sacrificing quality challenging. Existing methods mostly select historical frames based on attention scores, but their context decisions remain coarse. When multiple frames are generated in the same chunk, these methods often apply a shared history selection to the whole chunk, score historical frames solely by attention, and assign head-wise budgets either uniformly or by attention-pattern heuristics rather than explicit head-importance estimation. We show that frames within the same generated chunk can depend on distinct historical frames, that the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, and that masking different heads induces unequal generation degradation. Motivated by these findings, we propose Focused Forcing, a training-free KV selection method that focuses cached history along both generated-frame and head dimensions. For each generated frame, Focused Forcing preserves the most relevant and distinctive historical frames by combining attention scores with diversity scores of historical frames, while assigning larger budgets to heads with higher estimated importance. Across multiple autoregressive generation paradigms, Focused Forcing achieves up to 1.48× end-to-end acceleration without training, while improving visual quality and text alignment. Our code will be released on GitHub.
Autoregressive (AR) video diffusion models adopt a streaming generation framework, enabling long-horizon video generation with real-time responsiveness, as exemplified by the Self Forcing training paradigm. However, existing AR video diffusion models still suffer from significant attention complexity and severe memory overhead due to the redundant key-value (KV) caches across historical frames, which limits scalability. In this paper, we tackle this challenge by introducing KV cache compression into autoregressive video diffusion. We observe that attention heads in mainstream AR diffusion models exhibit markedly distinct attention patterns and functional roles that remain stable across samples and denoising steps. Building on our empirical study of head-wise functional specialization, we divide the attention heads into two categories: static heads, which focus on transitions across autoregressive chunks and intra-frame fidelity, and dynamic heads, which govern inter-frame motion and consistency. We then propose Forcing-KV, a hybrid KV cache compression strategy that performs structured static pruning for static heads and dynamic pruning based on segment-wise similarity for dynamic heads. While maintaining output quality, our method achieves a generation speed of over 29 frames per second on a single NVIDIA H200 GPU along with 30% cache memory reduction, delivering up to 1.35x and 1.50x speedups on LongLive and Self Forcing at 480P resolution, and further scaling to 2.82x speedup at 1080P resolution. Code and demo videos are provided at https://zju-jiyicheng.github.io/Forcing-KV-Page.