Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Agentic autoregressive video generation combines semantic planning, sequential generation, and visual verification for controllable synthesis. Growing visual histories enlarge the key-value (KV) cache, increasing memory use and attention costs across candidate rollouts. Existing compression methods primarily optimize individual generators without the planner's semantic constraints, risking low priority for states supporting requested attributes or actions. We introduce SemanticKV, a training-free method for semantically aligned KV cache compression. To our knowledge, SemanticKV is the first KV cache compression method for agentic autoregressive video generation that explicitly uses planner-extracted semantic constraints to guide retention. Text cross-attention identifies constraint-related video positions, whose self-attention queries are collected into separate groups. Attention weights are averaged within each group and maximized across groups, giving each cached entry a high score when strongly used by any represented constraint. A single top-k selection retains high-scoring entries under the memory budget, leaving their keys, values, and positions unchanged. For fixed query groups and equal entry costs, this selection minimizes an additive upper bound on worst-group discarded attention. The method operates independently within each candidate without retraining or additional video generations. We evaluate four AR video backbones on three benchmarks for 5-, 30-, and 60-second generation. On Causal Forcing at a compression ratio of 0.7, SemanticKV obtains a VBench Total Score of 84.10, compared with 84.05 for the strongest compressed baseline and 83.99 for Full KV. It achieves a 1.22x speedup over Full KV and reduces KV-cache memory from 6.04 GB to 1.81 GB, a 3.35x reduction.
Figures & tables
Figure 1: Agentic AR video generation. Semantic constraints R and prompt-specific exclusions N guide prompt refinement. The original-prompt and planned candidates from the same generator undergo randomized, anonymous visual verification, followed by conservative selection using Eq. 7 .
Figure 2: Overview of SemanticKV . Semantic constraints R are matched to prompt tokens. Text cross-attention identifies related video positions, from which we collect the generator’s self-attention queries. Their attention to each KV entry is averaged within groups and maximized across groups. A single top- k selection retains high-scoring entries within the byte budget, together with states required by the backbone. Retained keys, values, and positions remain unchanged.
Backbone
Method
Efficiency
KV Cache
Temporal Quality
VBench: Overall and Semantic Fidelity
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Disc. ↓
Dynamic ↑
Smooth. ↑
Total Score ↑
Object Class ↑
Multiple Objects ↑
Human Action ↑
Color ↑
Spatial Relation ↑
Scene ↑
Self Forcing
Full KV
6.95
1.00 ×
6.04
1.00 ×
2.25
91.67
97.77
82.96
89.83
75.55
92.00
90.54
71.67
48.05
R-KV
7.36
1.06 ×
1.81
3.35 ×
2.23
92.78
97.77
82.90
88.29
74.15
93.00
91.40
71.66
47.53
Dummy Forcing
8.16
1.17 ×
1.81
3.35 ×
2.23
91.39
97.81
82.87
89.26
77.07
92.00
91.12
71.01
47.43
Forcing-KV
7.92
1.14 ×
1.81
3.35 ×
2.62
92.78
97.55
82.69
88.21
75.88
92.60
90.64
71.44
47.72
SemanticKV
7.95
1.14 ×
1.81
3.35 ×
2.26
93.33
97.80
83.03
89.43
75.47
92.00
91.30
72.46
48.65
Table 1: Five-second VBench at compression ratio 0.7 : Total Score, temporal quality, and six shared semantic diagnostics. Table 3 gives all 16 dimensions. Full KV is the uncompressed reference; efficiency is DiT-only. ↑ / ↓ : higher/lower is better. Bold : best compressed result for Total Score.
Figure 3: Five-second VBench across compression ratios 0.5/0.7/0.9 , with the same five methods and six dimensions; higher is better. Table 4 gives exact scores and DiT-only efficiency.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Component and role
Example
Original prompt q0 Before refinement.
“ park , one girl one boy , red coat on girl blue coat on boy , yellow ball , girl gives it to boy then boy lifts it over his head , fixed camera ”
Semantic constraints R What to preserve.
r1 : One girl in a red coat. r2 : One boy in a blue coat. r3 : A yellow ball. r4 : The girl gives the ball to the boy. r5 : The boy then lifts the ball over his head. r6 : A park. r7 : A fixed camera.
Prompt-specific exclusions N What to avoid.
Swapped coat colors; reversed giver and receiver; reversed action order; camera movement; additional people, objects, or actions.
Refined prompt q1 After refinement.
“ In a park , one girl in a red coat gives a yellow ball to one boy in a blue coat . The boy then lifts the same ball over his head . The camera remains fixed .”
Appendix
Table 2: An illustrative example of conservative semantic planning. Matching colors trace the requested content from q0 through R to q1 : subjects and attributes , actions and their order , and scene and camera . Prompt-specific exclusions identify changes to avoid.
Backbone
Length
Method
Efficiency
KV Cache
Temporal Quality
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Temporal Flicker. ↑
Motion Smooth. ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Aesthetic Quality ↑
Self Forcing
5 s
Full KV
6.95
1.00 ×
6.04
1.00 ×
2.25
91.67
98.54
97.77
82.96
67.50
93.84
95.24
61.07
R-KV
7.36
1.06 ×
1.81
3.35 ×
2.23
92.78
98.45
97.77
82.90
67.52
93.54
95.01
61.26
Dummy Forcing
8.16
1.17 ×
1.81
3.35 ×
2.23
91.39
98.42
97.81
82.87
67.49
93.49
95.22
61.09
Forcing-KV
7.92
1.14 ×
1.81
3.35 ×
2.62
92.78
98.41
97.55
82.69
67.06
93.28
95.14
60.95
SemanticKV
7.95
1.14 ×
1.81
3.35 ×
2.26
93.33
98.50
97.80
83.03
67.56
93.49
95.10
60.99
Appendix
Table 3: Efficiency and generation quality on VBench for 5-second video generation at compression ratio 0.7 . Full KV retains the native cache without additional eviction and serves as the reference. Speedup is measured relative to Full KV. VBench metrics retain their reported percentage-scaled values. ↑ indicates higher is better and ↓ indicates lower is better. Bold marks the best result among compressed methods for each backbone.
Compression Ratio
Length
Method
Efficiency
KV Cache
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Temporal Flicker. ↑
Subject Cons. ↑
Background Cons. ↑
Object Class ↑
Human Action ↑
Color ↑
Full KV
6.57
1.00 ×
3.45
1.00 ×
99.26
96.06
96.68
90.84
92.60
90.34
0.5
5 s
R-KV
7.53
1.15 ×
1.72
2.01 ×
99.32
96.07
96.61
91.01
93.00
90.84
Dummy Forcing
7.32
1.11 ×
1.72
2.01 ×
98.81
95.61
95.95
90.09
93.60
88.95
Forcing-KV
7.67
1.17 ×
1.72
2.01 ×
99.29
95.92
96.63
90.84
94.00
89.73
SemanticKV
8.00
1.22 ×
1.72
2.01 ×
99.29
96.04
96.52
91.93
92.80
89.24
Appendix
Table 4: VBench results across compression ratios for 5-second generation, reporting the six dimensions in Figure 3 and DiT-only efficiency. Full KV is the shared uncompressed reference within this run group. Speedup is measured relative to Full KV. VBench metrics retain their reported percentage-scaled values. ↑ indicates higher is better. Bold marks the best result among compressed methods at each compression ratio.
Backbone
Length
Method
Efficiency
KV Cache
Temporal Quality
VBenchLong Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Motion Smooth. ↑
Aesthetic Quality ↑
LongLive 2.0
30 s
Full KV
4.86
1.00 ×
20.76
1.00 ×
3.41
58.75
80.44
66.89
97.79
97.35
98.87
55.35
SemanticKV
6.16
1.27 ×
6.23
3.33 ×
3.31
60.40
79.66
64.40
97.39
96.89
98.75
54.22
LongLive 2.0
60 s
Full KV
4.76
1.00 ×
20.76
1.00 ×
3.39
62.21
80.12
65.82
97.66
97.13
98.75
53.77
SemanticKV
6.12
1.29 ×
6.23
3.33 ×
3.39
61.15
78.84
62.72
97.20
96.78
98.62
51.88
Appendix
Table 5: Comparison of SemanticKV with Full KV on VBenchLong for 30- and 60-second video generation. Full KV retains the native cache without additional eviction and serves as the reference. Speedup is measured relative to Full KV. ↑ indicates higher is better and ↓ indicates lower is better. The comparison measures the quality–throughput trade-off relative to the uncompressed reference; no ranking among compression methods is reported.
Method
VisionReward ↑
Instruction Following ↑
Full KV
10.264
96.000
R-KV
7.506
98.000
Dummy Forcing
8.461
96.000
Forcing-KV
7.783
98.000
SemanticKV
7.964
98.000
Appendix
Table 6: VisionReward evaluation on LongLive at compression ratio 0.9. We use the 100 prompts from Causal Forcing and score all methods with the same evaluator runtime. Higher is better for both metrics.
Configuration
Semantic Score ↑
Object Class ↑
Multiple Objects ↑
Human Action ↑
Color ↑
Spatial Relation ↑
Scene ↑
Temporal Style ↑
Appearance Style ↑
Overall Cons. ↑
Uniform queries
76.94
90.44
76.02
94.60
90.26
72.12
44.36
25.05
20.61
27.00
Pooled constraints
76.78
91.00
75.90
93.80
89.40
73.05
43.71
25.04
20.52
26.98
SemanticKV
77.18
91.60
75.34
94.60
90.69
73.27
44.46
25.09
20.66
26.88
Appendix
Table 7: Component ablations on LongLive and 5-second VBench at compression ratio 0.9 with nq=32 . Panel A reports semantic fidelity; Panel B reports DiT-only efficiency and visual/temporal diagnostics. Semantic and Quality are the official VBench aggregate scores. VBench scores retain their reported percentage-scaled values. ↑ indicates higher is better and ↓ indicates lower is better. Bold marks the best displayed value among the three configurations.
nq
Efficiency
KV Cache
Temporal Quality
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Temporal Flicker. ↑
Motion Smooth. ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Aesthetic Quality ↑
16
9.93
1.51 ×
0.34
10.12 ×
3.54
60.00
99.02
98.57
81.54
67.35
94.00
93.68
62.45
32
10.00
1.52 ×
0.34
10.12 ×
3.54
61.39
99.02
98.58
81.68
67.36
94.07
93.70
62.30
64
9.78
1.49 ×
0.34
10.12 ×
3.55
58.33
99.02
98.57
81.45
67.27
93.95
93.81
62.34
Appendix
Table 8: Ablation of the semantic query limit nq on VBench. The compression ratio is fixed to 0.9 . VBench metrics retain their percentage-scaled values, except Chunk Discontinuity, which is the RAFT TopK-to-mean optical-flow ratio. ↑ indicates higher is better and ↓ indicates lower is better. The best displayed result in each column is highlighted in bold .
Figure 4: Paired semantic fidelity on matched prompt–seed tuples. (a) Tie-aware superiority rates. (b) Mean paired differences with prompt-clustered bootstrap 95% confidence intervals. Positive differences favor SemanticKV ; Full KV is the uncompressed reference.
Comparator
Object
Multiple objects
Human action
Color
Spatial relation
Scene
Macro
Full KV
+0.76
−0.99
+2.00
+0.32
+3.21
−0.28
+0.84
R-KV
+1.87
+0.85
+1.20
−0.11
+1.30
+0.90
+1.00
Dummy Forcing
+2.82
−0.35
+2.00
+0.48
+1.06
−0.87
+0.86
Forcing-KV
+3.69
+0.50
+0.80
+0.64
+1.27
+0.60
+1.25
Appendix
Table 9: Mean paired semantic-score difference ( SemanticKV minus comparator), in percentage points. The macro column averages the six dimensions without reweighting them.
Figure 5: Semantic fidelity as a function of planner-extracted prompt complexity. Values are macro-averaged paired differences between SemanticKV and each comparator across six semantic VBench dimensions. Shaded regions show prompt-clustered bootstrap 95% confidence intervals. The grouping uses the planner decomposition only and is fixed before inspecting the video scores.
Figure 6: Within-video drift on a fixed 96-prompt subset. (a) Change in CLIP image–prompt alignment relative to the first frame. (b) CLIP visual-feature similarity to the first frame. Curves show means and shaded prompt-bootstrap 95% confidence intervals over 96 videos per method. Full KV is the uncompressed reference.
Method
Alignment change (CLIP points)
Visual similarity to first frame
Full KV
−0.67[−0.95,−0.41]
0.942[0.934,0.950]
R-KV
−0.44[−0.84,−0.06]
0.852[0.836,0.867]
Dummy Forcing
−0.34[−0.70,+0.05]
0.858[0.845,0.870]
Forcing-KV
−0.61[−1.02,−0.22]
0.859[0.844,0.873]
SemanticKV
−0.57[−1.03,−0.12]
0.854[0.841,0.867]
Appendix
Table 10: End-of-video temporal analysis. Brackets give prompt-bootstrap 95% confidence intervals. A less negative alignment change and a higher visual similarity indicate less drift.
Figure 7: Human-action fidelity on LongLive. Prompt: “A person is dunking basketball” (seed 42). R-KV, Dummy Forcing, and Forcing-KV depict ball handling or dribbling and fail the VBench Human Action check. SemanticKV depicts the requested dunk and passes the check, matching the Full-KV reference. Rows compare methods and columns show uniformly sampled timestamps.
Figure 8: Object-class fidelity on LongLive. Prompt: “a snowboard” (seed 45). Full KV and all three compressed baselines generate a skateboard against a studio background and receive a per-video Object Class score of 0.00. SemanticKV generates a snowboard being ridden on snow and receives 1.00. The example illustrates that semantically guided retention can preserve the requested class even when a visually related concept is a strong failure mode.
Figure 9: Multi-object fidelity on LongLive. Prompt: “a tennis racket and a bottle” (seed 43). The compressed baselines replace the requested bottle with tennis balls, producing per-video Multiple Objects scores of 0.44, 0.12, and 0.00 for R-KV, Dummy Forcing, and Forcing-KV, respectively. SemanticKV retains both requested objects and scores 1.00; Full KV scores 0.81.
Figure 10: Compositional object fidelity on LongLive. Prompt: “an airplane and a train” (seed 44). The compressed baselines produce ambiguous airplane–train hybrids and obtain per-video Multiple Objects scores of 0.44, 0.44, and 0.38 for R-KV, Dummy Forcing, and Forcing-KV, respectively. SemanticKV keeps an airplane and a rail vehicle visually distinguishable and scores 1.00; Full KV scores 0.00.
Figure 11: Scene fidelity on LongLive. Prompt: “palace” (seed 45). Full KV and the three compressed baselines converge to a generic neoclassical institutional facade and receive per-video Scene scores of 0.00. SemanticKV produces palatial architecture across the sequence and scores 0.75.
Figure 12: Activity-rich scene fidelity on LongLive. Prompt: “construction site” (seed 43). SemanticKV preserves discriminative construction cues, including excavation, a worker, tools, and machinery, and obtains a per-video Scene score of 0.81. The corresponding scores are 0.00 for Full KV, 0.00 for R-KV, 0.06 for Dummy Forcing, and 0.00 for Forcing-KV.
Autoregressive video models rely on historical key-value (KV) caches to preserve scene consistency over extended rollouts, making KV storage and repeated attention over history a growing memory and computational bottleneck. Aggressive compression can degrade generation quality, while reconstructing full-dimensional KV leaves attention computation largely unchanged. We introduce NanoKV, a frozen-backbone framework for generation-aware high-ratio KV compression. NanoKV exploits redundancy across tokens and attention heads by storing shared content in cluster centers and encoding joint head-channel residuals into low-bit coefficients. Crucially, NanoKV uses different fitting objectives for Keys and Values according to their distinct roles in attention: Key decoders are optimized to preserve downstream block outputs, whereas Value encoders and decoders are fitted for faithful reconstruction through the deployed quantization path. Building on the same compressed representation, the optional NanoKV-Fast variant reuses the stored coefficients for mixed-dimensional attention, reducing computation without modifying the video backbone. Across three autoregressive video backbones, NanoKV consistently improves the quality-storage trade-off. On HY-WorldPlay, NanoKV achieves 26.09× effective KV compression, nearly twice that of KVTC, while improving PSNR by 1.0 dB. NanoKV-Fast retains 20.53× effective compression and achieves an approximately 1.15× end-to-end speedup over KVTC.
Autoregressive (AR) video diffusion models enable streaming generation of long videos but suffer from significant memory overhead due to the rapidly growing KV cache. Existing KV cache quantization methods are primarily designed for LLMs and transfer poorly to video models, where activations exhibit different statistics and strong spatiotemporal redundancy. We observe that the video KV cache is largely predictable over time, with a clear asymmetry. Keys are highly predictable from their temporal history, while values depend primarily on the current key and need a short history of preceding values. Based on these observations, we propose \method, a training-free predictive KV cache quantization framework for AR video generation. \method predicts keys and values according to their distinct dependencies and stores only the residuals at low precision. Both predictors are lightweight, calibrated once offline by linear regression on a small set of BF16 samples, and remain fixed across prompts and generation runs. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B show that \method consistently achieves higher generation quality than existing KV cache quantization methods at comparable or higher compression ratios, reaching up to 6.71× KV cache compression with the best overall quality--efficiency tradeoff. Our code will be released on GitHub.
Jiahui Hu, Weile Mo, Peiliang Cai +5
Shanghai Jiao Tong University · South China University of Technology · Alibaba Group
Autoregressive video diffusion models maintain a growing attention key-value (KV) cache, making memory a key bottleneck for long video generation. Recently, clustering-based quantization has emerged as an effective method to compress video KV caches by grouping similar key and value vectors then quantizing residual errors. However, when later frames attend to the compressed history, small reconstruction errors can compound as the video grows. Beyond the within-layer similarity exploited by clustering, we find strong dependencies across cache layers in video models. This motivates Coral-KV - Calibrated Offline, Residual-coded, and Adapted Locally - a KV-cache compression framework that uses cache states in each layer to predict keys and values in subsequent layers, then quantizes the remaining prediction residuals. We construct compact block-diagonal-plus-low-rank (block-DPLR) predictors: block-diagonal components model channel relationships within each attention head, while a global low-rank term models dependencies between heads. Coral-KV exposes a flexible fidelity-memory trade-off: it can reuse its predictors across videos or refit lightweight block-diagonal predictors to the current video for higher fidelity. Across extensive evaluations on both HY-WorldPlay and MAGI-1, our reusable configuration improves PSNR over QVG-4 by 0.32 dB and 4.95 dB, respectively, while using 48% and 37% less cache memory. Against QVG-Pro, our video-adapted configuration improves MAGI-1 PSNR by 3.77 dB while using 33.7% less cache memory. Coral-KV also maintains higher reconstruction fidelity over extended generation and reduces temporal discontinuities while retaining comparable generation latency.
Albert Ge, Akash Haridas, Utkarsh Saxena +5
Advanced Micro Devices Inc. (AMD) · University of Wisconsin-Madison