Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
Agentic autoregressive video generation combines semantic planning, sequential generation, and visual verification for controllable synthesis. Growing visual histories enlarge the key-value (KV) cache, increasing memory use and attention costs across candidate rollouts. Existing compression methods primarily optimize individual generators without the planner's semantic constraints, risking low priority for states supporting requested attributes or actions. We introduce SemanticKV, a training-free method for semantically aligned KV cache compression. To our knowledge, SemanticKV is the first KV cache compression method for agentic autoregressive video generation that explicitly uses planner-extracted semantic constraints to guide retention. Text cross-attention identifies constraint-related video positions, whose self-attention queries are collected into separate groups. Attention weights are averaged within each group and maximized across groups, giving each cached entry a high score when strongly used by any represented constraint. A single top-k selection retains high-scoring entries under the memory budget, leaving their keys, values, and positions unchanged. For fixed query groups and equal entry costs, this selection minimizes an additive upper bound on worst-group discarded attention. The method operates independently within each candidate without retraining or additional video generations. We evaluate four AR video backbones on three benchmarks for 5-, 30-, and 60-second generation. On Causal Forcing at a compression ratio of 0.7, SemanticKV obtains a VBench Total Score of 84.10, compared with 84.05 for the strongest compressed baseline and 83.99 for Full KV. It achieves a 1.22x speedup over Full KV and reduces KV-cache memory from 6.04 GB to 1.81 GB, a 3.35x reduction.
Figures & tables
Figure 1: Agentic AR video generation. Semantic constraints R and prompt-specific exclusions N guide prompt refinement. The original-prompt and planned candidates from the same generator undergo randomized, anonymous visual verification, followed by conservative selection using Eq. 7 .
Figure 2: Overview of SemanticKV . Semantic constraints R are matched to prompt tokens. Text cross-attention identifies related video positions, from which we collect the generator’s self-attention queries. Their attention to each KV entry is averaged within groups and maximized across groups. A single top- k selection retains high-scoring entries within the byte budget, together with states required by the backbone. Retained keys, values, and positions remain unchanged.
Backbone
Method
Efficiency
KV Cache
Temporal Quality
VBench: Overall and Semantic Fidelity
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Disc. ↓
Dynamic ↑
Smooth. ↑
Total Score ↑
Object Class ↑
Multiple Objects ↑
Human Action ↑
Color ↑
Spatial Relation ↑
Scene ↑
Self Forcing
Full KV
6.95
1.00 ×
6.04
1.00 ×
2.25
91.67
97.77
82.96
89.83
75.55
92.00
90.54
71.67
48.05
R-KV
7.36
1.06 ×
1.81
3.35 ×
2.23
92.78
97.77
82.90
88.29
74.15
93.00
91.40
71.66
47.53
Dummy Forcing
8.16
1.17 ×
1.81
3.35 ×
2.23
91.39
97.81
82.87
89.26
77.07
92.00
91.12
71.01
47.43
Forcing-KV
7.92
1.14 ×
1.81
3.35 ×
2.62
92.78
97.55
82.69
88.21
75.88
92.60
90.64
71.44
47.72
SemanticKV
7.95
1.14 ×
1.81
3.35 ×
2.26
93.33
97.80
83.03
89.43
75.47
92.00
91.30
72.46
48.65
Table 1: Five-second VBench at compression ratio 0.7 : Total Score, temporal quality, and six shared semantic diagnostics. Table 3 gives all 16 dimensions. Full KV is the uncompressed reference; efficiency is DiT-only. ↑ / ↓ : higher/lower is better. Bold : best compressed result for Total Score.
Figure 3: Five-second VBench across compression ratios 0.5/0.7/0.9 , with the same five methods and six dimensions; higher is better. Table 4 gives exact scores and DiT-only efficiency.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Component and role
Example
Original prompt q0 Before refinement.
“ park , one girl one boy , red coat on girl blue coat on boy , yellow ball , girl gives it to boy then boy lifts it over his head , fixed camera ”
Semantic constraints R What to preserve.
r1 : One girl in a red coat. r2 : One boy in a blue coat. r3 : A yellow ball. r4 : The girl gives the ball to the boy. r5 : The boy then lifts the ball over his head. r6 : A park. r7 : A fixed camera.
Prompt-specific exclusions N What to avoid.
Swapped coat colors; reversed giver and receiver; reversed action order; camera movement; additional people, objects, or actions.
Refined prompt q1 After refinement.
“ In a park , one girl in a red coat gives a yellow ball to one boy in a blue coat . The boy then lifts the same ball over his head . The camera remains fixed .”
Appendix
Table 2: An illustrative example of conservative semantic planning. Matching colors trace the requested content from q0 through R to q1 : subjects and attributes , actions and their order , and scene and camera . Prompt-specific exclusions identify changes to avoid.
Backbone
Length
Method
Efficiency
KV Cache
Temporal Quality
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Temporal Flicker. ↑
Motion Smooth. ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Aesthetic Quality ↑
Self Forcing
5 s
Full KV
6.95
1.00 ×
6.04
1.00 ×
2.25
91.67
98.54
97.77
82.96
67.50
93.84
95.24
61.07
R-KV
7.36
1.06 ×
1.81
3.35 ×
2.23
92.78
98.45
97.77
82.90
67.52
93.54
95.01
61.26
Dummy Forcing
8.16
1.17 ×
1.81
3.35 ×
2.23
91.39
98.42
97.81
82.87
67.49
93.49
95.22
61.09
Forcing-KV
7.92
1.14 ×
1.81
3.35 ×
2.62
92.78
98.41
97.55
82.69
67.06
93.28
95.14
60.95
SemanticKV
7.95
1.14 ×
1.81
3.35 ×
2.26
93.33
98.50
97.80
83.03
67.56
93.49
95.10
60.99
Appendix
Table 3: Efficiency and generation quality on VBench for 5-second video generation at compression ratio 0.7 . Full KV retains the native cache without additional eviction and serves as the reference. Speedup is measured relative to Full KV. VBench metrics retain their reported percentage-scaled values. ↑ indicates higher is better and ↓ indicates lower is better. Bold marks the best result among compressed methods for each backbone.
Compression Ratio
Length
Method
Efficiency
KV Cache
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Temporal Flicker. ↑
Subject Cons. ↑
Background Cons. ↑
Object Class ↑
Human Action ↑
Color ↑
Full KV
6.57
1.00 ×
3.45
1.00 ×
99.26
96.06
96.68
90.84
92.60
90.34
0.5
5 s
R-KV
7.53
1.15 ×
1.72
2.01 ×
99.32
96.07
96.61
91.01
93.00
90.84
Dummy Forcing
7.32
1.11 ×
1.72
2.01 ×
98.81
95.61
95.95
90.09
93.60
88.95
Forcing-KV
7.67
1.17 ×
1.72
2.01 ×
99.29
95.92
96.63
90.84
94.00
89.73
SemanticKV
8.00
1.22 ×
1.72
2.01 ×
99.29
96.04
96.52
91.93
92.80
89.24
Appendix
Table 4: VBench results across compression ratios for 5-second generation, reporting the six dimensions in Figure 3 and DiT-only efficiency. Full KV is the shared uncompressed reference within this run group. Speedup is measured relative to Full KV. VBench metrics retain their reported percentage-scaled values. ↑ indicates higher is better. Bold marks the best result among compressed methods at each compression ratio.
Backbone
Length
Method
Efficiency
KV Cache
Temporal Quality
VBenchLong Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Motion Smooth. ↑
Aesthetic Quality ↑
LongLive 2.0
30 s
Full KV
4.86
1.00 ×
20.76
1.00 ×
3.41
58.75
80.44
66.89
97.79
97.35
98.87
55.35
SemanticKV
6.16
1.27 ×
6.23
3.33 ×
3.31
60.40
79.66
64.40
97.39
96.89
98.75
54.22
LongLive 2.0
60 s
Full KV
4.76
1.00 ×
20.76
1.00 ×
3.39
62.21
80.12
65.82
97.66
97.13
98.75
53.77
SemanticKV
6.12
1.29 ×
6.23
3.33 ×
3.39
61.15
78.84
62.72
97.20
96.78
98.62
51.88
Appendix
Table 5: Comparison of SemanticKV with Full KV on VBenchLong for 30- and 60-second video generation. Full KV retains the native cache without additional eviction and serves as the reference. Speedup is measured relative to Full KV. ↑ indicates higher is better and ↓ indicates lower is better. The comparison measures the quality–throughput trade-off relative to the uncompressed reference; no ranking among compression methods is reported.
Method
VisionReward ↑
Instruction Following ↑
Full KV
10.264
96.000
R-KV
7.506
98.000
Dummy Forcing
8.461
96.000
Forcing-KV
7.783
98.000
SemanticKV
7.964
98.000
Appendix
Table 6: VisionReward evaluation on LongLive at compression ratio 0.9. We use the 100 prompts from Causal Forcing and score all methods with the same evaluator runtime. Higher is better for both metrics.
Configuration
Semantic Score ↑
Object Class ↑
Multiple Objects ↑
Human Action ↑
Color ↑
Spatial Relation ↑
Scene ↑
Temporal Style ↑
Appearance Style ↑
Overall Cons. ↑
Uniform queries
76.94
90.44
76.02
94.60
90.26
72.12
44.36
25.05
20.61
27.00
Pooled constraints
76.78
91.00
75.90
93.80
89.40
73.05
43.71
25.04
20.52
26.98
SemanticKV
77.18
91.60
75.34
94.60
90.69
73.27
44.46
25.09
20.66
26.88
Appendix
Table 7: Component ablations on LongLive and 5-second VBench at compression ratio 0.9 with nq=32 . Panel A reports semantic fidelity; Panel B reports DiT-only efficiency and visual/temporal diagnostics. Semantic and Quality are the official VBench aggregate scores. VBench scores retain their reported percentage-scaled values. ↑ indicates higher is better and ↓ indicates lower is better. Bold marks the best displayed value among the three configurations.
nq
Efficiency
KV Cache
Temporal Quality
VBench Quality
FPS ↑
Speedup ↑
Memory (GB) ↓
Reduction ( × ) ↑
Chunk Disc. ↓
Dynamic Degree ↑
Temporal Flicker. ↑
Motion Smooth. ↑
Total Score ↑
Imaging Quality ↑
Subject Cons. ↑
Background Cons. ↑
Aesthetic Quality ↑
16
9.93
1.51 ×
0.34
10.12 ×
3.54
60.00
99.02
98.57
81.54
67.35
94.00
93.68
62.45
32
10.00
1.52 ×
0.34
10.12 ×
3.54
61.39
99.02
98.58
81.68
67.36
94.07
93.70
62.30
64
9.78
1.49 ×
0.34
10.12 ×
3.55
58.33
99.02
98.57
81.45
67.27
93.95
93.81
62.34
Appendix
Table 8: Ablation of the semantic query limit nq on VBench. The compression ratio is fixed to 0.9 . VBench metrics retain their percentage-scaled values, except Chunk Discontinuity, which is the RAFT TopK-to-mean optical-flow ratio. ↑ indicates higher is better and ↓ indicates lower is better. The best displayed result in each column is highlighted in bold .
Figure 4: Paired semantic fidelity on matched prompt–seed tuples. (a) Tie-aware superiority rates. (b) Mean paired differences with prompt-clustered bootstrap 95% confidence intervals. Positive differences favor SemanticKV ; Full KV is the uncompressed reference.
Comparator
Object
Multiple objects
Human action
Color
Spatial relation
Scene
Macro
Full KV
+0.76
−0.99
+2.00
+0.32
+3.21
−0.28
+0.84
R-KV
+1.87
+0.85
+1.20
−0.11
+1.30
+0.90
+1.00
Dummy Forcing
+2.82
−0.35
+2.00
+0.48
+1.06
−0.87
+0.86
Forcing-KV
+3.69
+0.50
+0.80
+0.64
+1.27
+0.60
+1.25
Appendix
Table 9: Mean paired semantic-score difference ( SemanticKV minus comparator), in percentage points. The macro column averages the six dimensions without reweighting them.
Figure 5: Semantic fidelity as a function of planner-extracted prompt complexity. Values are macro-averaged paired differences between SemanticKV and each comparator across six semantic VBench dimensions. Shaded regions show prompt-clustered bootstrap 95% confidence intervals. The grouping uses the planner decomposition only and is fixed before inspecting the video scores.
Figure 6: Within-video drift on a fixed 96-prompt subset. (a) Change in CLIP image–prompt alignment relative to the first frame. (b) CLIP visual-feature similarity to the first frame. Curves show means and shaded prompt-bootstrap 95% confidence intervals over 96 videos per method. Full KV is the uncompressed reference.
Method
Alignment change (CLIP points)
Visual similarity to first frame
Full KV
−0.67[−0.95,−0.41]
0.942[0.934,0.950]
R-KV
−0.44[−0.84,−0.06]
0.852[0.836,0.867]
Dummy Forcing
−0.34[−0.70,+0.05]
0.858[0.845,0.870]
Forcing-KV
−0.61[−1.02,−0.22]
0.859[0.844,0.873]
SemanticKV
−0.57[−1.03,−0.12]
0.854[0.841,0.867]
Appendix
Table 10: End-of-video temporal analysis. Brackets give prompt-bootstrap 95% confidence intervals. A less negative alignment change and a higher visual similarity indicate less drift.
Figure 7: Human-action fidelity on LongLive. Prompt: “A person is dunking basketball” (seed 42). R-KV, Dummy Forcing, and Forcing-KV depict ball handling or dribbling and fail the VBench Human Action check. SemanticKV depicts the requested dunk and passes the check, matching the Full-KV reference. Rows compare methods and columns show uniformly sampled timestamps.
Figure 8: Object-class fidelity on LongLive. Prompt: “a snowboard” (seed 45). Full KV and all three compressed baselines generate a skateboard against a studio background and receive a per-video Object Class score of 0.00. SemanticKV generates a snowboard being ridden on snow and receives 1.00. The example illustrates that semantically guided retention can preserve the requested class even when a visually related concept is a strong failure mode.
Figure 9: Multi-object fidelity on LongLive. Prompt: “a tennis racket and a bottle” (seed 43). The compressed baselines replace the requested bottle with tennis balls, producing per-video Multiple Objects scores of 0.44, 0.12, and 0.00 for R-KV, Dummy Forcing, and Forcing-KV, respectively. SemanticKV retains both requested objects and scores 1.00; Full KV scores 0.81.
Figure 10: Compositional object fidelity on LongLive. Prompt: “an airplane and a train” (seed 44). The compressed baselines produce ambiguous airplane–train hybrids and obtain per-video Multiple Objects scores of 0.44, 0.44, and 0.38 for R-KV, Dummy Forcing, and Forcing-KV, respectively. SemanticKV keeps an airplane and a rail vehicle visually distinguishable and scores 1.00; Full KV scores 0.00.
Figure 11: Scene fidelity on LongLive. Prompt: “palace” (seed 45). Full KV and the three compressed baselines converge to a generic neoclassical institutional facade and receive per-video Scene scores of 0.00. SemanticKV produces palatial architecture across the sequence and scores 0.75.
Figure 12: Activity-rich scene fidelity on LongLive. Prompt: “construction site” (seed 43). SemanticKV preserves discriminative construction cues, including excavation, a worker, tools, and machinery, and obtains a per-video Scene score of 0.81. The corresponding scores are 0.00 for Full KV, 0.00 for R-KV, 0.06 for Dummy Forcing, and 0.00 for Forcing-KV.