Organizations: Georgia Institute of Technology, USA · Independent Researcher, USA · Annapurna Labs, Amazon, USA · Mercuria Energy Americas, USA · Google, USA · Intel, USA
KV cache compression is critical for large reasoning models, where most of the cost comes from long generated reasoning traces rather than the input prompt. These traces often contain redundant and low-utility tokens due to looping, self-verification, and corrections. However, such tokens may still receive high attention scores, making it difficult to remove them without harming the reasoning process. We propose RoCoKV, a KV cache compression method designed for large reasoning models. RoCoKV introduces two key components. First, uncertainty-guided adaptive pruning adjusts the compression strength based on the model's confidence, allocating more memory during uncertain phases and applying stronger pruning when the model is confident. Second, redundancy-aware token selection reduces the impact of tokens that are highly similar to recent reasoning content, encouraging the model to retain more informative and diverse tokens. Beyond the algorithm, we co-design RoCoKV with the serving stack: we develop fused three-phase CUDA decode kernels and integrate them into vLLM as a drop-in plugin that preserves its paged KV layout and CUDA graph capture, turning algorithmic savings into real end-to-end speedups. Extensive experiments across three reasoning benchmarks and two reasoning models demonstrate that RoCoKV consistently improves reasoning accuracy under constrained KV cache budgets, achieving higher accuracy compared to state-of-the-art KV cache compression methods. For DeepSeek-R1-Distill-Llama-8B in vLLM, RoCoKV achieves a 1.27X speedup and 38% higher throughput on AIME25, the most challenging evaluated reasoning benchmark, and a 1.43X speedup and 45% higher throughput on the long synthetic benchmark, compared with Full-KV.
Figures & tables
Figure 1: Observation 1 - Attention approximation errors are more harmful at uncertain decoding steps.
Figure 2: Observation 2 - Long generations are more likely to produce incorrect answers.
Figure 3: Overview of the RoCoKV sparse attention pipeline.
Figure 4: Comparing the accuracy of RoCoKV with other baselines and Full-KV on GSM8K (a-b), MATH500 (c-d), and AIME25 (e-f).
Method
Latency (s) ↓
Throughput (tok/s) ↑
Full-KV (vLLM-AIME25)
196.2 (1.00 × )
247.3 (1.00 × )
Full-KV (vLLM-synthetic)
864.9 (1.00 × )
115.0 (1.00 × )
R-KV Cai et al. (2025)
208.7 (0.94 × )
276.9 (1.12 × )
RoCoKV (vLLM-AIME25)
154.5 (1.27 × )
341.3 (1.38 × )
RoCoKV (vLLM-synthetic)
604.8 (1.43 × )
166.7 (1.45 × )
Table 1: End-to-end efficiency on A40 over R1-Llama-8B. Latency and throughput are normalized to Full-KV.
Model / Dataset
RoCoKV
Unc.
Redun.
Reverse
R1-Llama-8B/ AIME25
14.58
10.00
0.00
1.25
R1-Qwen-7B/ AIME25
18.75
11.25
0.00
0.00
R1-Llama-8B/ GSM8K
78.54
72.02
59.36
51.40
R1-Qwen-7B/ GSM8K
72.86
68.92
33.66
31.01
R1-Llama-8B/ MATH500
64.60
57.40
4.60
3.40
R1-Qwen-7B/ MATH500
80.00
78.40
9.20
5.80
Table 2: Component ablation of RoCoKV. Pass@1 accuracy (%) at token budget 128. Redun. and Unc. isolate redundancy-aware scoring and uncertainty-guided adaptive pruning, respectively. Reverse inverts the pruning selection.
Model / Dataset
RoCoKV
Lower Threshold ( τlow )
Upper Threshold ( τhigh )
0
0.05
0.1
0.2
0.5
0.6
0.7
0.8
R1-Llama-8B/ AIME25
14.58
6.67
9.58
9.58
7.92
14.58
10.42
6.67
12.08
R1-Qwen-7B/ AIME25
18.75
14.17
12.50
10.42
9.58
18.75
14.17
10.00
10.42
R1-Llama-8B/ GSM8K
78.54
71.57
72.18
69.98
72.86
70.74
71.95
71.57
78.54
R1-Qwen-7B/ GSM8K
72.86
67.85
69.90
70.43
69.52
72.86
70.43
66.49
60.12
R1-Llama-8B/ MATH500
64.60
62.60
59.60
59.20
56.20
64.60
59.20
55.20
64.20
Table 3: Ablation study on the lower ( τlow ) and upper ( τhigh ) uncertainty thresholds. Results are pass@1 accuracy (%) at a token budget of 128.
Model / Dataset
RoCoKV
Redundancy Penalty ( λ )
Redundancy Window ( wt )
0.01
0.2
0.5
4
10
15
R1-Llama-8B/ AIME25
14.58
9.17
9.17
5.42
7.08
10.00
10.00
R1-Qwen-7B/ AIME25
18.75
10.42
11.67
12.08
10.83
12.50
12.08
R1-Llama-8B/ GSM8K
78.54
70.43
71.87
71.65
71.80
72.33
71.72
R1-Qwen-7B/ GSM8K
72.86
66.34
72.71
76.12
68.54
68.01
68.54
R1-Llama-8B/ MATH500
64.60
60.60
59.40
58.40
59.20
59.20
59.20
Table 4: Ablation study on redundancy penalty coefficient ( λ ) and redundancy window length w . Results are pass@1 accuracy (%) at a token budget of 128.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Level
Accuracy
AIME25
Token-level
6.67%
AIME25
Thought-level ( W=4,K=8 )
3.33%
GSM8K
Token-level
80.59%
GSM8K
Thought-level ( W=4,K=8 )
78.09%
Appendix
Table 5: Accuracy of redundancy removal only (without uncertainty-guided adaptive pruning) using R1-Qwen-7B under a 256-token KV cache budget for GSM8K and AIME25 benchmarks.
K=4
K=8
K=16
W=2
77.63%
78.70%
78.01%
W=4
77.63%
78.70%
78.01%
W=8
77.63%
78.70%
78.01%
Appendix
Table 6: Thought-level hyperparameter sweep on GSM8K over R1-Qwen-7B, token KV cache budget 256. The accuracy of the token-level baseline is 80.59%.
Model
Dataset (τlow,τhigh)
Method
Accuracy
R1-Llama-8B
AIME25 ( 0.0 – 0.7 )
Token-level
26.67%
Hybrid-level ( W=4 )
6.67%
Hybrid-level ( W=8 )
13.33%
Hybrid-level ( W=16 )
6.67%
Hybrid-level ( W=32 )
10.00%
R1-Qwen-7B
AIME25 ( 0.0 – 0.6 )
Token-level
23.33%
Appendix
Table 7: Hybrid-level pruning, token KV cache budget 256. Token-level uses λ=0.1 , redundancy window 8; thought-level uses K=8 with the indicated window W . Uncertainty thresholds (τlow,τhigh) are identical across levels within each block.
Dataset
Model
Token Budget
Accuracy
GSM8K
R1-Llama-8B
Full-KV
79.23%
GSM8K
R1-Llama-8B
512
79.15%
GSM8K
R1-Qwen-7B
Full-KV
85.22%
GSM8K
R1-Qwen-7B
512
85.75%
Appendix
Table 8: Accuracy of different models under a 512-token KV cache budget for GSM8K benchmarks.