Organizations: Georgia Institute of Technology, USA · Independent Researcher, USA · Annapurna Labs, Amazon, USA · Mercuria Energy Americas, USA · Google, USA · Intel, USA
KV cache compression is critical for large reasoning models, where most of the cost comes from long generated reasoning traces rather than the input prompt. These traces often contain redundant and low-utility tokens due to looping, self-verification, and corrections. However, such tokens may still receive high attention scores, making it difficult to remove them without harming the reasoning process. We propose RoCoKV, a KV cache compression method designed for large reasoning models. RoCoKV introduces two key components. First, uncertainty-guided adaptive pruning adjusts the compression strength based on the model's confidence, allocating more memory during uncertain phases and applying stronger pruning when the model is confident. Second, redundancy-aware token selection reduces the impact of tokens that are highly similar to recent reasoning content, encouraging the model to retain more informative and diverse tokens. Beyond the algorithm, we co-design RoCoKV with the serving stack: we develop fused three-phase CUDA decode kernels and integrate them into vLLM as a drop-in plugin that preserves its paged KV layout and CUDA graph capture, turning algorithmic savings into real end-to-end speedups. Extensive experiments across three reasoning benchmarks and two reasoning models demonstrate that RoCoKV consistently improves reasoning accuracy under constrained KV cache budgets, achieving higher accuracy compared to state-of-the-art KV cache compression methods. For DeepSeek-R1-Distill-Llama-8B in vLLM, RoCoKV achieves a 1.27X speedup and 38% higher throughput on AIME25, the most challenging evaluated reasoning benchmark, and a 1.43X speedup and 45% higher throughput on the long synthetic benchmark, compared with Full-KV.
Figures & tables
Figure 1: Observation 1 - Attention approximation errors are more harmful at uncertain decoding steps.
Figure 2: Observation 2 - Long generations are more likely to produce incorrect answers.
Figure 3: Overview of the RoCoKV sparse attention pipeline.
Figure 4: Comparing the accuracy of RoCoKV with other baselines and Full-KV on GSM8K (a-b), MATH500 (c-d), and AIME25 (e-f).
Method
Latency (s) ↓
Throughput (tok/s) ↑
Full-KV (vLLM-AIME25)
196.2 (1.00 × )
247.3 (1.00 × )
Full-KV (vLLM-synthetic)
864.9 (1.00 × )
115.0 (1.00 × )
R-KV Cai et al. (2025)
208.7 (0.94 × )
276.9 (1.12 × )
RoCoKV (vLLM-AIME25)
154.5 (1.27 × )
341.3 (1.38 × )
RoCoKV (vLLM-synthetic)
604.8 (1.43 × )
166.7 (1.45 × )
Table 1: End-to-end efficiency on A40 over R1-Llama-8B. Latency and throughput are normalized to Full-KV.
Model / Dataset
RoCoKV
Unc.
Redun.
Reverse
R1-Llama-8B/ AIME25
14.58
10.00
0.00
1.25
R1-Qwen-7B/ AIME25
18.75
11.25
0.00
0.00
R1-Llama-8B/ GSM8K
78.54
72.02
59.36
51.40
R1-Qwen-7B/ GSM8K
72.86
68.92
33.66
31.01
R1-Llama-8B/ MATH500
64.60
57.40
4.60
3.40
R1-Qwen-7B/ MATH500
80.00
78.40
9.20
5.80
Table 2: Component ablation of RoCoKV. Pass@1 accuracy (%) at token budget 128. Redun. and Unc. isolate redundancy-aware scoring and uncertainty-guided adaptive pruning, respectively. Reverse inverts the pruning selection.
Model / Dataset
RoCoKV
Lower Threshold ( τlow )
Upper Threshold ( τhigh )
0
0.05
0.1
0.2
0.5
0.6
0.7
0.8
R1-Llama-8B/ AIME25
14.58
6.67
9.58
9.58
7.92
14.58
10.42
6.67
12.08
R1-Qwen-7B/ AIME25
18.75
14.17
12.50
10.42
9.58
18.75
14.17
10.00
10.42
R1-Llama-8B/ GSM8K
78.54
71.57
72.18
69.98
72.86
70.74
71.95
71.57
78.54
R1-Qwen-7B/ GSM8K
72.86
67.85
69.90
70.43
69.52
72.86
70.43
66.49
60.12
R1-Llama-8B/ MATH500
64.60
62.60
59.60
59.20
56.20
64.60
59.20
55.20
64.20
Table 3: Ablation study on the lower ( τlow ) and upper ( τhigh ) uncertainty thresholds. Results are pass@1 accuracy (%) at a token budget of 128.
Model / Dataset
RoCoKV
Redundancy Penalty ( λ )
Redundancy Window ( wt )
0.01
0.2
0.5
4
10
15
R1-Llama-8B/ AIME25
14.58
9.17
9.17
5.42
7.08
10.00
10.00
R1-Qwen-7B/ AIME25
18.75
10.42
11.67
12.08
10.83
12.50
12.08
R1-Llama-8B/ GSM8K
78.54
70.43
71.87
71.65
71.80
72.33
71.72
R1-Qwen-7B/ GSM8K
72.86
66.34
72.71
76.12
68.54
68.01
68.54
R1-Llama-8B/ MATH500
64.60
60.60
59.40
58.40
59.20
59.20
59.20
Table 4: Ablation study on redundancy penalty coefficient ( λ ) and redundancy window length w . Results are pass@1 accuracy (%) at a token budget of 128.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Level
Accuracy
AIME25
Token-level
6.67%
AIME25
Thought-level ( W=4,K=8 )
3.33%
GSM8K
Token-level
80.59%
GSM8K
Thought-level ( W=4,K=8 )
78.09%
Appendix
Table 5: Accuracy of redundancy removal only (without uncertainty-guided adaptive pruning) using R1-Qwen-7B under a 256-token KV cache budget for GSM8K and AIME25 benchmarks.
K=4
K=8
K=16
W=2
77.63%
78.70%
78.01%
W=4
77.63%
78.70%
78.01%
W=8
77.63%
78.70%
78.01%
Appendix
Table 6: Thought-level hyperparameter sweep on GSM8K over R1-Qwen-7B, token KV cache budget 256. The accuracy of the token-level baseline is 80.59%.
Model
Dataset (τlow,τhigh)
Method
Accuracy
R1-Llama-8B
AIME25 ( 0.0 – 0.7 )
Token-level
26.67%
Hybrid-level ( W=4 )
6.67%
Hybrid-level ( W=8 )
13.33%
Hybrid-level ( W=16 )
6.67%
Hybrid-level ( W=32 )
10.00%
R1-Qwen-7B
AIME25 ( 0.0 – 0.6 )
Token-level
23.33%
Appendix
Table 7: Hybrid-level pruning, token KV cache budget 256. Token-level uses λ=0.1 , redundancy window 8; thought-level uses K=8 with the indicated window W . Uncertainty thresholds (τlow,τhigh) are identical across levels within each block.
Dataset
Model
Token Budget
Accuracy
GSM8K
R1-Llama-8B
Full-KV
79.23%
GSM8K
R1-Llama-8B
512
79.15%
GSM8K
R1-Qwen-7B
Full-KV
85.22%
GSM8K
R1-Qwen-7B
512
85.75%
Appendix
Table 8: Accuracy of different models under a 512-token KV cache budget for GSM8K benchmarks.
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) their threshold-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) they typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM.
Recent reasoning models such as OpenAI-o1 and DeepSeek-R1 have shown strong performance on complex tasks including mathematical reasoning and code generation. However, this performance gain comes with substantially longer output sequences, leading to significantly increased deployment costs. In particular, long outputs require large KV caches, resulting in high memory consumption and severe bandwidth pressure during attention computation. Most existing KV cache optimization methods are designed for long-input, short-output scenarios and are ineffective for the long-output setting of reasoning models. Moreover, importance estimation in prior work is computationally expensive and becomes prohibitive when continuous re-evaluation is required during long generation. To address these challenges, we propose LongFlow, a KV cache compression method with an efficient importance estimation metric derived from an intermediate result of attention computation using only the current query. This design introduces negligible computational overhead and requires no auxiliary storage. We further develop a custom kernel that fuses FlashAttention, importance estimation, and token eviction into a single optimized operator, improving system-level efficiency. Experiments show that LongFlow achieves up to an 11.8 times throughput improvement with 80% KV cache compression with minimal impact on model accuracy.
Yi Su, Zhenxu Tian, Dan Qiao +3
School of Computer Science and Technology, Soochow University, China · ByteDance