Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
Figures & tables
Figure 1: (a) Attention weight distribution at layer 18, head 16 and (b) attention distance between query and key (AIME24 sample-0 on R1-Distill-Qwen-7B).
Figure 2: Output text tokens with Top- K attention weights for each query at layer 18, head 16 (AIME24 sample-0 on R1-Distill-Qwen-7B). The full text is provided in Appendix E .
Figure 3: Occurrence distribution of global queries across layers and heads (AIME24 sample-0 on R1-Distill-Qwen-7B). Additional details are provided in Appendix B .
Figure 4: (a) Cosine similarity between queries within a specific decoding-step interval at layer 18, head 16. (b) PCA of queries with low average cosine similarity at layer 18, head 16.
Figure 5: Illustration of KV cache compression methods for LRMs during long decoding. (a) Prior methods (e.g., RPC) compute attention-based scores for KV cache eviction relying solely on recent queries. (b) BeaconKV (ours) computes the attention-based scores using beacon queries and recent queries. (c) BeaconKV performs top- K selection via max aggregation of attention-based scores.
Figure 6: (a) Maximum cosine similarity to future decoding step queries and (b) AIME24 accuracy on R1-Distill-Qwen-7B under different past query set selections.
Figure 7: Peak GPU memory and accuracy degradation on R1-Distill-Qwen-7B under different observation query selection methods. We compare Continual FPS against K-Means Centroids, Centroid-Nearest Data Points, and Naive FPS.
Figure 8: Accuracy comparison of BeaconKV against RPC, SnapKV, and R-KV. We evaluate the accuracy on R1-Distill-Qwen-7B, R1-Distill-Llama-8B, Qwen3-4B and Qwen3-14B across four reasoning tasks: AIME24, MATH-500, GPQA-Diamond, and LiveCodeBench.
Method
(nrecent,nbeacon)
Accuracy (%) ↑
Latency (s) ↓
BeaconKV
(1, 31)
63.5
10871.3
(4, 28)
66.5
5624.5
(8, 24)
64.8
4762.1
(16, 16)
64.6
4355.7
(24, 8)
62.3
4194.5
(28, 4)
58.1
4179.5
Table 1: Ablation study on the number of beacon queries ( nbeacon ) and recent queries ( nrecent ) on Qwen3-4B for AIME24, with the maximum KV cache budget set to 2048. We report both accuracy and latency to identify the optimal configuration. The setting (nrecent=16,nbeacon=16) achieves the best trade-off.
Aggregation
Max KV Cache Budget
4096
2048
1024
512
256
Max
55.4
46.7
39.2
30.0
23.3
Mean
52.5
50.8
36.7
27.5
18.8
Table 2: Ablation study on aggregation in BeaconKV under different maximum KV cache budgets on R1-Distill-Qwen-7B for AIME24. We compare using Max vs. Mean aggregation consistently across heads and queries.
Model
Method
AIME 24
MATH -500
GPQA -Diamond
LiveCode Bench
R1-Distill-7B
BeaconKV
39.17
87.45
45.20
33.33
Initial+Recent
20.42
70.60
42.80
28.95
R1-Distill-8B
BeaconKV
45.00
86.70
48.23
36.11
Initial+Recent
44.58
81.40
45.07
33.16
Qwen3-4B
BeaconKV
33.33
79.75
50.25
42.21
Initial+Recent
19.58
55.95
49.87
33.93
Table 3: Accuracy comparison between BeaconKV and Initial+Recent across models and reasoning tasks under a maximum KV cache budget of 1024.
Method
Max KV Budget
Batch Size
Throughput (tokens/s)
Decode Latency
Peak Mem.
Acc.
Full KV
–
14
82.3
5573.4
77.0
54.4
BeaconKV
2K
14
356.4
1287.3
13.3
51.1
RPC vs. BeaconKV
RPC
2K
192
725.4
8672.5
79.0
44.8
BeaconKV
2K
192
704.8
8926.9
79.3
51.1
RPC
1K
320
1380.8
7593.7
72.0
29.9
Table 4: Efficiency evaluation on Qwen3-4B with a generation length of 32K. We compare the throughput, decoding latency (s), peak GPU memory usage (GB), and LiveCodeBench accuracy of Full KV, RPC, and BeaconKV.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Max KV Budget
Batch Size
Throughput (tokens/s)
Decode Latency
Peak Mem.
LiveCodeBench Acc.
SnapKV
1K
320
1380.7
7594.5
72.4
30.9
RPC
1K
320
1380.8
7593.7
72.0
29.9
BeaconKV
1K
320
1345.9
7790.9
72.5
42.2
Appendix
Table 5: Efficiency and accuracy comparison among SnapKV, RPC, and BeaconKV on Qwen3-4B under a 1K KV cache budget.
Model
AIME24
MATH-500
GPQA-Diamond
LiveCodeBench
R1-Distill-Qwen-7B
13459
4069
8210
11720
R1-Distill-Llama-8B
14253
4241
8755
11983
Qwen3-4B
14675
5342
6516
14100
Qwen3-14B
13801
4799
5358
12623
Appendix
Table 6: Average Number of Output Tokens by Model and Task.
Figure 9: Full token-level visualization of top- K attended positions for representative global and local queries in an AIME24 sample-0 on R1-Distill-Qwen-7B (Layer 18, Head 16). Colored underlines indicate the tokens most strongly attended to by each query. Global queries corresponding to Thought Revisiting Tokens (tokens 1068 and 1090) revisit earlier problem constraints and high-level solving plans, whereas local queries (tokens 1089 and 1091) mainly attend to nearby steps.
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.
Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks, it matches the strongest baseline in task performance while delivering 32-43% higher throughput than that method when deployed with vLLM. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.
Heng Wang, Jielin Qiu, Wenting Zhao +8
Salesforce AI Research · University of Illinois Urbana-Champaign
Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.