BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Janghyeon Kim; Minsoo Kim; Kyuhong Shim; Jungwook Choi

cs.LGSep 4, 2026

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Authors: Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi

Organizations: Hanyang University · Sungkyunkwan University

Abstract

Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to $5.8\times$ memory reduction while nearly preserving full cache accuracy and improving throughput by over $4.3\times$ .

Figures & tables

Figure 1: (a) Attention weight distribution at layer 18, head 16 and (b) attention distance between query and key (AIME24 sample-0 on R1-Distill-Qwen-7B).

Figure 2: Output text tokens with Top-

K

attention weights for each query at layer 18, head 16 (AIME24 sample-0 on R1-Distill-Qwen-7B). The full text is provided in Appendix E .

Figure 3: Occurrence distribution of global queries across layers and heads (AIME24 sample-0 on R1-Distill-Qwen-7B). Additional details are provided in Appendix B .

Figure 4: (a) Cosine similarity between queries within a specific decoding-step interval at layer 18, head 16. (b) PCA of queries with low average cosine similarity at layer 18, head 16.

Figure 5: Illustration of KV cache compression methods for LRMs during long decoding. (a) Prior methods (e.g., RPC) compute attention-based scores for KV cache eviction relying solely on recent queries. (b) BeaconKV (ours) computes the attention-based scores using beacon queries and recent queries. (c) BeaconKV performs top-

K

selection via max aggregation of attention-based scores.

Figure 6: (a) Maximum cosine similarity to future decoding step queries and (b) AIME24 accuracy on R1-Distill-Qwen-7B under different past query set selections.

Figure 7: Peak GPU memory and accuracy degradation on R1-Distill-Qwen-7B under different observation query selection methods. We compare Continual FPS against K-Means Centroids, Centroid-Nearest Data Points, and Naive FPS.

Figure 8: Accuracy comparison of BeaconKV against RPC, SnapKV, and R-KV. We evaluate the accuracy on R1-Distill-Qwen-7B, R1-Distill-Llama-8B, Qwen3-4B and Qwen3-14B across four reasoning tasks: AIME24, MATH-500, GPQA-Diamond, and LiveCodeBench.

Method	$(n_{\text{recent}},\,n_{\text{beacon}})$	Accuracy (%) $\uparrow$	Latency (s) $\downarrow$
BeaconKV	(1, 31)	63.5	10871.3
	(4, 28)	66.5	5624.5
	(8, 24)	64.8	4762.1
	(16, 16)	64.6	4355.7
	(24, 8)	62.3	4194.5
	(28, 4)	58.1	4179.5

Table 1: Ablation study on the number of beacon queries (

n_{\text{beacon}}

) and recent queries (

n_{\text{recent}}

) on Qwen3-4B for AIME24, with the maximum KV cache budget set to 2048. We report both accuracy and latency to identify the optimal configuration. The setting

(n_{\text{recent}}=16,n_{\text{beacon}}=16)

achieves the best trade-off.

Aggregation	Max KV Cache Budget
Aggregation	4096	2048	1024	512	256
Max	55.4	46.7	39.2	30.0	23.3
Mean	52.5	50.8	36.7	27.5	18.8

Table 2: Ablation study on aggregation in BeaconKV under different maximum KV cache budgets on R1-Distill-Qwen-7B for AIME24. We compare using Max vs. Mean aggregation consistently across heads and queries.

Model	Method	AIME 24	MATH -500	GPQA -Diamond	LiveCode Bench
R1-Distill-7B	BeaconKV	39.17	87.45	45.20	33.33
R1-Distill-7B	Initial+Recent	20.42	70.60	42.80	28.95
R1-Distill-8B	BeaconKV	45.00	86.70	48.23	36.11
R1-Distill-8B	Initial+Recent	44.58	81.40	45.07	33.16
Qwen3-4B	BeaconKV	33.33	79.75	50.25	42.21
Qwen3-4B	Initial+Recent	19.58	55.95	49.87	33.93

Table 3: Accuracy comparison between BeaconKV and Initial+Recent across models and reasoning tasks under a maximum KV cache budget of 1024.

Method	Max KV Budget	Batch Size	Throughput (tokens/s)	Decode Latency	Peak Mem.	Acc.
Full KV	–	14	82.3	5573.4	77.0	54.4
BeaconKV	2K	14	356.4	1287.3	13.3	51.1
RPC vs. BeaconKV
RPC	2K	192	725.4	8672.5	79.0	44.8
BeaconKV	2K	192	704.8	8926.9	79.3	51.1
RPC	1K	320	1380.8	7593.7	72.0	29.9

Table 4: Efficiency evaluation on Qwen3-4B with a generation length of 32K. We compare the throughput, decoding latency (s), peak GPU memory usage (GB), and LiveCodeBench accuracy of Full KV, RPC, and BeaconKV.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Max KV Budget	Batch Size	Throughput (tokens/s)	Decode Latency	Peak Mem.	LiveCodeBench Acc.
SnapKV	1K	320	1380.7	7594.5	72.4	30.9
RPC	1K	320	1380.8	7593.7	72.0	29.9
BeaconKV	1K	320	1345.9	7790.9	72.5	42.2

Appendix

Table 5: Efficiency and accuracy comparison among SnapKV, RPC, and BeaconKV on Qwen3-4B under a 1K KV cache budget.

Model	AIME24	MATH-500	GPQA-Diamond	LiveCodeBench
R1-Distill-Qwen-7B	13459	4069	8210	11720
R1-Distill-Llama-8B	14253	4241	8755	11983
Qwen3-4B	14675	5342	6516	14100
Qwen3-14B	13801	4799	5358	12623

Appendix

Table 6: Average Number of Output Tokens by Model and Task.

Figure 9: Full token-level visualization of top-

K

attended positions for representative global and local queries in an AIME24 sample-0 on R1-Distill-Qwen-7B (Layer 18, Head 16). Colored underlines indicate the tokens most strongly attended to by each query. Global queries corresponding to Thought Revisiting Tokens (tokens 1068 and 1090) revisit earlier problem constraints and high-level solving plans, whereas local queries (tokens 1089 and 1091) mainly attend to nearby steps.

Long-Context Reasoning in Language Models

See all 41 papers

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
Aug 5, 2026Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8Overthinking in Language Models KV Caching
1HKUST · 2XJTU · 3TSE +1
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Sep 3, 2026Heng Wang, Jielin Qiu, Wenting Zhao +8KV Caching Memory-Efficient Optimization
Salesforce AI Research · University of Illinois Urbana-Champaign
Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning
Oct 5, 2026Runguo LiCoT Reasoning KV-Cache Eviction
University of Illinois Urbana-Champaign

Aug 5, 2026J/K move · Enter open · S save

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Qiyuan Zhu, Dezhi Li, Pengyu Cheng, Tianle Chen+7

1HKUST · 2XJTU · 3TSE+1

Overthinking in Language Models KV Caching Efficient Language Model Reasoning LLM Inference Acceleration KV-Cache Compression Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning