In large language models performing long-form reasoning, the KV cache grows rapidly with decode length, creating bottlenecks in memory and inference stability. Existing reasoning-oriented KV compression has mostly followed an eviction-centered view: estimate token importance more accurately, then discard lower-ranked entries. Our analysis suggests that scorer refinement alone often fails to substantially reorganize the actual keep-set and may therefore not be the main lever for preserving reasoning behavior. We instead frame reasoning KV compression as a behavior-preserving structured consolidation problem. CASK partitions the decode-time reasoning trace into a protected core that anchors answer formation and intermediate state, and mergeable scratch with high redundancy. The core is preserved, while selective consolidation is applied only to the scratch. To address prompt-heavy regimes where the prefix can exhaust the budget before decode-stage compression becomes active, CASK further uses a two-stage design: prefix eviction followed by decode-stage consolidation. On the H100 reasoning gate, CASK shows higher full-KV continuation fidelity than TriAttention at matched budgets on both AIME24 and AIME25, with recurring cask@384 > triattention@512 crossings. In prompt-heavy replay, multi_news and vcsum act as decode-active witnesses, while qmsum and gov_report expose the prefix_budget_exhausted boundary. The overall evidence supports a simple conclusion: effective reasoning KV compression depends less on more elaborate scorer engineering than on combining core preservation with selective scratch consolidation to lower the usable budget frontier.
Figures & tables
Figure 1: Method overview. CASK separates prefix-stage eviction from decode-stage selective consolidation, while enforcing protected core preservation inside the decode trace.
Figure 2: H100 reasoning replay gate. CASK improves replay fidelity at matched budgets on both AIME24 and AIME25, and repeatedly crosses higher-budget TriAttention points.
Slice
Method
Budget
Top-1
Top-5
Mean NLL
First Mismatch
Saved Ratio
AIME24 ref6
TriAttention
256
86.1%
98.0%
0.463
8.7
65.3%
AIME24 ref6
CASK
256
88.4%
99.2%
0.359
8.7
65.3%
AIME24 ref6
TriAttention
384
88.2%
98.9%
0.383
8.7
61.6%
AIME24 ref6
CASK
384
90.7%
99.7%
0.268
8.7
61.6%
AIME24 ref6
TriAttention
512
89.4%
99.1%
0.333
8.7
43.6%
AIME24 ref6
CASK
512
91.7%
99.9%
0.233
8.7
43.5%
Table 1: Table 1. H100 reasoning fidelity gate
Figure 3: Prompt-heavy weighted aggregate. Across the package-level replay summary, CASK improves weighted top-1, weighted top-5, and weighted mean NLL at both budgets.
Figure 4: Prompt-heavy witness map. The tasks do not all play the same role: some are decode-active witnesses, some are prefix-dominant crossings, and some are retained boundaries.
Figure 5: Actual-output bridge. Replay-level gains are not purely diagnostic: they connect to lexical, semantic, and task-level output behavior on selected witnesses.
Task
Variant
Budget
Sequence Ratio
Prefix Ratio
Output Ratio
Task Metric
Terminal Saved
Compression Events
qasper
TriAttention
512
17.3%
3.7%
100.0%
11.94
-
-
qasper
CASK
256
23.8%
5.5%
100.0%
12.77
90.9%
1.0
multi_news
TriAttention
384
0.0%
0.0%
100.0%
0.00
-
-
multi_news
CASK
384
16.9%
8.1%
100.0%
15.16
84.1%
3.0
hotpotqa
TriAttention
256
100.0%
100.0%
100.0%
27.27
-
-
hotpotqa
CASK
256
100.0%
100.0%
100.0%
27.27
97.6%
0.0
Table 9
Figure 6: Mass recovery on the hexagon witness. Directly protected core mass is insufficient on its own; representative reconstruction recovers a large share of oracle-relevant mass.
Figure 7: Healthy representative preservation versus boundary collapse. Collapse is structural: when available slots collapse, unrecovered oracle-relevant mass grows sharply.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Prompt-heavy per-task mean NLL view. This appendix-level figure preserves the per-task NLL readout underlying the package-level prompt-heavy interpretation.
Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks, it matches the strongest baseline in task performance while delivering 32-43% higher throughput than that method when deployed with vLLM. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.
Heng Wang, Jielin Qiu, Wenting Zhao +8
Salesforce AI Research · University of Illinois Urbana-Champaign
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.