Large language models (LLMs) have shown great performance on complex reasoning tasks but often require generating long intermediate thoughts before reaching a final answer. During generation, LLMs rely on a key-value (KV) cache for autoregressive decoding. However, the memory footprint of the KV cache grows with output length. Prior work on KV cache optimization mostly focus on compressing the long input context, while retaining the full KV cache for decoding. For tasks requiring long output generation, this leads to increased computational and memory costs. In this paper, we introduce ZoomR, a novel approach that enables LLMs to adaptively compress verbose reasoning thoughts into summaries and uses a dynamic KV cache selection policy that leverages these summaries while also strategically "zooming in" on fine-grained details. By using summary keys as a coarse-grained index during decoding, ZoomR uses the query to retrieve details for only the most important thoughts. This hierarchical strategy significantly reduces memory usage by avoiding full-cache attention at each step. Experiments across math and reasoning tasks show that our approach achieves competitive performance compared to baselines, while reducing inference memory requirements by more than 4×. These results demonstrate that a multi-granularity KV selection enables more memory efficient decoding, especially for long output generation.
Figures & tables
Figure 1: Fine-grained details are often needed in order to generate the correct reasoning. Attending to only a compressed source of information like summaries can lead to incorrect intermediate steps.
Figure 2: An overview of the ZoomR workflow. Mean summary keys are computed and cached on GPU. When the KV cache selection needs to be updated, approximate attention scores are calculated across all layers. A global top- c selection identifies which summaries to zoom into. The final sparse KV cache loaded to GPU is a multi-granularity mix of an attention sink (green), selected detailed segments, summaries, and fixed recent window (blue). The symbolic attention map illustrates the dynamic nature of ZoomR during decoding.
Figure 3: Efficiency, consensus, and ablation studies. Fig. (a,b) shows the GPU memory and throughput during inference, respectively. Fig. (d,e) show consensus dynamics between correct and incorrect answers over time. Fig. (c,f) illustrates the relationships between c and k on performance and KV cache selection size.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
LoRA Rank ( r )
16
LoRA Alpha ( α )
32
Number of Epochs
3
Learning Rate
1×10−4
Batch Size
1
Gradient Accumulation Steps
4
Appendix
Table 2: LoRA Fine-tuning Hyperparameters
Figure 4: Prompt used to generate summaries
Figure 5: An example of a segment of reasoning process with intermediate summaries.
Figure 6: An example from AIME2025. ZoomR generated thoughts with summarization.
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95% of the performance of Full KV inference engines while delivering over 2.1× speedup.
Mengqi Liao, Lu Wang, Chaoyun Zhang +6
School of Computer Science and Technology, Beijing Jiaotong University · Microsoft · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory overhead by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) their threshold-triggered compression policy may provide limited throughput improvement or even reduce throughput, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) they typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and effectively improves output throughput. Extensive experiments demonstrate consistent performance improvements of proposed Kara and KvLLM.