Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

Mengqi Liao; Lu Wang; Chaoyun Zhang; Bo Qiao; Si Qin; Qingwei Lin; Saravan Rajmohan; Dongmei Zhang

cs.DCMar 1, 2026

Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

Authors: Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang,

Organizations: School of Computer Science and Technology, Beijing Jiaotong University · Microsoft · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence

Abstract

With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95% of the performance of Full KV inference engines while delivering over 2.1 $\times$ speedup.

Figures & tables

Figure 1: Illustration of Compressed PagedAttention. Here,

N_{\max}=4,b=4,w=2

. The figure depicts two requests requiring compression. After compression, the kept KV cache entries are moved to the first three blocks, while the fourth block is reserved for subsequent decoding. The remaining blocks are released.

Figure 2: State transition diagram of requests under hybrid scheduling.

Figure 3: Illustration of block allocation and release strategies for prefix cache.

Figure 4: Average time per step and ratio during inference with Qwen3 0.6B and Qwen3 8B on AMC 23 under non-asynchronous compression settings.

Figure 5: Comparison of average TPOT (ms) of all requests across different configurations on three workloads.

Figure 6: Comparison of TPS (tokens/s) across different configurations on three workloads.

Panel of Figure 7 — Figure 7: The figure shows Qwen3 8B’s performance using Zipage or Nano-vLLM on AMC 23, including:(a) real-time throughput, (b) per-step real-time decoding time, (c) average per-step time at different concurrency range, (d) and the ratio of steps to total steps under different concurrency range.

Figure 8: Comparison of TPS and pass@1 performance across different methods for Qwen3 8B and DS LLaMA 8B. The numbers in parentheses indicate the maximum batch size.

Panel of Figure 9 — Figure 9: Evaluation results under varying KV cache budgets on (a) AMC 23, (b) AIME 24, and (c) LiveCodeBench. Dashed lines represent the results of full KV, and the percentages following the numerical values indicate the relative performance compared to full KV.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Workloads	Number of Questions	Sample Times	Max Output Length
AMC 23	40	32	16384
AIME 24	30	32	32768
LiveCodeBench v1	400	8	16384
Mixture	40+1319	4	16384
LongBench	150	4	4096

Appendix

Table 1: Evaluation settings for different workloads or benchmarks.

Figure 10: Average time for compression and decoding step under different layer stride.

\mathbf{M}_{u,v}=\begin{cases}-\infty&\text{if }u+b-w>v\\ 0&\text{otherwise}\end{cases}

Appendix

Algorithm 1 Block Attention Logits Computation

$\alpha$	0	0.4	0.8	0.85	0.9	1
Qwen3 8B	0.6906	0.7375	0.7718	0.7468	0.7484	0.7531
DS Llama 8B	0.7484	0.7656	0.7643	0.7584	0.7578	0.7515

Appendix

Table 2: Experimental results with different decay rates

\alpha

Figure 11: This figure illustrates the computation of the original redundancy scores when

N=2

and

b=4

. Here,

\mathbf{K}_{i}

represents the key states of the

i

-th block. This approach has a computational complexity of

\mathcal{O}(N^{2}\times b^{2})

and a memory complexity of

\mathcal{O}(N^{2}\times b^{2})

Figure 12: This figure illustrates the computation of flash redundancy score. The green blocks represent data stored in global memory, i.e., activations. The results of other computations, shown in different colors, are temporarily stored using registers or shared memory. This approach has a computational complexity of

\mathcal{O}(N^{2}\times b^{2})

and a memory complexity of

\mathcal{O}(N^{2}\times b)

Figure 13: The average execution time of different operations when asynchronous compression is disenabled. A value of 0.0 indicates that the average execution time is less than 0.1 milliseconds.

Figure 14: The average execution time of different operations when asynchronous compression is enabled.

Figure 15: This figure illustrates the computation of lightning redundancy score. This approach has a computational complexity of

\mathcal{O}(N\times b^{2})

and a memory complexity of

\mathcal{O}(N\times b)

Figure 16: The average decoding time and compression time under both asynchronous and non-asynchronous compression settings.

$\lambda$	0.05	0.1	0.2	0.5	0.9
Flash	80.78	82.96	84.37	84.53	77.81
Lightning	83.59	84.84	84.84	84.21	75.00

Appendix

Table 3: Qwen3 8B on AMC 23

$\lambda$	0.05	0.1	0.2	0.5	0.9
Flash	82.96	82.03	83.59	78.59	74.21
Lightning	80.62	85.15	85.00	80.78	73.90

Appendix

Table 4: DS-Llama 8B on AMC 23

Figure 17: pass@1 performance for Qwen3 8B and DS Llama 8B under diffirent

\lambda

and

\tau

Figure 18: Comparison under different KV budgets for Qwen3 8B and DS Llama 8B, using global and redundancy scores versus redundancy-only scores.

Model	w/o pooling	pooling once	w/ pooling
Qwen3 8B	87.10	87.03	80.85
DS Llama 8B	84.21	83.20	84.53

Appendix

Table 5: Comparison of pooling strategies for LLaMA-8B and Qwen-8B.

Panel of Figure 19 — Figure 19: Number of running requests during inference.

Panel of Figure 20 — Figure 20: Number of waiting requests during inference.

Panel of Figure 21 — Figure 21: Throughput during inference.

Panel of Figure 22 — Figure 22: Block occupancy during inference.

Panel of Figure 23 — Figure 23: The number of running and waiting requests during inference during inference on AMC23.

Panel of Figure 24 — Figure 24: The real-time throughput, number of running and waiting requests during inference during inference on the mixture of GSM8K and AMC 23.

Panel of Figure 25 — Figure 25: The real-time throughput, number of running and waiting requests during inference during inference on LongBench.

Figure 26: TPS and pass@1 performance across different model sizes. TP=2 indicates that the model is run on two GPUs using tensor parallelism, while all other experiments are conducted on a single GPU by default.

Panel of Figure 27 — Figure 27: The figure shows Qwen3 0.6B’s performance using Zipage or Nano-vLLM on AMC 23, including:(a) real-time throughput, (b) per-step real-time decoding time, (c) average per-step time at different concurrency range, (d) and the ratio of steps to total steps under different concurrency range.

Panel of Figure 28 — Figure 28: The figure shows Qwen3 0.6B’s performance using Zipage or Nano-vLLM on AMC 23, including:(a) real-time throughput, (b) per-step real-time decoding time, (c) average per-step time at different concurrency range, (d) and the ratio of steps to total steps under different concurrency range.

Panel of Figure 29 — Figure 29: The figure shows Qwen3 32B’s performance using Zipage or Nano-vLLM on AMC 23, including:(a) real-time throughput, (b) per-step real-time decoding time, (c) average per-step time at different concurrency range, (d) and the ratio of steps to total steps under different concurrency range. (Tensor parallelism = 2)

Panel of Figure 30 — Figure 30: The figure shows DS Llama 8B’s performance using Zipage or Nano-vLLM on AMC 23, including:(a) real-time throughput, (b) per-step real-time decoding time, (c) average per-step time at different concurrency range, (d) and the ratio of steps to total steps under different concurrency range.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,048 papers

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Sep 3, 2026Qiankun Ma, Yanjiang Zhou, Zinan Xiong +5LLM Inference KV Caching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences
ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval
Apr 13, 2026David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4Memory-Efficient Inference Efficient Language Model Reasoning
Rensselaer Polytechnic Institute · IBM Research
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
Sep 7, 2026Michael Wang, Keith Li, Roozbeh BostandoostLLM Inference Efficiency LLM Inference
Lake Washington School District · XSchool · UMass Amherst

Sep 3, 2026J/K move · Enter open · S save

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

Qiankun Ma, Yanjiang Zhou, Zinan Xiong, Haofei Wang+4

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences

LLM Inference KV Caching Memory-Efficient Optimization KV-Cache Management LLM Inference Acceleration KV-Cache Compression+1

Open paper AskPDF arXiv

Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workloads exhibit substantial demand variation: different requests require different KV capacities, and the attention demand of an individual request evolves during generation. We introduce GrowPage, an on-demand KV budgeting framework that treats KV capacity as a runtime resource. GrowPage maintains lightweight dual-timescale query summaries to capture recent and long-term attention behaviors, and uses their relative attention working sets to estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page when broader demand emerges. By integrating with PagedAttention's page-level memory abstraction, GrowPage preserves continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models show that GrowPage achieves a superior performance--throughput trade-off over existing approaches.

Abstract

LLM Inference Acceleration

Explore similar work

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference