Large Language Models (LLMs) are increasingly deployed in scenarios demanding ultra-long context reasoning, such as agentic workflows and deep research understanding. However, long-context inference is constrained by the KV cache, a transient memory structure that grows linearly with sequence length and batch size, quickly dominating GPU memory usage. Existing memory reduction techniques, including eviction and quantization, often rely on static heuristics and suffer from degraded quality under tight budgets. In this paper, we propose ARKV, a lightweight and adaptive framework that dynamically allocates precision levels to cached tokens based on per-layer attention dynamics and token-level importance. During a short prefill phase, ARKV estimates the original quantization (OQ) ratio of each layer by computing statistical scores such as attention entropy, variance and kurtosis. During decoding, tokens are assigned to one of three states, Original (full precision), Quantization (low precision), or Eviction, according to a fast heavy-hitter scoring strategy. Our experiments on LLaMA3 and Qwen3 models across diverse long- and short-context tasks demonstrate that ARKV preserves ~97% of baseline accuracy on long-context benchmarks while reducing KV memory usage by 4x, with minimal throughput loss. On short-context tasks, ARKV matches full-precision baselines; on GSM8K math reasoning, it significantly outperforms uniform quantization. These results highlight the practical viability of ARKV for scalable LLM deployment, offering fine-grained, data-driven memory control without retraining or architectural modifications. The source code and artifacts can be found in: https://github.com/Large-scale-Sustainable-Computing-LSC/ARKV
Figures & tables
Fig. 1: Overview of the ARKV framework. During the Prefill Phase, attention statistics are used to compute an Original–Quantization (OQ) ratio and allocate per-layer cache budgets. In the Decoding Phase, tokens are scored by importance and dynamically assigned to Original, Quantization, or Eviction states. The reconstructed KV cache feeds into the attention mechanism with mixed-precision handling to balance memory efficiency and model fidelity.
Metric / Benchmark
Description
TPS (Tokens Per Second)
Token generation rate
GSM8k [ 27 ]
Arithmetic reasoning
MMLU [ 28 ]
General knowledge
CommonsenseQA [ 29 ]
Commonsense questions
LongBench [ 23 ]
Long-context understanding
TABLE I: Evaluation Metrics and Benchmarks
Fig. 2: LongBench performance across different models and KV cache strategies. The top panel shows absolute scores, while the bottom panel presents performance relative to the full-cache Base model (normalized to 1.0). Each group includes results for Origin (green), Quant (orange), and ARKV (blue), evaluated at token budgets of 512, 1024, and 2048 (from light to dark). ARKV consistently outperforms quantization-only baselines and closely matches origin-only performance, even under tight memory budgets.
Base
Origin
Quant
ARKV
1.00
0.979
0.398
0.972
TABLE II: Overall Relative Performance on LongBench
task type
Base
Origin
Quant
ARKV
Code
1.00
1.00
0.34
0.98
Few shot
1.00
1.00
0.48
1.00
Multi-doc QA
1.00
0.94
0.52
0.95
Single-doc QA
1.00
0.96
0.34
0.94
Summarization
1.00
0.95
0.27
0.93
Synthetic
1.00
1.04
0.59
1.03
TABLE III: Average relative performance by task type
language
Base
Origin
Quant
ARKV
EN
1.00
0.98
0.49
0.98
Python/C#/Java
1.00
1.00
0.38
0.99
Python/Java
1.00
0.99
0.31
0.97
ZH
1.00
0.95
0.28
0.94
TABLE IV: Average relative performance by language
Fig. 3: GSM8K performance under various KV cache strategies and budgets across different models. The top panel shows absolute accuracy scores, while the bottom panel presents performance relative to the full-cache Base model (normalized to 1.0). Green bars represent origin-only retention, orange bars indicate uniform quantization, and blue bars denote ARKV. Each budget level (512, 1024, 2048 tokens) is visualized from light to dark. ARKV consistently preserves high accuracy across all models, whereas quantization-only methods degrade sharply, especially for small budgets and smaller models.
Budget
Base
Origin
Quant
ARKV
512
0.791
0.696
0.010
0.697
1024
0.791
0.780
0.401
0.789
2048
0.791
0.794
0.789
0.788
TABLE V: Mean accuracy on GSM8K under different KV cache budgets. ARKV consistently matches Origin and substantially outperforms Quant, especially under tight memory constraints. The best result within each budget block is underlined.
Model
Budget
Base
Origin
Quant
ARKV
LLaMA3.2-3B
512
1.000
0.898
0.717
0.835
LLaMA3.2-3B
1024
1.000
0.864
0.683
0.810
LLaMA3.2-3B
2048
1.000
0.876
0.685
0.823
LLaMA3.1-8B
512
1.000
0.860
0.760
0.885
LLaMA3.1-8B
1024
1.000
0.890
0.773
0.883
LLaMA3.1-8B
2048
1.000
0.875
0.769
0.881
TABLE VI: Relative inference throughput (TPS) of each method compared to the base model (1.0), across four LLM variants. ARKV maintains ∼ 85% of base throughput on average, with significantly better efficiency than Quant under all settings.
Fig. 4: Relative throughput (TPS) across different models under various KV cache strategies and budgets. Each group shows performance of Base, Origin, Quant, and ARKV at budgets of 512, 1024, and 2048 tokens (from light to dark shades). Green bars represent full-precision eviction (Origin), orange bars show uniform quantization (Quant), and blue bars indicate ARKV. ARKV consistently retains high throughput ( ∼ 85–88%) across models while enforcing cache constraints.
Fig. 5: Histograms of FP8 quantization ratios (top) and eviction ratios (bottom) under ARKV. Dashed lines indicate the mean and peak values. Quantization ratios cluster around ∼ 0.14, while eviction ratios vary widely and are strongly influenced by the KV cache budget.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97% of full-cache performance using only 3% of the KV cache on LongBench question-answering tasks and achieves 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies.
Xintong Yang, Hao Gu, Binxing Xu +6
The Hong Kong University of Science and Technology · Zhejiang University
The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.
Daeha Lee, Do-Hyung Kim, Jae-Hong Kim
Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea