MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

Yangchengyu Zhou; Zhiping Wu; Runchang Jiang; Dongdong Ren; Youfang Han; Wei Zhao; Dongqi Cai; Wenbin Li

cs.LGOct 13, 2026

MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

Authors: Yangchengyu Zhou, Zhiping Wu, Runchang Jiang, Dongdong Ren, Youfang Han, Wei Zhao, Dongqi Cai, Wenbin Li

Organizations: Nanjing University · The Hong Kong University of Science and Technology · Geely · Goertek

Abstract

The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.

Figures & tables

Figure 1: Motivation for budget-aware precision allocation under a fixed KV cache budget. Different layers exhibit heterogeneous demands for layer precision enhancement and token preservation. Existing layer-wise and token-wise methods optimize individual preservation strategies, while MultiAlloc profiles these demands and allocates the shared memory budget across preservation strategies.

Figure 2: Overview of MultiAlloc . An offline profiling stage estimates layer-wise precision demands and token-wise preservation demands, followed by a budget-coupled precision allocation module that derives the static policy

(\mathbf{q}^{\star},\mathbf{k}^{\star})

under a fixed KV cache memory budget

B

. The policy is then directly applied during online inference.

Model	Method	KV Budget	GPQA	GSM8K	LongBench-v2	Avg
Llama-3-8B-Instruct	FP16	1024 MB	32.83	80.89	24.79	46.17
	OTT (ACL 2025)	423 MB	32.83	76.95	23.14	44.31
	PM-KVQ (ICLR 2026)	423 MB	30.30	79.98	24.56	44.95
	KVmix (AAAI 2026)	423 MB	32.83	74.83	26.94	44.87
	MultiAlloc (Ours)	423 MB	32.83	80.36	25.85	46.35
Mistral-7B-Instruct	FP16	1024 MB	27.27	48.75	24.78	33.60

Table 1: Main performance comparison of compressed methods under the same KV cache memory budgets.

Variant	Layer alloc.	Token presv.	LongBench-v2
Layer-only	$\checkmark$	$\times$	25.58
Token-only	$\times$	$\checkmark$	24.88
MultiAlloc (Ours)	$\checkmark$	$\checkmark$	25.85

Table 2: Component ablation of layer-wise allocation and token-wise preservation.

Allocation strategy	GSM8K	LongBench-v2
Manual Split (65/35)	80.29	25.59
Manual Split (50/50)	77.79	25.83
Manual Split (35/65)	77.41	25.56
MultiAlloc (Profiled)	80.36	25.85

Table 3: Comparison between profiling-guided budget allocation and manual budget splits.

Method / Budget	$B_{1}$	$B_{2}$	$B_{3}$	$B_{4}$
Llama-3-8B-Instruct (FP16: 80.89)
Budget (MB)	283	383	423	463
OTT	76.88	77.56	76.95	78.09
PM-KVQ	58.83	79.00	79.98	78.62
KVmix	75.13	74.07	74.83	75.21
MultiAlloc (Ours)	77.33	79.15	80.36	80.29

Table 4: Comparison of GSM8K accuracy (%) across multiple models under different KV cache budgets.

Panel of Figure 3 — Figure 3: Allocation behavior of MultiAlloc across different tasks under heterogeneous precision demands.

Figure 4: Runtime efficiency comparison on Llama-3-8B-Instruct under the same KV cache budget with varying input and generation lengths.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

\Delta K^{(b)}_{i,l}=Q_{b}(K_{i,l})-K_{i,l}

Appendix

Algorithm 1 Offline Profiling for Layer-wise Precision Demand and Token-level Trace

B_{\mathrm{token}}(k_{i})\leftarrow M_{\mathrm{token}}(k_{i})

Appendix

Algorithm 2 Collective Allocation under a Fixed KV Cache Budget

Panel of Figure 5 — Figure 5: Layer-wise KV cache precision allocation patterns on the evaluated 7B models. The assigned bit-widths vary across layers according to the profiled precision demands under fixed KV cache budgets.

Panel of Figure 6 — Figure 6: Layer-wise KV cache precision allocation patterns on larger-scale models. The distinct distributions demonstrate model-dependent precision demands and the necessity of profiling-guided allocation.

Panel of Figure 7 — Figure 7: Token preservation capacity allocation patterns on the evaluated 7B models. The available preservation budget is distributed non-uniformly across layers according to the profiled token-level preservation demands.

Panel of Figure 8 — Figure 8: Token preservation capacity allocation patterns on larger-scale models. The distributions vary across models and tasks, indicating workload-dependent token-level preservation demands under fixed KV cache budgets.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 101 papers

RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
Aug 9, 2026Dongjie Xu, Kai Qian, Julius +6KV Caching KV-Cache Management
1Soochow University · 2Tencent Inc. · 3The Hong Kong University of Science and Technology +2
Exploring a Layer-Wise Design Space for KV Cache Eviction
Jun 13, 2026Chao Fei, Kaihua Liang, Hanzhi Hu +4KV Caching KV-Cache Eviction
King Abdullah University of Science and Technology · Fudan University
Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache Quantization On-Device Language Model Inference
Clemson University, USA

Aug 9, 2026J/K move · Enter open · S save

RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

Dongjie Xu, Kai Qian, Julius, Weijie Shi+5

1Soochow University · 2Tencent Inc. · 3The Hong Kong University of Science and Technology+2

KV Caching KV-Cache Management KV-Cache Compression

Open paper AskPDF arXiv

Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging. Existing methods rely on proxies such as layer depth, attention statistics, or representation change. These proxies do not measure how perturbations at each layer propagate to the output and may therefore cause sensitive layers to be underallocated while tolerant layers are overallocated. To address this issue, we propose RippleKV, which allocates cache across layers by estimating how perturbations to each layer's value cache affect the final predictive distribution. RippleKV independently injects norm-adaptive perturbations into each layer's value cache and measures the induced KL divergence at the model output over a small calibration set. Averaging these responses yields a sensitivity profile specific to the model that need not vary monotonically with depth. RippleKV then converts the sensitivity profile into layer budget multipliers by normalizing the sensitivity scores and applying an exponential mapping. A ratio parameter controls the allocation disparity between sensitive and tolerant layers, while a final normalization preserves the KV cache budget. Experiments on LongBench demonstrate that RippleKV achieves the highest average performance among the evaluated KV cache compression methods under matched cache budgets.

Abstract

KV-Cache Quantization

Explore similar work

RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

Exploring a Layer-Wise Design Space for KV Cache Eviction

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs