Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

Seifeldin Abdellatif

cs.ARSep 2, 2026

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

Authors: Seifeldin Abdellatif

Organizations: College of Engineering, Al Ain University

Abstract

Low-bit key--value (KV) caches reduce the memory required for autoregressive decoding, but the resulting quality loss depends on the model and quantizer. We keep the quantizer fixed and distill the floating-cache model's behavior into low-rank Q/K/V projection updates while the student executes a physically packed incremental cache. Across three seeds, 4-bit affine-cache adapters recover $54.24\%\pm2.47\%$ of the held-out perplexity gap on TinyLlama-1.1B and $75.96\%\pm4.04\%$ on Gemma-4-12B. On the same frozen NF4 Llama-3.1-8B base, one validation-selected run per quantizer recovers $60.42\%$ under KIVI K2V2 and $37.61\%$ under KVarN K4V2, while preserving 180-case associative retrieval. Gemma's score on an official 4K/8K RULER subset rises from 42.80 with the unadapted 4-bit cache to 48.33 after adaptation (46.15 floating), with substantial task heterogeneity. Finally, a 2-bit rank--token sweep reduces TinyLlama's 2-bit PPL from 576.10 to $11.4000\pm0.0059$ across three seeds, versus 10.3988 floating, but restores only 11--12 of 180 retrieval cases. These results show that low-rank projection adaptation can recover held-out quality across fixed cache formats, while perplexity recovery need not restore long-context retrieval.

Figures & tables

Model / adapter (base)	Float	Packed	Adapted	$R$ (%)
Tiny, QK (FP16)	10.3988	10.7228	$10.5792\pm.0017$	$44.32\pm.52$
Tiny, QKV (FP16)	10.3988	10.7228	$10.5471\pm.0080$	$54.24\pm 2.47$
Gemma, QK (NF4)	26.5067	27.6208	$26.7745\pm.0450$	$75.96\pm 4.04$

Table 1: Main held-out PPL. Adapted values and

R

are mean

\pm

sample SD over three seeds. Base denotes weight precision; caches are 4-bit affine.

Figure 1: Calibration loss decreases smoothly. TinyLlama’s three QKV seeds nearly overlap (left), while QKV finishes lowest among the Llama-3.1 placements (right). Held-out quality is reported in Tables 1 and 2 .

TinyLlama-1.1B : float 10.3988; packed 10.7228
Place	Rank	Params (M)	Adapted PPL	$R$ (%)
Q	12	1.081	10.5818	43.54
K	22	1.115	10.6109	34.55
QK	8	1.126	10.5787	44.47
QKV	6	1.149	10.5383	56.95
Llama-3.1-8B : float 8.8195; packed 9.0475

Table 2: QKV gives the strongest recovery under matched adapter budgets for both models. Float and packed PPL are shared within each model; Params denotes trainable adapter parameters.

Setting	Packed	Adapted	$R$ (%)
Affine 2-bit, short QK	576.1030	18.7758	98.52
Affine 2-bit, 20M QKV	576.1030	$11.4000\pm.0059$	$99.823\pm.001$
Affine 4-bit	10.7228	10.5787	44.47
Affine 8-bit	10.4012	10.3983	$121.1^{\dagger}$
Symmetric 4-bit	11.2072	10.7570	55.69

Table 3: TinyLlama operating points (

P_{f}=10.3988

throughout). Scaling the 2-bit adapter closes most of the PPL gap; the 8-bit recovery ratio is unstable because its initial gap is near zero. Extended 2-bit values are mean

\pm

sample SD over three seeds; other rows are one seed.

Figure 2: Adaptation transfers to both established cache formats: it recovers part of each held-out PPL gap (left) while preserving first-code NIAH accuracy (right). Each method is one validation-selected training run.

Figure 3: Increasing rank and calibration monotonically improves TinyLlama’s 2-bit validation PPL (left), but retrieval reaches only 1.1–5.0% versus 95% with the floating cache (right). The unadapted 2-bit cache retrieves 0%.

Setting	Float	Packed	Seed 0	Seed 1	Seed 2
Tiny QKV, 4-bit	171/180	163/180	168/180	169/180	167/180
Tiny QK, 2-bit (short)	86/90	0/90	0/90	–	–
Tiny QKV, 2-bit (20M)	170/180	0/180	11/180	12/180	11/180
Tiny QK, 4-bit	86/90	81/90	86/90	87/90	81/90
Tiny QK, 8-bit	86/90	86/90	86/90	–	–
Gemma QK, 4-bit	114/120	112/120	116/120	117/120	115/120

Table 4: Associative NIAH first-code retrieval. Adapted columns are independently trained adapters; cases are shared within each row.

Cache	Length	Single	Multi	VT	CWE
Floating	4K	84	98	0.8	29.6
	8K	60	84	0.0	12.8
Packed	4K	80	85	0.8	37.6
	8K	60	61	0.0	18.0
Adapted	4K	56	97	0.0	69.2
	8K	48	90	0.0	26.4

Table 5: Official RULER subset scores (%) on Gemma-4-12B. Single and Multi are needle retrieval; VT is variable tracking and CWE common-word extraction. Eight-cell macro means are 46.15 (floating), 42.80 (packed), and 48.33 (adapted).

Figure 4: Cell-wise change in associative retrieval from the unadapted packed cache to the mean of three independently trained adapters. Green cells improve and red cells decline; values give the percentage-point difference.

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 102 papers

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
Jun 7, 2026Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang +3Mixed-Precision Quantization KV Caching
University of California San Diego, San Diego, CA, USA · Dnotitia Inc., Seoul, Republic of Korea · Hanyang University, Seoul, Republic of Korea
KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
Sep 21, 2026Sihyeon Ha, Jaeho Lee, Yo-Seb JeonLow-Rank Compression KV-Cache Compression
Pohang University of Science and Technology (POSTECH)
Read What Matters: Query-Adaptive Quantization for KV Caches
Oct 8, 2026Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian +1KV-Cache Quantization KV-Cache Compression
Amazon · University of California, Berkeley

Jun 7, 2026J/K move · Enter open · S save

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang, SeungJae Lee+2

University of California San Diego, San Diego, CA, USA · Dnotitia Inc., Seoul, Republic of Korea · Hanyang University, Seoul, Republic of Korea

Mixed-Precision Quantization KV Caching Low-Rank Matrix Decomposition LLM Inference Acceleration Low-Rank Compression Low-Rank Approximation+1

Open paper AskPDF arXiv

Low-rank projection has emerged as a promising approach for compressing the KV cache by exploiting hidden-dimension redundancy. However, prior methods rely on fixed or heuristic rank selection and struggle to achieve aggressive compression with minimal accuracy degradation. We propose STAR-KV, an adaptive low-rank KV cache compression framework with fine-grained rank control. STAR-KV encompasses 1) a differentiable thresholding mechanism that enables optimal rank selection at both attention-head and block levels, 2) a hybrid decomposition strategy that applies different low-rank factorizations according to the sensitivity of key and value projections, and 3) a low-rank-aware mixed precision quantization that leverages data statistics for near lossless low-bit quantization. Evaluated across multiple LLMs and benchmarks, STAR-KV achieves up to 75% KV cache compression and up to 20x overall KV cache reduction when combined with quantization. Enabled by custom Triton-based GPU kernels, STAR-KV delivers up to 6.9x speedup for the attention module and 3.1x end-to-end generation throughput. Our code is publicly available at: https://github.com/PriyanshBhatnagar/STAR-KV.

Abstract

KV-Cache Quantization

Explore similar work

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation

Read What Matters: Query-Adaptive Quantization for KV Caches