InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

Sayed Mohammadreza Tayaranian Hosseini; Amir Ardakani; Warren J. Gross

cs.LGFeb 26, 2026

InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

Authors: Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross

Organizations: Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada

Abstract

When transformer-based language models are deployed for text generation, most of the inference time is spent in the decoding stage, where output tokens are generated sequentially. Reducing the hardware cost of each decoding step is therefore critical for efficient long-context generation. A major bottleneck is the key-value (KV) cache, whose size grows with sequence length and often dominates the model's memory footprint. Prior work has proposed quantization methods to compress the KV cache while minimizing its loss of precision. We present InnerQ, a hardware-aware KV cache quantization scheme that reduces decode latency without compromising evaluation performance. InnerQ performs group-wise quantization by grouping cache matrices along their inner dimension. This grouping strategy aligns dequantization with vector-matrix multiplication and increases data reuse across GPU compute units. As a result, InnerQ reduces memory access and accelerates dequantization, achieving an average $1.3\times$ speedup over prior KV cache quantization methods and $2.7\times$ over the non-quantized baseline. To maintain fidelity under aggressive compression, InnerQ incorporates three techniques: (i) hybrid quantization, which chooses symmetric or asymmetric quantization for each group based on local statistics; (ii) high-precision windows for both recent tokens and attention sink tokens to mitigate outlier leakage; and (iii) per-channel normalization of the key cache, computed once during prefill and folded into the model parameters to eliminate runtime overhead. Beyond reducing latency, experiments on Llama and Mistral models show that InnerQ also improves few-shot evaluation scores relative to prior KV cache quantization methods.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of the vector-matrix multiplication between the floating-point vector $Q$ and the quantized matrix $\hat{K}_{\mathrm{cache}}$ in an illustrative example. Cells with the same color are in the same quantization group and share a scale factor and zero point. A similar visualization is true for the vector $P$ and the quantized matrix $\hat{V}_{\mathrm{cache}}$ .

Panel of Figure 2 — Figure 2: Overall schematics of using our proposed KV cache quantization method in the attention computation process (Equation 3 ). The value cache has a similar behavior. A small high-precision window consisting of sink tokens ( $K_{\mathrm{sink}}$ ) and recent tokens ( $K_{\mathrm{recent}}$ ) is maintained along with the quantized kernels ( $\hat{K}$ ). The portions of the output corresponding to the quantized kernel and the full-precision windows are computed separately and then merged.

Panel of Figure 3 — Figure 3: High-level view of the quantization process in the hybrid quantization mode for a quantization group $\mathcal{G}_{i,g}$ .

Dataset	Model	Baseline	KIVI	KIVI Sink	TurboQuant	InnerQ Base	InnerQ Hybrid	InnerQ Small
GSM8k	Llama 3.2-1B	$6.22$	$2.81$	$4.02$	5.84	$5.53$	$5.31$	$4.63$
	Llama 3.2-3B	$26.46$	$19.33$	$23.35$	$25.47$	$27.14$	$23.12$	$24.03$
	Llama 3.1-8B	$49.96$	$42.84$	$47.84$	$48.90$	$48.22$	$47.31$	$45.94$
	Llama 2-7B	$13.04$	$13.12$	$11.90$	$12.59$	$13.42$	$11.30$	$0.46$
	Mistral 7B v0.3	$36.92$	$32.30$	$33.81$	$34.72$	$36.47$	$33.89$	$33.89$
	Average	$26.52$	$22.08$	$24.18$	$25.50$	$26.16$	$24.19$	$21.79$

Table 1: Evaluation metric score of models on different language generation tasks with quantized KV cache, compared to the non-quantized FP16 baseline. We report flexible_extract for GSM8k, math_verify for Minerva Math, and pass_at_1 for Humaneval and MBPP tasks. We use instructtion-tuned models for the Humaneval and MBPP tasks.

Dataset	Model	Baseline	KIVI	KIVI Sink	InnerQ Base	InnerQ Hybrid	InnerQ Small
qasper	Llama 3.2-1B	$20.29$	$18.00$	$17.23$	$18.37$	$20.43$	$19.84$
	Llama 3.2-3B	$40.63$	$37.09$	$38.49$	$40.32$	$39.59$	$38.84$
	Llama 3.1-8B	$44.58$	$44.08$	$43.61$	$44.59$	$42.40$	$42.36$
	Llama 2-7B	$20.99$	$21.11$	$20.30$	$21.84$	$22.18$	$13.12$
	Llama 2-13B	$16.59$	$15.09$	$15.48$	$17.85$	$16.56$	$17.50$
gov-report	Llama 3.2-1B	$29.27$	$28.57$	$28.87$	$28.89$	$26.72$	$26.70$

Table 2: Evaluation metric score of models on select tasks from the longbench suite with quantized KV cache, compared to the non-quantized FP16 baseline. We report F1 for qasper and triviaqa, ROUGE-L for gov-report, multinews, and samsum, accuracy for trec, and pass_at_1 for repobench-p. We use instruction-tuned models for Llama 3 models and chat version of Llama 2 models.

		KIVI	TurboQuant	InnerQ Base	InnerQ Hybrid	InnerQ Small
Key Cache	Integer Bit-width	$2$	$4$	$3$	$3$	$3$
	Scale Factor Overhead	$0.5$	$-$	$0.5$	$0.5$	$0.5$
	Zero-point Overhead	$0.5$	$-$	$-$	$-$	$-$
	Channel Norm Overhead	$-$	$0.25$	$-$	$-$	$-$
Value Cache	Integer Bit-width	$2$	$3$	$3$	$2$	$2$
	Scale Factor Overhead	$0.5$	$-$	$0.5$	$0.5$	$0.5$

Table 3: The per-number effective bit-width of different KV cache quantization methods. Bits for scale factors and zero-points are averaged to represent the overhead on each quantized number. We assume that scale factors and zero-points are represented using FP16 while channel norms are represented using FP32 . The group size is

32

and the head dimension is

128

		Sequence Length
	Method	512	1024	2048	4096	8192	16384	32768
Key Cache (Equation ( 3 ))	Baseline ( FP16 )	$76$	$147$	$291$	$576$	$1148$	$2291$	$4593$
	KIVI	$39$	$72$	$138$	$270$	$535$	$1063$	$2120$
	TurboQuant	$34$	$62$	$118$	$230$	$453$	$901$	$1796$
	InnerQ Base	$30$	$53$	$99$	$192$	$378$	$749$	$1492$
	InnerQ Hybrid	$30$	$53$	$99$	$192$	$378$	$749$	$1492$

Table 4: Latency breakdown ( µ s) of the fused dequantize-GEMV kernels for Llama 3.1-8B using different KV cache quantization methods along with the FP16 GEMV as the baseline.

Panel of Figure 4 — Figure 4: Total speedup in the operations in Llama 3.1-8B associated with KV cache when using InnerQ variants for KV cache quantization, relative to (left) non-quantized FP16 , (middle) KIVI [ 14 ] , and (right) TurboQuant [ 23 ] .

	KIVI	TurboQuant	InnerQ Base	InnerQ Hybrid	InnerQ Small
Key Cache	$1.0$	$15.9$	$17.3$	$17.3$	$17.3$
Value Cache	$21.1$	$16.1$	$1.0$	$1.4$	$0.9$
Total	$22.1$	$31.9$	$18.3$	$18.7$	$18.2$

Table 5: Latency breakdown ( µ s) of the quantization operations during the decode phase for Llama 3.1-8B using different quantization methods.

Panel of Figure 5 — Figure 5: Effect of changing $w_{\mathrm{sink}}$ on the evaluation metric performance of (a) Llama 3.1-8B and (b) Mistral 7B v0.3. At each step $w_{\mathrm{recent}}=128-w_{\mathrm{sink}}$ . We report flexible_extract for GSM8k, math_verify for Minerva Math, and pass_at_1 for Humaneval and MBPP tasks. We use instruct tuned models for the Humaneval and MBPP tasks.

Sparsity of $\mathcal{M}$	Sequence Length
Sparsity of $\mathcal{M}$	1024	4096	16384	32768
$99\%$	$59.0$	$214.4$	$841.9$	$1685.4$
$90\%$	$61.2$	$218.6$	$849.0$	$1701.5$
$50\%$	$65.3$	$231.2$	$900.1$	$1800.7$
$1\%$	$65.9$	$233.1$	$910.1$	$1814.9$

Table 6: Latency ( µ s) of the fused hybrid dequantization and GEMV in Llama 3.1-8B for Equation ( 5 ) for different cache sizes and levels of sparsity of

\mathcal{M}

Bit-width	Quantization Mode	Model
Bit-width	Quantization Mode	Llama 3.2-3B	Llama 3.1-8B	Llama 2-7B	Mistral 7B v0.3
$K$ :3, $V$ :3	$K$ :Sym, $V$ :Sym	$27.14$	$48.22$	$13.42$	$36.47$
	$K$ :Sym, $V$ :Asym	$25.78$	$48.67$	$12.66$	$35.86$
	$K$ :Asym, $V$ :Sym	$25.17$	$48.60$	$13.27$	$36.39$
	$K$ :Asym, $V$ :Asym	$23.88$	$47.23$	$12.13$	$32.37$
	$K$ :Sym, $V$ :Hybrid	$27.29$	$48.60$	$13.27$	$37.60$

Table 7: flexible_extract evaluation score on GSM8K for different KV-cache quantization modes across four language models. In all configurations, both

K

and

V

are quantized along their inner dimensions.

Hardware-Aware Quantization

See all 17 papers

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache Quantization On-Device Language Model Inference
Clemson University, USA
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Mar 17, 2026Yixuan Wang, Qingyu Shi, Jiayu Zhou +3KV-Cache Compression KV-Cache Quantization
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2
Tailoring the Quantization Space for 1-Bit KV Cache Compression
Oct 2, 2026Minsoo Cheong, Donghyun Son, Sungjoo YooBinary Quantization KV-Cache Compression
Stanford University · Seoul National University

Apr 6, 2026J/K move · Enter open · S save

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods, Gabriel Hillesheim+1

Clemson University, USA

KV-Cache Quantization On-Device Language Model Inference LLM Inference Acceleration Mixed-Precision Quantization

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable progress across reasoning, generation, and decision-making tasks, yet deploying them on mobile, embedded, and edge devices remains particularly challenging. On-device LLM inference is heavily constrained by the memory and bandwidth overhead of the key-value (KV) cache, which grows linearly with context length and often dominates decoding cost. Existing KV-cache quantization schemes typically rely on fixed precision or hand-crafted heuristics, thereby wasting bits on low-impact tokens while over-compressing informative ones, leading to avoidable accuracy degradation. Inspired by Huffman coding's principle of variable-length allocation, we propose adaptive KV-cache quantization, a learned policy that assigns bit-width proportional to token importance, minimizing expected memory and latency without sacrificing competitive accuracy. Our framework extracts lightweight token-level features, including token frequency, quality score, attention variance, and entropy-based uncertainty, and feeds them into a compact data-driven controller that dynamically selects KV precision from {2-bit, 4-bit, 8-bit, FP16} during decoding. This adaptive precision policy reduces KV memory footprint and latency while improving accuracy compared to static KV quantization and rule-based baselines, and maintaining competitive accuracy close to FP16 inference across standard LLM benchmarks. Extensive experiments across multiple commonsense reasoning benchmarks using SmolLM-135M, SmolLM-360M, and SmolLM-1.7B demonstrate that our controller consistently improves the accuracy-latency trade-off. For instance, with SmolLM-360M on HellaSwag, our method reduces decoding latency (ms/token) by 17.75% relative to static KV quantization, improves accuracy by 7.60 points, and remains within only 0.30 points of FP16 inference.

Abstract

Hardware-Aware Quantization

Explore similar work

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization

Tailoring the Quantization Space for 1-Bit KV Cache Compression