cs.CVApr 6, 2026

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

Authors: Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods, Gabriel Hillesheim, Abolfazl Razi

Organizations: Clemson University, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) have achieved remarkable progress across reasoning, generation, and decision-making tasks, yet deploying them on mobile, embedded, and edge devices remains particularly challenging. On-device LLM inference is heavily constrained by the memory and bandwidth overhead of the key-value (KV) cache, which grows linearly with context length and often dominates decoding cost. Existing KV-cache quantization schemes typically rely on fixed precision or hand-crafted heuristics, thereby wasting bits on low-impact tokens while over-compressing informative ones, leading to avoidable accuracy degradation. Inspired by Huffman coding's principle of variable-length allocation, we propose adaptive KV-cache quantization, a learned policy that assigns bit-width proportional to token importance, minimizing expected memory and latency without sacrificing competitive accuracy. Our framework extracts lightweight token-level features, including token frequency, quality score, attention variance, and entropy-based uncertainty, and feeds them into a compact data-driven controller that dynamically selects KV precision from {2-bit, 4-bit, 8-bit, FP16} during decoding. This adaptive precision policy reduces KV memory footprint and latency while improving accuracy compared to static KV quantization and rule-based baselines, and maintaining competitive accuracy close to FP16 inference across standard LLM benchmarks. Extensive experiments across multiple commonsense reasoning benchmarks using SmolLM-135M, SmolLM-360M, and SmolLM-1.7B demonstrate that our controller consistently improves the accuracy-latency trade-off. For instance, with SmolLM-360M on HellaSwag, our method reduces decoding latency (ms/token) by 17.75% relative to static KV quantization, improves accuracy by 7.60 points, and remains within only 0.30 points of FP16 inference.

Figures & tables

Model#Param#Layers#HeadHidden DimLRBS
SmolLM-135M135M30915363e-31M
SmolLM-360M362M321525603e-31M
SmolLM-1.7B1.71B243281925e-42M
Table 1: Architecture of SmolLM models used in our experiments.
DatasetCategoryTrainValidationTest
HellaSwagCommonsense Reasoning39,90510,04210,003
ARC-CScience Reasoning99,8421,53114,042
OBQAScience QA4,957500500
Table 2: Statistics of language datasets used in our experiments.
Number of Parameters == SmolLM-135M
BaselinesMetricsHellaSwagOBQAARC-C
FP16Accuracy (%)37.2029.0028.20
Latency ( s/tokens/token )3.451.972.57
Static 4-bit KVAccuracy (%)33.6027.6025.00
Latency ( s/tokens/token )2.931.722.25
Dynamic KVAccuracy (%)29.9023.8025.50
Table 3: Comparison of the proposed method with baselines.
Number of Parameters ≈\approx 135M
ModelsHellaSwagOBQAARC-CAverage
OPT-125M29.2016.6018.9021.57
Pythia-70M27.0013.2018.5019.57
Pythia-160M29.3016.0018.1021.13
Galactica-125M29.6028.2026.2028.00
LaMini-GPT-124M30.2029.6026.0028.60
Table 4: Comparison of the proposed method with LLMs.
More in this topic

KV-Cache Quantization

Key-value cache quantization compresses transformer attention state to reduce memory use and bandwidth during long-context inference.

See all 99 papers

Explore similar work

CardsList
Feb 19, 2026

ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

Large Language Models (LLMs) are increasingly deployed in scenarios demanding ultra-long context reasoning, such as agentic workflows and deep research understanding. However, long-context inference is constrained by the KV cache, a transient memory structure that grows linearly with sequence length and batch size, quickly dominating GPU memory usage. Existing memory reduction techniques, including eviction and quantization, often rely on static heuristics and suffer from degraded quality under tight budgets. In this paper, we propose ARKV, a lightweight and adaptive framework that dynamically allocates precision levels to cached tokens based on per-layer attention dynamics and token-level importance. During a short prefill phase, ARKV estimates the original quantization (OQ) ratio of each layer by computing statistical scores such as attention entropy, variance and kurtosis. During decoding, tokens are assigned to one of three states, Original (full precision), Quantization (low precision), or Eviction, according to a fast heavy-hitter scoring strategy. Our experiments on LLaMA3 and Qwen3 models across diverse long- and short-context tasks demonstrate that ARKV preserves ~97% of baseline accuracy on long-context benchmarks while reducing KV memory usage by 4x, with minimal throughput loss. On short-context tasks, ARKV matches full-precision baselines; on GSM8K math reasoning, it significantly outperforms uniform quantization. These results highlight the practical viability of ARKV for scalable LLM deployment, offering fine-grained, data-driven memory control without retraining or architectural modifications. The source code and artifacts can be found in: https://github.com/Large-scale-Sustainable-Computing-LSC/ARKV

Jianlong Lei, Shashikant Ilager

Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands

KV-Cache ManagementLong-Context Language Model InferenceKV-Cache QuantizationKV-Cache Eviction
AskPDF
Feb 26, 2026

InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models

When transformer-based language models are deployed for text generation, most of the inference time is spent in the decoding stage, where output tokens are generated sequentially. Reducing the hardware cost of each decoding step is therefore critical for efficient long-context generation. A major bottleneck is the key-value (KV) cache, whose size grows with sequence length and often dominates the model's memory footprint. Prior work has proposed quantization methods to compress the KV cache while minimizing its loss of precision. We present InnerQ, a hardware-aware KV cache quantization scheme that reduces decode latency without compromising evaluation performance. InnerQ performs group-wise quantization by grouping cache matrices along their inner dimension. This grouping strategy aligns dequantization with vector-matrix multiplication and increases data reuse across GPU compute units. As a result, InnerQ reduces memory access and accelerates dequantization, achieving an average 1.3×1.3\times speedup over prior KV cache quantization methods and 2.7×2.7\times over the non-quantized baseline. To maintain fidelity under aggressive compression, InnerQ incorporates three techniques: (i) hybrid quantization, which chooses symmetric or asymmetric quantization for each group based on local statistics; (ii) high-precision windows for both recent tokens and attention sink tokens to mitigate outlier leakage; and (iii) per-channel normalization of the key cache, computed once during prefill and folded into the model parameters to eliminate runtime overhead. Beyond reducing latency, experiments on Llama and Mistral models show that InnerQ also improves few-shot evaluation scores relative to prior KV cache quantization methods.

Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross

Department of Electrical and Computer Engineering McGill University Montreal, QC, Canada

Hardware-Aware QuantizationKV-Cache QuantizationLLM Inference AccelerationNeural Network Quantization
AskPDF
Apr 21, 2026

SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

KV-cache memory is a major bottleneck in real-world LLM serving, where systems must simultaneously support latency-sensitive small-batch requests and high-throughput concurrent workloads. Although many KV-cache compression methods improve offline accuracy or compression ratio, they often violate practical serving constraints such as paged memory layouts, regular memory access, and fused attention execution, limiting their effectiveness in deployment. In this work, we identify the minimal set of 4-bit KV-cache quantization methods that remain viable under these constraints. Our central finding is that a simple design--token-wise INT4 quantization with block-diagonal Hadamard rotation--consistently achieves the best accuracy-efficiency trade-off. Across multiple models and benchmarks, this approach recovers nearly all of the accuracy lost by naive INT4, while more complex methods such as vector quantization and Hessian-aware quantization provide only marginal additional gains once serving compatibility is taken into account. To make this practical, we implement a fused rotation-quantization kernel that integrates directly into paged KV-cache layouts and introduces zero measurable end-to-end overhead, matching plain INT4 throughput across concurrency levels. Our results show that effective KV-cache compression is fundamentally a systems co-design problem: under real serving constraints, lightweight block-diagonal Hadamard rotation is a viable method that delivers near-lossless accuracy without sacrificing serving efficiency.

Jinda Jia, Jisen Li, Zhongzhu Zhou +8

Together AI

LLM ServingKV CachingMemory-Efficient Optimization4-Bit QuantizationRotation-Based QuantizationLLM Inference Acceleration+2
AskPDF