BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

Hangyu Zheng; Zhihao Shu; Kunxiong Zhu; Minghai Qin; Xiaolong Ma; Gagan Agrawal; Wei Niu

cs.AROct 31, 2026

BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs

Authors: Hangyu Zheng, Zhihao Shu, Kunxiong Zhu, Minghai Qin, Xiaolong Ma, Gagan Agrawal, Wei Niu

Organizations: University of Georgia Athens, GA, USA · Western Digital Research San Jose, CA, USA · University of Arizona Tucson, AZ, USA

Abstract

The rapid scaling of large language models (LLMs) has unlocked remarkable capabilities across various domains. However, deploying LLMs on mobile devices remains challenging because of their high computational and memory requirements. Low-bit quantization (e.g., 4-bit weights, 8-bit activations) can reduce model size while preserving accuracy, but its practical efficacy on mobile GPUs has been limited so far. In this paper, we identify that the performance gap stems primarily from frequent data type conversions between quantized and floating-point values, and dynamic and inefficient memory access patterns caused by the growing Key-Value (KV) cache during autoregressive decoding. We introduce BitLLM, a mobile inference engine that addresses type conversion overhead and suboptimal memory accesses in quantized LLMs. Our approach features an adaptive layout and packing strategy for the KV cache that ensures efficient memory access throughout LLM decoding. This is coupled with an integer-native execution pipeline that defers and overlaps type conversion outside the compute hot path, keeping the critical MatMul stages in the integer domain. Implemented on off-the-shelf mobile GPUs, BitLLM outperforms leading engines, i.e., LLAMA.CPP, MNN-LLM, LiteRT-LM, and MLC-LLM, achieving average speedups of 1.78 $\times$ , 1.73 $\times$ , 1.58 $\times$ , and 1.47 $\times$ , respectively, across various models and quantization levels. Under long-context generation, BitLLM reduces average energy consumption (per token) by up to 88.9%.

Figures & tables

Figure 1. Computational graph of an autoregressive transformer decoder. Each new query (

Q

) attends to all cached keys ( K-cache ) and values ( V-cache ), causing the KV cache to grow and dynamically alter tensor shapes. Flow diagram of a decoder layer with Q, K, and V branches; the K-cache and V-cache append steps introduce a context-length dimension that grows with each decoded token.

Figure 2. Overview of quantization techniques. (a) Quantization scheme. (b) Quantization at different granularities. (c) Type conversion pipeline for INT8 computation, involving promotion to INT16/INT32 and dequantization back. Three panels: mapping a floating-point range to INT8, four shading patterns for per-tensor, per-row, per-column, and per-group granularity, and an 8-bit multiply promoted to 16 bits, accumulated in 32 bits, and dequantized.

Figure 3. Illustration of 2D texture memory layout. A 16-element logical array mapped onto GPU 2D texture memory in two different layouts, each drawn as four stacked RGBA channel planes.

Figure 4. Decoding latency and computational workload per token as functions of decode sequence index. Line chart over 10240 decoded tokens: per-token latency rises in stair-steps from about 80 ms to over 900 ms, while GMACs per token increase smoothly and linearly.

Model	Data	Latency per Token		GPU
Model	Type	CPU	GPU	Speedup
TinyLLama-1B ( Zhang et al., 2024b )	FP16	61.9ms	56ms	1.11 $\times$
TinyLLama-1B ( Zhang et al., 2024b )	INT8	31.2ms	67.8ms	0.46 $\times$
Qwen3-1.7B ( Yang et al., 2025 )	FP16	78.8ms	73.6ms	1.07 $\times$
Qwen3-1.7B ( Yang et al., 2025 )	INT8	45.4ms	130.2ms	0.35 $\times$
Llama-3.2-3B ( Grattafiori et al., 2024 )	FP16	140.5ms	137.4ms	1.02 $\times$

Table 1. Comparison of LLM decoding latency on mobile platforms with FP16 versus low-bit representation.

Figure 5. Execution pipelines for a length-8 dot product with INT4, INT8, and FP32 formats. FP32 uses direct fused multiply-add ( fma ); INT8 performs integer mul with 32-bit accumulation ; INT4 additionally unpacks sub-byte weights with shift and mask instructions. Three panels for INT4, INT8, and FP32 dot products; INT4 adds a shift-and-mask unpack step, INT8 multiplies and accumulates in 32 bits, and FP32 uses a single fused multiply-add.

Figure 6. Layout optimization pipeline. Four-stage flowchart: a 3D tensor is permuted into candidate layouts, packed into a 2D texture, and its elements reordered into an interleaved quantization packing.

Figure 7. Two folding examples. Two matrix multiplication diagrams showing the K transpose operand folded along T and the V operand folded along D into narrower stacked tiles.

Figure 8. Different quantization packing schemes. Three byte-index grids with their unpacking formulas: contiguous packing needs 31 shifts and 31 masks, while lane-interleaved and block-swizzled packing need 1 shift and 2 masks.

Platform	Instruction
Adreno/MALI GPU	udot8_acc / dot8_acc
AMD GPU (GCN/RDNA)	v_dot4_u32_u8 / v_dot4_i32_i8
Intel GPU (Xe)	dpas.s32.s8.s8
NVIDIA GPU	dp4a.s32.s8.s8

Table 2. 8-bit dot-product primitives on different platforms.

Figure 9. Tile-based pipelining for type conversion-hiding computation. The pipeline consists of three stages: (1) Load & Unpack , (2) Integer Compute , and (3) Deferred Conversion . Operations overlap across tiles, ensuring the hot path remains fully in the integer domain. Three pipeline stages, load and unpack, dp4a dot-product, and dequantize, with a timeline showing the stages of consecutive tiles overlapping in time.

Model	#Params	Perplexity $\downarrow$
Model	#Params	FP16	W8A8	W4A8
Llama-3.2-1B ( Grattafiori et al., 2024 )	1.24B	13.75	14.31	15.34
Qwen3-1.7B ( Yang et al., 2025 )	1.72B	9.84	9.97	10.28
Gemma-2-2B ( Team et al., 2024 )	2.61B	18.09	20.33	21.40
Phi-2-2B ( Javaheripi et al., 2023 )	2.78B	17.41	18.15	19.94

Table 3. Model perplexity on the WikiText2 ( Merity et al., 2016 ) dataset (lower is better) for different precisions.

Figure 10. Average latency per token comparison across models, precisions, and Prompt–Output (

P

–

O

) window. Grouped bar charts of per-token latency for four models at FP16, W8A8, and W4A8 under three prompt-output settings; the Ours bar is the shortest in nearly every group.

Figure 11. Per-token decoding latency as sequence length grows (up to

10K

tokens). MLC-LLM does not support W8A8 or long contexts at W4A8. Our method (red) achieves the lowest latency and reduces the stair-step spikes observed in baselines. Line charts of per-token latency versus sequence length at W8A8 and W4A8; GPU LLAMA.CPP shows large stair-step jumps while Ours stays lowest and nearly flat.

Figure 12. Normalized (by ours) per-token and average energy consumption for low-bit precisions. Energy per token versus sequence length and average energy bars for LLaMA-3.2-1B at W8A8 and W4A8, normalized to Ours; baselines use roughly 2 to 4 times more energy on average.

Figure 13. Optimization breakdown on mobile GPU. (a) Cumulative speedup over OurB (baseline = 1) as each optimization is enabled. LT , PK , and TCH denote adaptive layout, quantization-aware packing, and type-conversion hiding, respectively. (b) Fine-grained breakdown of the incremental speedup gains from PK and TCH over OurB . The PK gain is separated into hardware-aligned packing ( PK-HAP ) and reduced INT4 unpacking ( PK-Unpack ). Bar charts of cumulative speedup as layout, packing, and conversion-hiding optimizations are enabled, and of the percentage gain contributed by packing and conversion hiding.

Figure 14. Memory, cache, and ALU utilization analysis. Three bar charts comparing memory bandwidth, L1 and L2 cache hit rate, and ALU utilization across engines; Ours is highest in each metric.

Figure 15. LEP calibration and selection analysis. (a) Impact of different LEP weights $(\alpha_{1},\alpha_{2})$ . (b) LEP search overhead and performance gap to oracle across mobile GPUs.

Figure 16. Portability evaluation. Empty bars indicate that the framework does not support the configuration due to missing operator support or insufficient device memory. Bar charts of average latency on a Mali-G715 and an Adreno 740 GPU for two models at three precisions; Ours is the lowest bar in each group.

On-Device Language Model Inference

On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.

See all 177 papers

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache Quantization On-Device Language Model Inference
Clemson University, USA
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
Aug 21, 2026Luka Ribar, Jeevan Bhoot, Douglas OrrVLM Quantization Efficient VLM Inference
Graphcore Research · Arm
SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
Sep 18, 2026Harish KB, Jagadeeswaran M, Pradheep P +2LLM Inference LLM Inference Acceleration
Computer Science and Engineering Vellore Institute of Technology Vellore, India

Apr 6, 2026J/K move · Enter open · S save

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods, Gabriel Hillesheim+1

Clemson University, USA

KV-Cache Quantization On-Device Language Model Inference LLM Inference Acceleration Mixed-Precision Quantization

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable progress across reasoning, generation, and decision-making tasks, yet deploying them on mobile, embedded, and edge devices remains particularly challenging. On-device LLM inference is heavily constrained by the memory and bandwidth overhead of the key-value (KV) cache, which grows linearly with context length and often dominates decoding cost. Existing KV-cache quantization schemes typically rely on fixed precision or hand-crafted heuristics, thereby wasting bits on low-impact tokens while over-compressing informative ones, leading to avoidable accuracy degradation. Inspired by Huffman coding's principle of variable-length allocation, we propose adaptive KV-cache quantization, a learned policy that assigns bit-width proportional to token importance, minimizing expected memory and latency without sacrificing competitive accuracy. Our framework extracts lightweight token-level features, including token frequency, quality score, attention variance, and entropy-based uncertainty, and feeds them into a compact data-driven controller that dynamically selects KV precision from {2-bit, 4-bit, 8-bit, FP16} during decoding. This adaptive precision policy reduces KV memory footprint and latency while improving accuracy compared to static KV quantization and rule-based baselines, and maintaining competitive accuracy close to FP16 inference across standard LLM benchmarks. Extensive experiments across multiple commonsense reasoning benchmarks using SmolLM-135M, SmolLM-360M, and SmolLM-1.7B demonstrate that our controller consistently improves the accuracy-latency trade-off. For instance, with SmolLM-360M on HellaSwag, our method reduces decoding latency (ms/token) by 17.75% relative to static KV quantization, improves accuracy by 7.60 points, and remains within only 0.30 points of FP16 inference.

Abstract

On-Device Language Model Inference

Explore similar work

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs

SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference