GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices

Yipin Guo; Arun M George; Siddharth Joshi

cs.AIOct 28, 2026

GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices

Authors: Yipin Guo, Arun M George, Siddharth Joshi

Organizations: University of Notre Dame

Abstract

Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. GAAC adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving. Across five tested models, GAAC achieves 1.405-1.407 $\times$ lossless BF16 KV compression. We also study compression performance on low-precision tensors FP8 E4M3 caches ( 1.09-1.14 $\times$ ) and E5M2 caches (1.25-1.30 $\times$ ). On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27 $\times$ and end-to-end W4A16 decode steps by 1.04-1.15 $\times$ . In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. On Apple M4, GAAC achieves 1.02-1.25 $\times$ attention speedups, demonstrating portability beyond NVIDIA GPUs.

Figures & tables

Figure 1: Overview of GAAC. (a) GAAC exploits the concentration of BF16 exponent values in contiguous high-frequency regions. Each group of eight exponents shares a 2-bit tag indicating 3-bit, 4-bit, or raw mode. The sign and mantissa bits of each element are stored unchanged in a single byte. Each code is an offset from a calibrated per-channel base,

c_{i}=e_{i}-b_{w,i}

. In the illustrated example, one exponent outside the 3-bit window causes the entire group

g{+}1

to use 4-bit mode. Bits per element include the sign-mantissa byte and the mode tag. (b) Compared with standard attention, GAAC reduces KV loading time by reading fewer bytes from the compressed cache, with only a small computational overhead from on-the-fly decompression.

o

denotes the attention output. Example values and timelines are illustrative.

Figure 2: Concentration and contiguity of BF16 KV exponents. Most channels require fewer than eight consecutive exponents to cover 95% of their values.

Figure 3: System design of GAAC. (a) New tokens are appended to an uncompressed BF16 tail until the block is full. (b) The full block is encoded once. If its compressed size

S

does not exceed the slot capacity

C

, it is stored in a compressed slot; otherwise, it remains in BF16 format. (c) The paged KV cache holds compressed and raw finalized blocks in block-table order, followed by the incomplete BF16 tail.

Model	$n_{kv}$	KV exp. entropy (bit)	Ratio	3-bit	4-bit	Raw
Qwen3-4B	8	2.78	1.405 $\times$	89.0%	10.8%	0.2%
Qwen3-8B	8	2.78	1.405 $\times$	88.9%	10.9%	0.2%
Qwen3.8-27B	4	2.69	1.405 $\times$	88.3%	11.5%	0.2%
Qwen3.6-35B-A3B	2	2.71	1.405 $\times$	88.4%	11.4%	0.2%
Llama-3.1-8B	8	2.70	1.407 $\times$	89.9%	9.9%	0.2%

Table 1: Lossless KV compression across models. Entropy denotes the empirical entropy of the BF16 exponent stream. Ratio is the BF16 size divided by encoded size. Mode percentages are measured over 8-element groups.

Figure 4: Compression ratio and codec latency on KV states from one layer of Qwen3.8-27B. Compression latency is measured for a single KV block; decompression latency is measured for a 64K-token cache.

Figure 5: Decode-attention speedup of GAAC using compressed KV over FlashAttention-2 using BF16 KV on DGX Spark, with a batch size of 1 and a 64K-token context.

Figure 6: End-to-end W4A16 decode-step speedup of GAAC over the BF16-KV baseline on DGX Spark at a batch size of 1.

Model	Context	Budget	Batch	tok/s	Batch	tok/s	Gain
			BF16 KV		GAAC
Qwen3-8B	16K	10 GB	4	60.3	5	68.1	+13%
Qwen3-8B	64K	20 GB	2	19.0	3	22.8	+20%
Qwen3-32B	64K	35 GB	2	8.47	3	10.51	+24%
Llama-3.1-8B	64K	18 GB	2	20.0	3	24.9	+25%
Qwen3.8-27B	64K	14 GB	3	23.3	4	29.6	+27%

Table 2: Serving throughput under approximately equal KV-memory budgets.

Figure 7: Decode-attention speedup of GAAC’s Metal implementation over a BF16-KV kernel on Apple M4.

	3-bit groups(%)		4-bit groups(%)		Raw groups(%)		Ratio( $\times$ )
Evaluation task $B$	$A\!\rightarrow\!B$	$B\!\rightarrow\!B$	$A\!\rightarrow\!B$	$B\!\rightarrow\!B$	$A\!\rightarrow\!B$	$B\!\rightarrow\!B$	$A\!\rightarrow\!B$	$B\!\rightarrow\!B$
WikiText-2 / language modeling	88.3	88.3	11.5	11.5	0.20	0.20	1.405	1.405
HumanEval / code	87.1	88.8	12.6	11.0	0.35	0.16	1.402	1.406
GSM8K / math	87.5	89.0	12.3	10.9	0.24	0.14	1.404	1.406
MMLU / knowledge	87.5	87.8	12.2	11.9	0.28	0.28	1.403	1.404
PTB / language modeling	88.3	88.9	11.5	11.0	0.23	0.14	1.405	1.406

Table 3: Cross-task calibration generalization. Paired columns compare calibration on WikiText-2 (

A\!\rightarrow\!B

) with oracle calibration on the target task (

B\!\rightarrow\!B

). The lengths of the calibration prompts are consistent.

Scheme	Ratio	Compress ( $\mu$ s)	Decode ( $\mu$ s)
2+3+4+raw	1.439 $\times$	42.6	526.7
3+4+raw (ours)	1.406 $\times$	42.1	187.1
4+raw	1.304 $\times$	36.7	158.9

Table 4: Precision-mode ablation on Qwen3.8-27B.

Component	FA2	Decomp.-first	GAAC
KV loading	1117.8	1117.8	823.5
Exponent decoding	-	930.8	-
Computation	241.5	241.5	290.7
Measured total	1359.3	2290.1	1114.2

Table 5: Decode-attention time breakdown over a 64K-token context. Times are in

\mu

s. Decomp.-first decompresses the cache before running attention.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

KV Format	Additional Ratio	Total vs. BF16
BF16	1.405–1.406 $\times$	1.405–1.406 $\times$
FP8 E4M3	1.09–1.14 $\times$	2.18–2.28 $\times$
FP8 E5M2	1.25–1.30 $\times$	2.50–2.60 $\times$

Appendix

Table 7: Lossless compressibility after FP8 KV quantization.

Segment	K ( $\leq$ 8K)	K (32K)	V ( $\leq$ 8K)
0–4K	38	40	39
4–8K	37	37	38
8–12K	52	37	39
12–16K	117	37	41
16–20K	177	38	38
20–24K	183	37	40

Appendix

Table 8: Overflow storage in layer 5 of Qwen3-8B over a 32K-token prompt. For each successive 4K-token segment, we report the maximum across heads of the mean overflow bytes per block. Parentheses indicate calibration prompt lengths.

Field	Bytes	Bits / elem.
Sign and mantissa	2048	8.000
3-bit exponent codes	768	3.000
Mode tags (2 bits / group)	64	0.250
Escape bytes (mean)	32	0.125
Block-table entry	4	0.016
Unused escape slot	96	0.375

Appendix

Table 9: Storage breakdown for a compressed 16-token tile covering 128 head dimensions, measured on Qwen3.8-27B.

Method	Latency ( $\mu$ s)	Speedup
FA2 (block 256)	1359.3 $\pm$ 0.8	1.22 $\times$
FlashInfer (page 16)	1343.4 $\pm$ 0.5	1.21 $\times$
FlashInfer (page 256)	1339.8 $\pm$ 0.8	1.20 $\times$
FA3, FA4	not supported on GB10
GAAC	1114.2 $\pm$ 2.1	–

Appendix

Table 10: Decode-attention baseline comparison over a 64K-token context on one Qwen3.8-27B layer. Speedup is the baseline latency divided by GAAC latency.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 271 papers

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM Serving KV-Cache Compression
University of Notre Dame IN, USA
Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache Quantization On-Device Language Model Inference
Clemson University, USA
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
Sep 7, 2026Michael Wang, Keith Li, Roozbeh BostandoostLLM Inference Efficiency LLM Inference
Lake Washington School District · XSchool · UMass Amherst

May 3, 2026J/K move · Enter open · S save

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

Yipin Guo, Siddharth Joshi

University of Notre Dame IN, USA

Disaggregated LLM Serving KV-Cache Compression KV Caching LLM Serving Lossless Data Compression LLM Inference Acceleration+1

Open paper AskPDF arXiv

Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale, especially for long-input and agentic workloads. Existing lossless codecs are unsuitable here as they primarily target offline weight compression, run on CPUs, or use variable-length coding whose compression cannot keep up with KV production during prefill. We introduce SplitZip, a GPU-friendly lossless compressor for KV cache transfer that preserves KV tensors bitwise and integrates into existing serving frameworks without modifying model execution. SplitZip exploits redundancy in floating-point exponents of KV activations, encoding frequent exponent values with fixed-length codes and routing rare exponents through a sparse escape stream of (position, value). A calibrated top-16 exponent codebook eliminates online histogramming, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves

613.3

GB/s compression throughput and

2181.8

GB/s decompression throughput, outperforming prior lossless compressors on the critical codec path. End-to-end transfer experiments show up to

1.32\times

speedup for BF16 KV cache transfer,

1.30\times

speedup for TTFT, and

1.23\times

increase in Request Throughput. The same approach extends to FP8 KV caches, providing up to

1.14\times

compression over native E5M2. Code is available at https://github.com/Intelligent-Microsystems-Lab/SplitZip

Abstract

KV-Cache Compression

Explore similar work

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference