EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model
Authors: S M Mahbub Murshed, Hayden Bright, Rezaul Karim, Ali Shiri Sichani
Organizations: Causal Dynamics Lab · Electrical Engineering and Computer Science Department, University of Missouri-Columbia · Computer Science & Engineering, University of Chittagong
Deploying language models (LMs) on the edge presents unique memory, latency, and power challenges absent in high-performance computing. KV-cache compression emerges as a promising approach to address these challenges. However, the effectiveness of current KV-cache compression methods has not been thoroughly examined on edge devices. Moreover, we show that existing evaluation methods do not always involve the KV-cache under test. We propose EdgeLLM-Nano, a measurement study of KV-cache compression using a nano-scale decoder-only LM (transformer) testbed that has 8.39M parameters, i.e., 33.6 MB, to enable end-to-end examination of each backend, seed, and power sample. Our results give a negative verdict for a polar-guided key-cache quantizer (round-to-nearest (RTN) control causes an order of magnitude less damage), as well as a consistency-anchored progressive weight pruner (one-shot magnitude-based pruning performs as well as, if not better). We also introduce Long-Context Decode Fidelity (LCDF) evaluation, a token-by-token examination of teacher-forced log-losses using the KV-cache under test. Unlike perplexity that gives the same 5.461 score to all methods, LCDF distinguishes different KV-cache compression backends thanks to its paired statistics across seeds and effect sizes. Our companion Raspberry Pi 5 measurement shows all compression backends we tested to be less energy-efficient per token compared to uncompressed FP16 inference. Except for KIVI that is close to lossless in terms of LCDF, our findings indicate KV-cache compression can be improved. LCDF offers a simple and effective way of shedding light on KV-cache compression methods, whose negative impacts have been overlooked by perplexity.
Figures & tables
Method
Venue
Lane
Core mechanism
Bits/elt
Cache eval
LCDF verdict
KIVI [ 3 ]
ICML’24
K + V
Per-channel K, per-token V, tuning-free
2.99 / 3.79 †
Yes
Near-free: +0.0053 ; +0.0059†
KVQuant [ 8 ] ‡
NeurIPS’24
K + V
Calib. per-channel + outlier fallback
3.50
Yes
Small, significant: +0.0254
QuaRot [ 9 ]
NeurIPS’24
K + V
Global Hadamard + RTN
4.00
Yes
Small, significant: +0.0117
SnapKV [ 10 ]
NeurIPS’24
K + V
Attention-guided eviction
16, no eviction
Yes
Jitter below 10−4 ; wrapper overhead
QJL [ 11 ]
AAAI’25
K
1-bit JL sign projection
PGKQ component
Yes
Never earns its +1.25 bits here
RotateKV [ 2 ]
IJCAI’25
K + V
Channel reorder + FWHT
—
Yes
Not measured; shares PGKQ steps 1–2
TABLE I: Method positioning and LCDF verdicts on this testbed
Fig. 1: PGKQ’s forward compression path. The first four stages form the main polar code. The optional QJL stage stores a sign-projected residual and its norm.
Model
Tokenizer
Vocab
Params
Steps
Val ppl
NanoGPT-256 char
byte
256
6.36 M
20 000
1.846
NanoGPT-256 BPE
SP-BPE
8 192
8.39 M ∗
20 000
5.117
GPT-2-124M scale
tiktoken-gpt2
50 257
∼ 124 M
2 000
8.225
TABLE II: Base-model training summary
Fig. 2: GATE-M1 Pareto sweep over 30 TinyStories sequences at ctx =512 . This single-dataset result is not comparable to Table IV . The axes show lane-averaged cache-hook bits per element and log-scale LCDF Δ . KIVI lies inside the pass region; the filled star uses our port and the open diamond uses the reference policy. Every PGKQ-KV variant lies outside. The K-only point at 9.86 bits and 0.480 averages the 3.75-bit K lane with FP16 V.
Backend
TPS mean [range]
p99 ms mean [range]
KV bytes
fw-default, native
265.94 [265.49, 266.48]
3.91 [3.87, 3.96]
N/A
FP16
232.16 [231.97, 232.45]
4.46 [4.46, 4.47]
786 432
KIVI
197.80 [196.87, 198.86]
5.23 [5.17, 5.26]
147 456
KVQuant-style
187.14 [186.89, 187.43]
5.54 [5.51, 5.56]
172 032
QuaRot
94.97 [94.89, 95.01]
10.81 [10.77, 10.84]
196 608
SnapKV
219.33 [218.41, 220.03]
4.73 [4.70, 4.76]
786 432
TABLE III: RTX 5090 backend comparison
Backend
Mean Δ [range]
Sig.
δ
124M, sig.
KIVI
0.0053 [ − 0.0032,0.0239]
3/6
[ − 0.05,0.60]
− 0.001; 3/6
KVQuant-style
0.0254 [0.0084,0.0478]
6/6
[0.03,1.00]
− 0.006; 5/6
QuaRot
0.0117 [ − 0.0128,0.0333]
6/6
[ − 0.38,1.00]
− 0.009; 5/6
SnapKV
∣Δ∣<10−4
2/6
[ − 0.03,0.33]
<10−4 ; 2/6
PGKQ K
0.1908 [0.0795,0.2806]
6/6
[0.17,1.00]
0.037; 3/6
PGKQ-KV k3v3
0.3190 [0.1982,0.5463]
6/6
[0.38,1.00]
0.112; 6/6
TABLE IV: Mean LCDF Δ against FP16 at ctx =512
Fig. 3: Raspberry Pi 5 energy efficiency over ten seeds, 30 runs per seed, and six datasets. Bars show mean tokens per joule; whiskers give dataset ranges, and the dashed line marks FP16. Every compressed-cache backend falls below FP16 under our ports.
TinyStories ppl
Target
Recov.
PSPCA
W
M
SG rand / real
Rand.
0.30
4.1×10−4
5.60 ± 0.11
5.65 ± 0.06
5.39
5.49 / 5.36
56.5
0.50
4.2×10−3
7.08 ± 0.17
7.06 ± 0.13
7.19
7.70 / 5.84
5.0×103
0.70
2.1×10−2
19.77 ± 0.53
24.22 ± 1.06
16.90
26.51 / 10.62
1.4×108
TABLE V: PSPCA versus one-shot pruning at matched sparsity
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory +2