LazyWeights: Decode-on-Demand Edge LLMs
Organizations: The University of Texas at Austin · Case Western Reserve University
Abstract
Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, llama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best llama.cpp format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU llama.cpp, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).
Figures & tables
| Jetson Orin Nano Super | Jetson Nano | |
| Memory / bandwidth | 8 GB, 102 GB/s | 4 GB, 22.5 GB/s (meas.) |
| GPU | Ampere, 1024 cores | Maxwell, 128 cores |
| int8 dp4a / toolchain | yes / CUDA 12.6 | no / CUDA 10.2 |
| DRAM use, packed | 40–64% | 34–35% |
| Mistral-7B fits | yes | no (streamed, § IV-E ) |
| Bound by | memory bandwidth | instructions + capacity |
| Change | Measured on | Effect |
|---|---|---|
| Speculative attention, | SmolLM 8-token pass, depth 400 | 44.4 31.2 ms |
| GGUF-width twins | Phi-3 8-token pass, depth 900 | 77.5 52.5 ms |
| Nano kernels | SmolLM 1% twin, greedy token | 474 152 ms |
| SmolLM 3.5% | 282 134 ms | |
| Phi-3 | 585 261 ms | |
| Ordered reduction | Nano, 2-token coded pass | 418 301 ms |
| Task accuracy | Orin, greedy | Orin, speculation | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Budget | Format | b/w | PPL | HellaSwag | ARC-e | PIQA | GSM8K | Saving | ms / mJ | ms / mJ | ||||
| Phi-3-mini | 1% | IQ4_XS+im | 4.311 | +0.65% | .783 | .807 | .798 | .785 | 35.0 / 691 | 27.4 / 495 | |||||
| LazyWeights | 4.046 | +0.11% | .778 | .803 | .789 | .773 | 6.2% | 31.4 / 627 | 15/ 10% | 9% | 17.9 / 366 | 39/ 35% | 26% | ||
| 3.5% | Q3_K_M+im | 4.108 | +2.91% | .776 | .786 | .799 | .779 | 44.0 / 840 | 27.6 / 512 | ||||||
| LazyWeights | 3.649 | +2.97% | .774 | .811 | .796 | .770 | 11.2% | 31.4 / 616 | 37/ 29% | 27% | 18.3 / 370 | 42/ 34% | 28% | ||
| Mistral-7B | 1% | Q5_K_M | 5.668 | +0.44% | .842 | .716 | .768 | .425 | 83.3 / 1784 | 61.6 / 1276 | |||||
| CPU ( llama.cpp ) | GPU ( LazyWeights runtime) | ||||
| Model | Budget | Baseline | LazyWeights GGUF | Software | + unit |
| Greedy | |||||
| Phi-3 | 1% | 657 / 4.48 | 654 / 4.40 | 591 / 4.24 | 261 / 2.05 |
| 3.5% | 736 / 4.80 | 663 / 4.45 | 590 / 4.46 | 263 / 2.08 | |
| SmolLM | 1% | 366 / 2.47 | 364 / 2.21 | 484 / 3.22 | 152 / 1.17 |
| 3.5% | 317 / 2.12 | 317 / 2.03 | 289 / 2.10 | 134 / 0.97 | |
| ASAP7 | Nangate45 | |||||
| Block | m 2 | mW | slack | m 2 | mW | slack |
| Comparator lane | 0 1,566 | 0 3.59 | +117 ps | 23,462 | 25.2 | +0.5 ns |
| Table lane (2048 entries) | 14,076 | 94.4 | 31 ns | – | – | – |
| Arbiter, 32 lanes | 0 1,443 | 0 0.83 | met † | 18,639 | 4.25 | +1.5 ns |
| Arbiter, 64 lanes | 9,174 | – | 0.29 ns | – | – | – |
| Engine | 0 51,550 | 0 115.7 | – | – | – | – |
| Model | Budget | Full | =0 | RTN |
|---|---|---|---|---|
| Phi-3 | 3.5% | 3.649, +2.97 | 4.046, +0.11 | 3.646, +12.07 |
| Mistral | 1% | 3.886, +0.94 | 4.241, +0.32 | 3.886, 0.04 |
| 3.5% | 3.422, +1.46 | 3.837, +2.61 | 3.422, +2.78 | |
| SmolLM | 1% | 5.162, +0.88 | 5.621, +0.31 | 5.162, +0.57 |
| 3.5% | 3.934, +3.27 | 4.321, +2.24 | 3.934, +6.14 |
On-Device Language Model Inference
On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.
See all 189 papers