cs.LGNov 12, 2026

LazyWeights: Decode-on-Demand Edge LLMs

Authors: Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali

Organizations: The University of Texas at Austin · Case Western Reserve University

Abstract

Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, llama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best llama.cpp format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU llama.cpp, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

Figures & tables

Explore similar work

CardsList
  1. NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching

    Aug 23, 2026Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4Parallel LLM InferenceLLM Inference

  2. Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

    Oct 1, 2026JuneHyung Kim, Sankeerth Durvasula, Nandita VijaykumarLLM InferenceLLM Inference Acceleration

  3. Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs

    Apr 6, 2026Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Patrick Woods +2KV-Cache QuantizationOn-Device Language Model Inference