cs.DCMar 18, 2026

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

Authors: Ruibo Fan, Xiangrui Yu, Xinglin Pan, Zeyu Li, Weile Luo, Qiang Wang, Wei Wang, Xiaowen Chu

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Lossless model compression holds tremendous promise for alleviating the memory and bandwidth bottlenecks in bit-exact Large Language Model (LLM) serving. However, existing approaches often result in substantial inference slowdowns due to fundamental design mismatches with GPU architectures: at the kernel level, variable-length bitstreams produced by traditional entropy codecs break SIMT parallelism; at the system level, decoupled pipelines lead to redundant memory traffic. We present ZipServ, a lossless compression framework co-designed for efficient LLM inference. ZipServ introduces Tensor-Core-Aware Triple Bitmap Encoding (TCA-TBE), a novel fixed-length format that enables constant-time, parallel decoding, together with a fused decompression-GEMM (ZipGEMM) kernel that decompresses weights on-the-fly directly into Tensor Core registers. This "load-compressed, compute-decompressed" design eliminates intermediate buffers and maximizes compute intensity. Experiments show that ZipServ reduces the model size by up to 30%, achieves up to 2.21x kernel-level speedup over NVIDIA's cuBLAS, and expedites end-to-end inference by an average of 1.22x over vLLM. ZipServ is the first lossless compression system that provides both storage savings and substantial acceleration for LLM inference on GPUs.

Figures & tables

Figure 1. Execution time of lossless compression pipelines on NVIDIA L40S GPU with GateUp_Proj layers.
Figure 2. Exponent bit distribution in LLM weights.
Figure 3. Existing Lossless Compression for BF16 Weights. Illustrated with Huffman Encoding.
Figure 4. Existing lossless compression inference pipeline.
Figure 5. Roofline analysis.
Figure 6. Overview of ZipServ . ZipServ features an offline lossless compressor (left) and an online inference engine (right).
Figure 7. Tensor-Core-Aware Triple Bitmap Encoding. The 4×44\times 4 FragTile shown is illustrative; the actual size is 8×88\times 8 .
Figure 8. Data movement and instruction pipeline.
Figure 9. The Decompressor Design.
Figure 10. Hierarchical software pipeline design.
Figure 11. Kernel performance comparison on NVIDIA RTX4090 and L40S GPUs.
Figure 12. Micro-level kernel performance analysis.
Figure 13. Standalone decompression kernel comparison.
Figure 14. Cross-generation performance comparison.
Figure 15. ZipServ performance under different N settings.
Figure 16. End-to-end performance comparison.
Figure 17. Breakdown of end-to-end inference time and memory consumption.
Figure 18. Performance on training-oriented GPUs.
More in this topic

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 118 papers

Explore similar work

CardsList

  1. SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

    May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM ServingKV-Cache Compression

    University of Notre Dame IN, USA

  2. HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

    Sep 28, 2026Thai Nguyen, Khang Tran, NhatHai PhanDiffusion Language Model InferenceDiffusion Language Models

    New Jersey Institute of Technology

  3. Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended

    May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Efficient Neural Network InferenceGPU Kernel Optimization

    University of Washington Seattle, WA, USA · KAUST Thuwal, Saudi Arabia