ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs

Jinwu Yang; Jiaan Wu; Zedong Liu; Xinyang Ma; Hairui Zhao; Yida Gu; Yuanhong Huang; Xingchen Liu

cs.ARMar 28, 2026

ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs

Authors: Jinwu Yang, Jiaan Wu, Zedong Liu, Xinyang Ma, Hairui Zhao, Yida Gu, Yuanhong Huang, Xingchen Liu,

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Huawei Technologies Co., Ltd., Shenzhen, Guangdong, China

Abstract

The rapid scaling of Large Language Models presents significant challenges for their deployment and inference, particularly on resource-constrained specialized AI hardware accelerators such as Huawei's Ascend NPUs, where weight data transfer has become a critical performance bottleneck. While lossless compression can preserve model accuracy and reduce data volume, existing lossless compression algorithms exhibit extremely low throughput when ported to the Ascend NPU architecture. In this paper, we propose ENEC, a novel lossless compression method specifically customized for AI model weights and optimized for Ascend Neural Processing Units. ENEC adopts a block-based fixed-length encoding scheme and incorporates a series of NPU-specific optimizations: bit-width quantization with hierarchical halving bit-packing, vectorized branch-free integer transformation, and dependency-decoupled intra-segment scan for efficient prefix-sum computation. Experimental results demonstrate that ENEC outperforms existing state-of-the-art NPU compressors in both compression ratio and throughput. Compared to leading GPU solutions, ENEC achieves a 3.43X higher throughput than DietGPU and a 1.12X better compression ratio than nvCOMP. By reducing weight transmission overhead, ENEC significantly improves end-to-end inference performance, achieving up to a 6.3X speedup. On Ascend NPUs, ENEC is the first open-source lossless compression algorithm for model weights that achieves performance comparable to state-of-the-art GPU compressors, offering an effective solution for deploying large-scale AI models.

Figures & tables

Fig. 1 : Time breakdown of Qwen3-32B Inference on NPU 910B2 (left) and throughput of ANS across multiple platforms (right).

Fig. 2 : DaVinci architecture (decoupled AIC and AIV).

LC-Alg	Models
LC-Alg	deepseek-llm-7b-base [ 12 ]	Qwen3-8b [ 49 ]	Llama-3.1-8B-Instruct [ 17 ]
HCLOG	98.23%	98.30%	99.36%
Others	1.77%	1.70%	0.64%

TABLE I : LC search results. “Others” represents the set of all non-HCLOG algorithmic variants within the LC framework.

Fig. 3 : Linear relationship between exponent values and frequency rankings in model weights. The red circle indicates outliers.

Fig. 4 : Execution challenges and constraints on Ascend NPUs.

Fig. 5 : Overview of optimized ENEC compression design on Ascend NPUs.

Fig. 6 : Compressed stream layout. The bit mask is used to distinguish anomalous groups within each data block. Prefix sums provide direct starting positions for each thread’s compressed data.

Fig. 7 : Vectorized branch-free integer transformation.

Fig. 8 : Prefix sum process. This figure illustrates the step-by-step implementation of the prefix sum algorithm, using an $8\times 8$ binary local tensor $M$ as a working example.

Arch	Compressors	BF16					FP16		FP32
Arch	Compressors	Falcon	Qwen3-8B	DeepSeek	Qwen3-32B	Llama	Mistral	Diffusion	OLMo	BERT	Wav2Vec
CPU	ZipNN	1.51	1.50	1.51	1.50	1.51	1.19	1.18	1.20	1.20	1.21
GPU	NV_Zstd	1.28	1.27	1.28	1.27	1.29	1.09	1.08	1.08	1.08	1.08
	NV_Deflate	1.28	1.27	1.28	1.27	1.29	1.09	1.08	1.08	1.08	1.08
	NV_GDeflate	1.27	1.27	1.27	1.27	1.28	1.09	1.08	1.07	1.08	1.08
	NV_ANS	1.25	1.24	1.25	1.25	1.27	1.08	1.04	1.06	1.04	1.03

TABLE II : Compression ratio results on different datasets.

Type	Models	Size (GB)	Layer Size (MB)
BF16	Falcon-7B [ 45 ]	14.40	39.38 $\sim$ 563.56
	Qwen3-8B [ 49 ]	16.38	8.00 $\sim$ 1187.00
	deepseek-llm-7b-base [ 12 ]	13.82	32.00 $\sim$ 800.00
	Qwen3-32B [ 49 ]	65.60	10.00 $\sim$ 1483.75
	Llama-3.1-8B-Instruct [ 17 ]	16.06	8.00 $\sim$ 1002.00
FP16	CapybaraHermes-2.5-Mistral-7B [ 29 ]	14.50	8.00 $\sim$ 250.02

TABLE III : Popular models’ weights data used in this work.

Fig. 9 : Throughput of compression (upper) and decompression (lower) across different datasets and methods. The Y-axis (Throughput) is on a logarithmic scale to visualize performance differences spanning several orders of magnitude.

Fig. 10 : End-to-end inference performance comparison of ENEC across different models and batch sizes. Latency metrics include Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT).

BF16			FP16 & FP32
Type	Models	(b, n, m, L)	Type	Models	(b, n, m, L)
BF16	Falcon	(122, 6, 3, 16)	FP16	Mistral	(7, 4, 3, 16)
BF16	Qwen3-8B	(123, 6, 3, 16)	FP16	Diffusion	(11, 5, 3, 16)
BF16	DeepSeek	(123, 6, 3, 16)	FP32	OLMo	(121, 6, 3, 16)
BF16	Qwen3-32B	(122, 6, 3, 16)	FP32	BERT	(123, 6, 3, 16)
BF16	Llama	(121, 6, 3, 16)	FP32	Wav2Vec	(125, 6, 3, 16)

TABLE IV : ENEC compression parameters adopted by the majority of tensors in the models.

Fig. 11 : Performance of several common operations under fixed input conditions across different data block sizes.

Models	Compression Ratio	Comp. / Decomp. Thr. (GB/s)
Falcon-7B [ 45 ]	1.34 (Optimal: 1.34, 0%↓)	367 / 213 (Optimal: 359 / 212)
Qwen3-8B [ 49 ]	1.35 (Optimal: 1.35, 0%↓)	375.2 / 220.1 (Optimal: 377 / 216)
Qwen3-32B [ 49 ]	1.34 (Optimal: 1.34, 0%↓)	351 / 209 (Optimal: 361 / 214)
Llama-3.1-8B-Instruct [ 17 ]	1.31 (Optimal: 1.36, 5%↓)	338 / 199 (Optimal: 373 / 216)

TABLE V : Compression performance on other models using the parameters searched from DeepSeek-V3. “Optimal” is the performance achieved using the parameters searched on the target model.

Arch	Compressors	1	2	4	8	16	32	64	128	256	512
CPU	ZipNN	1.51	1.51	1.51	1.51	1.51	1.51	1.51	1.50	1.50	1.50
GPU	NV_Zstd	1.28	1.28	1.28	1.28	1.28	1.28	1.28	1.27	1.27	1.28
	NV_Deflate	1.28	1.28	1.28	1.28	1.28	1.28	1.28	1.27	1.27	1.28
	NV_GDeflate	1.27	1.27	1.27	1.27	1.27	1.27	1.27	1.27	1.27	1.27
	NV_ANS	1.20	1.20	1.20	1.20	1.20	1.23	1.25	1.26	1.26	1.26
	NV_Bitcomp	1.33	1.33	1.33	1.33	1.33	1.33	1.33	1.36	1.36	1.36

TABLE VI : Compression ratios with different input file sizes (MB).

Fig. 12 : Comparison of throughput performance across various methods under different input file sizes.

Fig. 13 : Performance of several ENEC versions.

Hardware	Compressors	CR	Comp. / Decomp. Thr. (GB/s)
CPU	ZipNN	1.50	0.4 / 1.0
CPU	ENEC-CPU	1.35	3.91 / 1.56
GPU	NV-ANS	1.24	139.7 / 176.7
	NV-Bitcomp	1.32	291.8 / 155.2
	Diet-ANS	1.23	154.4 / 194.5
	Diet-Float	1.47	271.9 / 271.3

TABLE VII : Performance comparison of ENEC and its implementations on other platforms using Qwen-32B.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 216 papers

NeuralZip: Reusable Setup for Fast Lossless Compression
Oct 7, 2026Martín Bravo, Samuel Horváth, Gonzalo Navarro +1Lossless Data Compression Neural Network Compression
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Computer Science, University of Chile, Santiago, Chile · National Center for Artificial Intelligence (CENIA), Santiago, Chile
EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates
Sep 28, 2026Hong Zhang, Zhongjie Duan, Yingda ChenEfficient Neural Network Inference Weight-Only Quantization
Alibaba Inc.
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
May 29, 2026Aditya K Kamath, Arvind Krishnamurthy, Marco Canini +1Efficient Neural Network Inference GPU Kernel Optimization
University of Washington Seattle, WA, USA · KAUST Thuwal, Saudi Arabia

Oct 7, 2026J/K move · Enter open · S save

NeuralZip: Reusable Setup for Fast Lossless Compression

Martín Bravo, Samuel Horváth, Gonzalo Navarro, Andrés Abeliuk

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Computer Science, University of Chile, Santiago, Chile · National Center for Artificial Intelligence (CENIA), Santiago, Chile

Lossless Data Compression Neural Network Compression Entropy Coding

Open paper AskPDF arXiv

Lossless compression can reduce the storage and movement of model weights without changing their floating-point values, but repeated statistical analysis and code construction add computational overhead. We study whether the statistical structure of exponents can be prepared once and reused. For this, we introduce NeuralZip, which groups chunks with similar exponent distributions, shares Huffman codes, and selectively represents recurring exponent tuples using packed exponents, thereby achieving additional moderate compression ratios. A setup chooses these representations before subsequent encodings, while every encoding still processes the current tensor values. In floating-point model checkpoints, post-setup compression is 1.81-21.33

\times

faster than the baselines and achieves exact bit-to-bit reconstruction. We show that this setup can be precomputed and transferred from another compatible architecture, preserving similar compression ratios and avoiding the need to amortize setup costs. Therefore, compression adaptation is transferable and reusable. Training checkpoints demonstrate continued reuse as the weights evolve. Finally, GPU experiments reduce active memory usage by up to 27.5

\%

while reproducing the logits exactly.

Abstract

Neural Network Compression

Explore similar work

NeuralZip: Reusable Setup for Fast Lossless Compression

EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates

Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended