HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

Thai Nguyen; Khang Tran; NhatHai Phan

cs.AISep 28, 2026

HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

Authors: Thai Nguyen, Khang Tran, NhatHai Phan

Organizations: New Jersey Institute of Technology

Abstract

Large language models (LLMs) have shown strong potential for lossless data compression, but existing approaches are constrained by the high computational cost and low throughput of autoregressive decoding. We propose HyperZip, an efficient and scalable LLM-based compression framework that leverages diffusion-based LLMs (dLLMs) with Multi-Token Prediction (MTP) to accelerate LLM-based data compression processes. We identify a trade-off in diffusion-based compression, where increasing decoding throughput degrades the compression rate. To mitigate this trade-off, HyperZip employs a hypernetwork to generate data-specific updates from a context representation, adapting the dLLM to the target data without costly fine-tuning, resulting in a low compression rate and high throughput. Extensive experiments show that HyperZip achieves a superior trade-off between compression rate and speed compared with state-of-the-art baselines.

Figures & tables

Figure 1 : Compression–throughput trade-off across MTP mechanisms.

Panel of Figure 3 — Figure 3 : Compression rate and throughput of HyperZip vs considered baselines.

Panel of Figure 4 — Figure 4 : t-SNE visualization.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : Compression–throughput trade-off across inference methods and configurations. Lower compressed size and higher throughput are preferred. Each curve connects configurations from the corresponding hyperparameter sweep. The throughput axis uses a base-2 logarithmic scale. Detail of how these methods custom speed present in Baseline

Backbone	Parameters	Layers	Hidden size	Vocabulary
BD3LM-135M	135M	30	576	49152
BD3LM-0.5B	0.5B	24	896	151936
Fast-dLLM-V2-1.5B	1.5B	28	1536	151936

Appendix

Table 2: Diffusion-language-model configurations used in our experiments. Replace the placeholders with the exact checkpoint information.

Component	Configuration
Document encoder	GTE-Qwen2-1.5B-Instruct
Document-embedding dimension	1536
Encoder input length	32,000 tokens
Module-embedding dimension $d_{m}$	32
Layer-embedding dimension $d_{\ell}$	32
LoRA rank $r$	8

Appendix

Table 3: Hypernetwork configuration.

Model	$\gamma$	CR incl. update vector ( $\downarrow$ )				Throughput (tok/s) ( $\uparrow$ )
Model	$\gamma$	Scientific	Literature	Programming	General Web	Throughput (tok/s) ( $\uparrow$ )
Fast-dLLM	0.5	16.1	15.7	18.0	16.3	141.0
FineZip	–	9.23	8.46	9.24	8.22	12.6
Fine-tuning	0.5	15.9	15.4	17.7	16.1	140.2
HyperZip (ours)	0.5	15.6	15.3	16.3	15.8	138.9

Appendix

Table 4 : Compression ratio across document topics and compression throughput. Lower compression ratio and higher throughput are preferred.

Explore similar work

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression
Aug 4, 2026Angelo Nardone, Paolo FerraginaLarge Language Model Compression Data Compression Methods
University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy
ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Apr 30, 2026Wenxiang Lin, Xinglin Pan, Ruibo Fan +2Large Language Model Compression Large Language Model Training
Harbin Institute of Technology, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), China · The Hong Kong University of Science and Technology, Hong Kong SAR
LLM-based Source Code Compression via Thresholded Symbol Ranking
Jul 27, 2026Angelo Nardone, Paolo FerraginaLarge Language Model Compression Task-Aware Compression
University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy

Aug 4, 2026, cs.CLJ/K move · Enter open · S save

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

Angelo Nardone, Paolo Ferragina

University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy

Large Language Model Compression Data Compression Methods Diffusion Language Models Autoregressive Language Models Diffusion Models Structured Context

Open paper AskPDF arXiv

We study the problem of lossless text compression, motivated by the rapid growth in the collection and storage of digital textual data - including plain text, source code, and structured formats such as XML - and by recent advances in neural language model-based compression. In particular, recent LLM-based approaches, whether built on symbol-ranking pipelines or paired with a statistical compressor, have demonstrated compression ratios significantly superior to general-purpose compressors such as zstd, gzip, or bzip on text and code. However, these neural approaches suffer from severe throughput limitations, making them not yet practically usable. For the first time in the context of lossless neural text compression, we introduce Diffusion Language Models (DLMs) as an alternative inference paradigm to autoregressive LLM-based approaches. We argue that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation. However, achieving these improvements requires addressing algorithmic challenges introduced by applying DLMs to lossless compression, where the architecture allows the number and positions of symbols encoded at each forward pass to be decided independently. We design efficient and effective strategies to solve these challenges and evaluate them experimentally against LLM-based and general-purpose compressors on enwik8, a well-established textual benchmark. Our results show that the newly proposed DLM-based framework advances the state of the art in lossless text compression. Moreover, as DLMs are still a relatively young paradigm, recent advances toward increasingly capable and efficient models suggest substantial room for further improvements.

Abstract

Explore similar work

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training

LLM-based Source Code Compression via Thresholded Symbol Ranking