Efficient Learned Data Compression via Dual-Stream Feature Decoupling

Huidong Ma; Xinyan Shi; Hui Sun; Xiaofei Yue; Xiaoguang Liu; Gang Wang; Wentong Cai

cs.CLApr 8, 2026

Efficient Learned Data Compression via Dual-Stream Feature Decoupling

Authors: Huidong Ma, Xinyan Shi, Hui Sun, Xiaofei Yue, Xiaoguang Liu, Gang Wang, Wentong Cai

Organizations: College of Computer Science, TMCC, SysNet, DISSec, GTIISC, Nankai University · College of Computing and Data Science, Nanyang Technological University · Beijing Institute of Technology

Abstract

While Learned Data Compression (LDC) has achieved superior compression ratios, balancing precise probability modeling with system efficiency remains challenging. Crucially, uniform single-stream architectures struggle to simultaneously capture micro-syntactic and macro-semantic features, necessitating deep serial stacking that exacerbates latency. Compounding this, heterogeneous systems are constrained by device speed mismatches, where throughput is capped by Amdahl's Law due to serial processing. To this end, we propose a Dual-Stream Multi-Scale Decoupler that disentangles local and global contexts to replace deep serial processing with shallow parallel streams, and incorporate a Hierarchical Gated Refiner for adaptive feature refinement and precise probability modeling. Furthermore, we design a Concurrent Stream-Parallel Pipeline, which overcomes systemic bottlenecks to achieve full-pipeline parallelism. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both compression ratio and throughput, while maintaining the lowest latency and memory usage. The code is available at https://github.com/huidong-ma/FADE.

Figures & tables

Figure 1: Trade-off between compression ratio and throughput. Top-right is better.

Figure 2: Overview of the proposed method. The embedded input $\bm{X}$ is disentangled into local and global contexts by the DMD, then fused and dynamically refined by the HGR to generate the final representation $\bm{H}$ .

Figure 3: Verification of dual dependency patterns on Silesia. (a) Mutual information decay exhibits a sharp initial drop (micro-syntactic) followed by a persistent non-zero tail (macro-semantic). (b) The self-similarity matrix corroborates this observation via a prominent diagonal band and recurring off-diagonal blocks. (c) Feature saliency heatmaps of the Global and Local streams, illustrating the distinct patterns captured by each branch.

Figure 4: Illustration of execution strategies: Serial, Temporal Parallelism, and Data Parallelism. $P$ and $E$ denote the probability distribution prediction executed on the GPU and the entropy coding process performed on the CPU, respectively. $W^{A/B}$ and $R^{A/B}$ represent writing to and reading from Buffer A/B, respectively.

Dataset	Type	Size (MB)	Hartley ( $\bm{H}_{0}$ )
Enwik9	text	954	7.69
LJSpeech	audio	281	8.00
TestImages	image	449	8.00
UVG	video	890	7.79
CESM	float	954	8.00
DNACorpus	genome	654	2.00

Table 1: Statistical information of the datasets.

\bm{H}_{0}=\text{log}\mathcal{|A|}

, where

\mathcal{A}

represents the alphabet set.

Method	Venue	Enwik9	LJSpeech	TestImages	UVG	CESM	DNACorpus	Silesia	Average $\uparrow$
Method	Venue	text	audio	image	video	float	genome	hete.	Average $\uparrow$
Traditional Compressor
Gzip	-	3.100	1.168	1.359	1.578	1.369	3.685	3.133	2.199
7z	-	4.689	1.370	1.670	1.887	1.829	4.450	4.352	2.892
PBZip2	-	3.936	1.363	1.723	2.054	1.413	3.805	3.878	2.596
zstd	-	4.249	1.238	1.524	1.819	1.404	4.276	4.008	2.645

Table 2: Compression ratios

\uparrow

of all 11 compressors on 7 datasets. Bold values denote the best results.

Method	Pipeline	Throughput (KB/min) $\uparrow$			Impr.	FLOPs $\downarrow$	Params $\downarrow$	Latency $\downarrow$	PGMU $\downarrow$
Method	(Cmp. \| Decmp.)	Cmp.	Decmp.	Total	(%)	(G)	(M)	(ms)	(GB)
DZip	Serial \| Serial	466	1365	695	525.5	16.56	26.18	1.98	0.496
TRACE	Serial \| Serial	2755	2187	2438	78.3	9.13	2.37	1.87	0.431
PAC	Serial \| Serial	2898	2349	2595	67.5	4.33	8.48	1.63	0.386
MSDZip	Serial \| Serial	1988	1814	1897	129.2	6.52	12.72	2.63	0.563
SEP	Parallel \| Serial	1954	1636	1781	144.1	5.41	10.57	2.26	1.053

Table 3: Efficiencies of LDC methods on Silesia. PGMU represents Peak GPU Memory Usage.

Module	Base +	CR $\uparrow$	TP (KB/min) $\uparrow$
Module	Base +	CR $\uparrow$	Cmp.	Decmp.	Total
DMD	MLP	3.412	6353	4853	5502
DMD	CNN	4.086	5600	4372	4910
HGR	CCI w/o gate	4.408	4983	3995	4435
	CCI w/ gate	4.565	4755	3836	4246
	FNR	5.400	3730	3188	3438

Table 4: Ablation study on the progressive integration of proposed components on Silesia.

Figure 5: Analysis of different settings on Silesia.

Figure 6: (a-b) NLL characteristics of the dual-stream architecture. (c-d) Impact of worker count and batch size on throughput. CT/DT and CTP/DTP denote Running Time and Throughput of Cmp./Decmp.

Pipeline	TRACE	PAC	MSDZip	SEP	EDPC
Standard	2438	2595	1897	1781	3461
w/ CSPP (Ours)	3130	3260	2295	2046	4257
Impr. (%)	28.38	25.63	20.98	14.88	23.00

Table 5: Analysis of the CSPP across LDC methods.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Type	Description	Link
Enwik9	text	First $10^{9}$ bytes of the English Wikipedia dump on 2006.	Page
LJSpeech	audio	First 10,000 files of the LJSpeech audio dataset.	Page
TestImages	image	A classical 8-bit benchmark dataset for image compression evaluation.	Page
UVG	video	The video ShakeNDry from the UVG benchmark featuring 1080p 8-bit YUV format.	Page
CESM	float	First $10^{9}$ bytes of floating-point data from the CESM-ATM climate dataset.	Page
DNACorpus	genome	A corpus of DNA sequences from 15 different species.	Page

Appendix

Table 6: Descriptions and links of multi-source datasets.

Method	Ref.	Version	Language	Methods	Link
Traditional Compressor
Gzip	Gailly and Adler (1992)	1.10	C/C++	LZ77, HC	Page
7z	Pavlov (1999)	24.08	C/C++	LZ77, AC	Page
PBZip2	Gilchrist (2003)	1.1.13	C/C++	BWT, HC	Page
zstd	Collet (2015)	1.5.6	C/C++	LZ77, HC	Page
Learned Compressor

Appendix

Table 7: Implementation details and characteristics of the baseline methods. LZ77 : repeated strings are coded by offset and length of previous occurrence; HC : Huffman Coding; BWT : Burrows-Wheeler Transform; AC : Arithmetic Coding.

Figure 7: Model loss trajectories of progressive ablation study.

Figure 8: Analysis of the content-adaptive router value dynamics.

Batch Size	Enwik9	LJSpeech	TestImages	UVG	CESM	DNACorpus	Silesia	Avg. CR	FLOPs	PGMU
Batch Size	text	audio	image	video	float	genome	hete.	Avg. CR	(G)	(GB)
512 (default)	6.288	1.880	2.402	2.603	2.939	4.503	5.400	3.716	7.83	0.367
1024	6.365	1.884	2.407	2.613	2.952	4.515	5.409	3.735	15.65	0.509
2048	6.423	1.888	2.410	2.623	2.951	4.548	5.390	3.748	31.31	0.796
4096	6.465	1.888	2.411	2.631	2.954	4.566	5.371	3.755	62.61	1.375
8192	6.491	1.887	2.409	2.643	2.948	4.568	5.335	3.754	125.22	2.532

Appendix

Table 8: Impact of batch size on compression ratio and performance.

Figure 9: Analysis of datasets via local entropy.

HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks
Sep 28, 2026Thai Nguyen, Khang Tran, NhatHai PhanDiffusion Language Model Inference Diffusion Language Models
New Jersey Institute of Technology
Learned Subspace Compression for Communication-Efficient Pipeline Parallelism
Jun 3, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLLM Compression Communication-Efficient Distributed Training
Concordia University · Mila Quebec AI Institute · CNRS, Sorbonne University
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Communication-Efficient Distributed Training Decentralized Learning
Pluralis Research

Sep 28, 2026J/K move · Enter open · S save

HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

Thai Nguyen, Khang Tran, NhatHai Phan

New Jersey Institute of Technology

Diffusion Language Model Inference Diffusion Language Models Lossless Data Compression

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong potential for lossless data compression, but existing approaches are constrained by the high computational cost and low throughput of autoregressive decoding. We propose HyperZip, an efficient and scalable LLM-based compression framework that leverages diffusion-based LLMs (dLLMs) with Multi-Token Prediction (MTP) to accelerate LLM-based data compression processes. We identify a trade-off in diffusion-based compression, where increasing decoding throughput degrades the compression rate. To mitigate this trade-off, HyperZip employs a hypernetwork to generate data-specific updates from a context representation, adapting the dLLM to the target data without costly fine-tuning, resulting in a low compression rate and high throughput. Extensive experiments show that HyperZip achieves a superior trade-off between compression rate and speed compared with state-of-the-art baselines.

Abstract

Explore similar work

HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

Learned Subspace Compression for Communication-Efficient Pipeline Parallelism

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism