OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data

Yan Zhao; Zhengxue Cheng; Junxuan Zhang; Dajiang Zhou; Qunshan Gu; Qi Wang; Li Song

cs.LGFeb 25, 2026

OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data

Authors: Yan Zhao, Zhengxue Cheng, Junxuan Zhang, Dajiang Zhou, Qunshan Gu, Qi Wang, Li Song

Organizations: Shanghai Jiao Tong University · Ant Group

Abstract

Lossless compression is essential for efficient data storage and transmission. Although learning-based lossless compressors achieve strong results, most of them are designed for a single modality, leading to redundant compressor deployments in multi-modal settings. Designing a unified multi-modal compressor is critical yet challenging, as different data types vary largely in format, dimension, and statistics. Multi-modal large language models offer a promising resolution but remain too complex for practical use. Thus, we propose OmniZip, a unified and lightweight lossless compressor for multi-modal data (like image, text, speech, tactile, database, and gene sequence). Built on a lightweight backbone, OmniZip incorporates three key components to enable efficient multi-modal lossless compression: a modality-unified tokenizer that reversibly transforms diverse data into tokens, a modality-routing context learning mechanism that enables flexible multi-modal context modeling, and a modality-routing feedforward design that further enhances the model's nonlinear representation flexibility. A reparameterization training strategy is used to enhance model capacity. OmniZip outperforms or matches other state-of-the-art compressors on multiple modalities, achieving 42%, 57%, 62% and 42%, 53% higher compression efficiency than gzip on CLIC-M, TouchandGo, enwik9, LibriSpeech, and WikiSQL datasets, respectively. It also supports near real-time inference on resource-constrained edge devices, reaching about 1MB/s on MacBook CPUs and iPhone NPUs. Our code is released at https://github.com/adminasmi/OmniZip-CVPR2026.

Figures & tables

Figure 1: Left : The world is multi-modal, motivating the need for unified lossless compression. Multi-modal compressors’ performance is shown across modalities, where points closer to the edge indicate better efficiency (lower bits/Byte). Right : Lightweight design ensures broad usability across platforms. OmniZip achieves near real-time inference (hundreds of KB/s to MB/s) even on edge devices.

Figure 2: Overview of the proposed OmniZip framework. Diverse data is first converted into a unified, fully reversible token space. A predictive model then estimates each token’s contextual probability, followed by arithmetic coding to generate the bitstream.

Model	#Params $\downarrow$	MACs $\downarrow$	bits/Byte $\downarrow$	Speed (KB/s) $\uparrow$
Transformer [ 65 ]	0.2M	0.88M	2.197	714
Transformer [ 65 ]	3.2M	4.43M	1.984	280
RWKV-7 [ 54 ]	0.2M	0.85M	1.910	2292
RWKV-7 [ 54 ]	3.2M	3.68M	1.658	856

Table 1: Comparison for model backbone selection. Models are evaluated using text lossless compression. Inference speed is measured on the MacBook Pro CPU, with a batch size of 128.

Figure 3: One block of OmniZip’s predictive model. The model stacks $N$ such blocks in total. Built on a lightweight RWKV7 backbone, it incorporates two modality-routing MoE modules for contextual learning and feedforward processing.

Model	Blocks	Embed Dim	#Params	MACs
OmniZip-S	2	320	4.8M	3.88M
OmniZip-M	2	912	38M	18.2M
OmniZip-L	3	1488	152M	56.0M

Table 2: Details of the proposed OmniZip models.

	Compressor	#Params $\downarrow$	MACs $\downarrow$	Speed (KB/s) $\uparrow$	Multi Modal	bits/Byte $\downarrow$
			MACs $\downarrow$	Speed (KB/s) $\uparrow$	Multi Modal	Kodak	CLIC-P	CLIC-M	DIV2K	TouchandGo	ObjectFolder	Axial	Coronal	Sagittal
Classical	gzip † [ 52 ]	-	-	14500 ′	✓	4.349 ${}_{\textbf{(0\%)}}$	4.039 ${}_{\textbf{(0\%)}}$	3.947 ${}_{\textbf{(0\%)}}$	4.224 ${}_{\textbf{(0\%)}}$	2.298 ${}_{\textbf{(0\%)}}$	3.969 ${}_{\textbf{(0\%)}}$	5.346 ${}_{\textbf{(0\%)}}$	4.563 ${}_{\textbf{(0\%)}}$	5.571 ${}_{\textbf{(0\%)}}$
	bzip2 † [ 59 ]	-	-	3300 ′	✓	4.359	4.033	3.931	4.208	2.288	4.031	5.403	4.617	5.628
	zstd † [ 48 ]	-	-	11000 ′	✓	4.350	4.042	3.952	4.229	2.263	3.966	5.343	4.560	5.568
	PNG † [ 11 ]	-	-	200 ′	✗	4.349	4.041	3.952	4.229	2.500	3.964	5.361	4.581	5.580
	FLIF † [ 62 ]	-	-	652 ′	✗	2.903	2.792	2.497	2.914	0.808	3.765	4.860	3.999	5.160

Table 3: Lossless compression results on image-like (natrual image, medical image, and tactile) datasets. Multi-modal compressors (except ours) are shown in gray.

*

denotes pretrained LLMs,

{\dagger}

indicates that some results are reprodeced by us.

\prime

indicates compression speeds measured on a MacBook CPU, and

{\ddagger}

denotes speeds on an A100 GPU (batch size 128). Other unmarked values are taken from their papers.

	Compressor	#Params $\downarrow$	MACs $\downarrow$	Speed (KB/s) $\uparrow$	Multi Modal	bits/Byte $\downarrow$
			MACs $\downarrow$	Speed (KB/s) $\uparrow$	Multi Modal	enwik9	Gutenberg	Spider	WikiSQL	GenoSeq	DNACorpus	LibriSpeech
Classical	gzip † [ 52 ]	-	-	14500 ′	✓	2.590 ${}_{\textbf{(0\%)}}$	3.819 ${}_{\textbf{(0\%)}}$	2.289 ${}_{\textbf{(0\%)}}$	1.672 ${}_{\textbf{(0\%)}}$	2.390 ${}_{\textbf{(0\%)}}$	2.318 ${}_{\textbf{(0\%)}}$	6.511 ${}_{\textbf{(0\%)}}$
	bzip2 † [ 59 ]	-	-	3300 ′	✓	2.082	3.607	2.326	1.403	2.202	2.148	5.647
	zstd † [ 48 ]	-	-	11000 ′	✓	2.364	3.703	2.258	1.520	2.404	2.324	6.467
	$\text{FLAC}^{{\dagger}}$ [ 18 ]	-	-	2000 ′	✗	-	-	-	-	-	-	4.961
Learning-based	$\text{Llama3}^{*{\dagger}}$ [ 22 ]	8B	7.80G	20 ‡	✓	0.722	1.030	0.770	0.645	1.889	1.808	3.616

Table 4: Lossless compression results on text-like (natural text, gene sequence, database) and speech datasets. Multi-modal compressors (except ours) are shown in gray.

*

denotes pretrained LLMs,

{\dagger}

indicates that some results are reprodeced by us.

\prime

indicates compression speeds measured on a MacBook CPU, and

{\ddagger}

denotes speeds on an A100 GPU (batch size 128). Other values are taken from their papers.

Figure 4: Comparison of learning-based lossless compressors across multi-modal datasets. The x-axis shows the model size (in millions of parameters), and the y-axis indicates compression efficiency (bits/Byte, lower is better). Models closer to the lower-left corner achieve better compression with fewer parameters. The dashed orange line represents the performance baseline of gzip.

Figure 5: OmniZip’s inference speed across various platforms (CPU of MacBook Pro, NPU of iPhone17 Pro, and GPU of NVIDIA A100) and batch sizes (1, 16, 128, 512, 1024).

Figure 6: Expert usage of OmniZip-S across blocks and routing modules. Each bar shows the expert usage percent (%) for a modality in a routing module (context learning or feedforward).

Unifed Tokenizer	Contextual Routing	Feedforward Routing	Reparam	Multi Modal	#Params $\downarrow$	MACs $\downarrow$	Speed (KB/s) $\uparrow$	bits/Byte $\downarrow$
Unifed Tokenizer	Contextual Routing	Feedforward Routing	Reparam	Multi Modal	#Params $\downarrow$	MACs $\downarrow$	Speed (KB/s) $\uparrow$	Kodak	TouchandGo	Coronal	enwik9	Spider	GenoSeq	LibriSpeech
✗	✗	✗	✗	✗	3.2M	3.68M	856	-	-	-	1.658	-	-	-
✗	✗	✗	✓	✗	3.2M	3.68M	856	-	-	-	1.590	-	-	-
✓	✗	✗	✓	✓	3.2M	3.68M	856	3.383	1.453	4.545	1.660	1.662	1.984	4.365
✓	✗	✓	✓	✓	4.0M	3.73M	633	3.352	1.431	4.314	1.424	1.448	1.915	4.295
✓	✓	✗	✓	✓	3.0M	3.82M	780	3.339	1.419	4.188	1.410	1.482	1.889	4.302

Table 5: Ablations on all our proposals: modality-unified tokenization, modality-routing contextual learning, modality-routing feedforward, and reparameterization training. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange.

Contextual Routing			#Params $\downarrow$		MACs $\downarrow$	Speed (KB/s) $\uparrow$	bits/Byte $\downarrow$
Receptance	Key	Value	Total	Activated	MACs $\downarrow$	Speed (KB/s) $\uparrow$	Kodak	TouchandGo	Coronal	enwik9	Spider	GenoSeq	LibriSpeech
✓	✗	✗	4.8M	2.7M	3.88M	500	3.315	1.372	4.224	1.311	1.324	1.795	4.281
✗	✓	✗	4.8M	2.7M	3.88M	500	3.614	1.533	4.398	1.388	1.470	1.862	4.391
✗	✗	✓	4.8M	2.7M	3.88M	500	3.307	1.338	4.179	1.370	1.320	1.868	4.155
✓	✓	✗	5.4M	3.0M	4.02M	436	3.317	1.352	4.080	1.292	1.361	1.860	4.280
✓	✗	✓	5.4M	3.0M	4.02M	436	3.313	1.360	4.140	1.297	1.309	1.813	4.178

Table 6: Ablations on modality-routing contextual learning. Routing is applied to all blocks with 4 experts and top-

k

=2. We report the total parameters and parameters activated per token. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange.

Blocks	top- $k$	Experts	Hidden Factor	#Params $\downarrow$		MACs $\downarrow$	Speed (KB/s) $\uparrow$	bits/Byte $\downarrow$
Blocks	top- $k$	Experts	Hidden Factor	Total	Activated	MACs $\downarrow$	Speed (KB/s) $\uparrow$	Kodak	TouchandGo	Coronal	enwik9	Spider	GenoSeq	LibriSpeech
1	1	4	$2\times$	3.6M	2.2M	3.65M	594	3.436	1.150	4.267	1.391	1.511	1.878	4.336
1	2	4	$2\times$	3.6M	2.5M	3.85M	570	3.358	1.420	4.230	1.380	1.344	1.896	4.307
2	2	4	$2\times$	4.8M	2.7M	3.88M	500	3.307	1.338	4.179	1.370	1.320	1.868	4.155
2	2	3	$2\times$	4.0M	2.7M	3.88M	500	3.355	1.424	4.154	1.369	1.410	1.882	4.300
2	2	5	$2\times$	5.2M	2.7M	3.88M	500	3.232	1.416	4.190	1.358	1.352	1.855	4.206

Table 7: Ablations on the configurations of the two modality-routing modules (e.g., routed blocks, expert count, top-

k

). We report the total parameters and parameters activated per token. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange.

Figure 7: Key structural differences between RWKV-7 and OmniZip model. Left: an RWKV-7 block. Right: an OmniZip block.

Figure 8: Illustration of the reparameterization training strategy.

Dataset	Type	Train	Test	Description
Kodak [ 21 ]	image	✗	✓	24 film-scanned photos ( $768\times 512$ ) with balanced natural and synthetic scenes. We use it for testing.
CLIC-P [ 17 ]	image	✗	✓	41 high-resolution DSLR images (mostly 2K) with rich color and texture diversity. We use it for testing.
CLIC-M [ 17 ]	image	✗	✓	61 smartphone photos (mostly 2K) containing real-world noise and artifacts. We use it for testing.
DIV2K [ 2 ]	image	✓	✓	900 2K-resolution images with a large diversity of contents. We follow the official split (800 train, 100 test).
Axial [ 10 ]	medical	✓	✓	2D knee MRI slices ( $256\times 256$ ) in the axial plane. We follow the official split ( $\sim$ 38K train, $\sim$ 4.1K test).
Coronal [ 10 ]	medical	✓	✓	2D knee MRI slices ( $256\times 256$ ) in the coronal plane. We follow the official split ( $\sim$ 33K train, $\sim$ 3.5K test).

Table 8: Details of the utilized multi-modal datasets (16 datasets from 7 modalities).

Compressor	#Params $\downarrow$	Multi Modal	Adjusted bits/Byte $\downarrow$
	#Params $\downarrow$	Multi Modal	Kodak	CLIC-P	CLIC-M	DIV2K	TouchandGo	ObjectFolder	Axial	Coronal	Sagittal
$\text{Llama3}^{*{\dagger}}$ [ 22 ]	8B	✓	8490321	971320	553618	291465	322642	214197	693352	750368	970961
$\text{RWKV}^{*{\dagger}}$ [ 22 ]	7B	✓	7429031	849906	484417	255032	282312	187423	606683	656573	823341
DLPR ∗† [ 5 ]	22.3M	✗	23670	2710	1545	815	900	601	-	-	-
L3C ∗† [ 46 ]	5M	✗	5310	610	349	185	203	138	439	473	594
P2LLM [ 13 ]	8B	✗	8490319	971318	553616	291463	-	-	-	-	-

Table 9: Adjusted compression performance (bits/Byte) on image-like (natual image, medical image, and tactile) datasets.

*

denotes pretrained LLMs,

{\dagger}

indicates that some results are reprodeced by us. Other values are taken from their papers.

Compressor	#Params $\downarrow$	Multi Modal	Adjusted bits/Byte $\downarrow$
	#Params $\downarrow$	Multi Modal	enwik9	Gutenberg	Spider	WikiSQL	GenoSeq	DNACorpus	LibriSpeech
$\text{Llama3}^{*{\dagger}}$ [ 22 ]	8B	✓	129	362459	2125	10835	500	1138	212
$\text{RWKV}^{*{\dagger}}$ [ 22 ]	7B	✓	113	317152	1859	9480	437	996	186
tszip [ 9 ]	169	✗	4	7658	46	230	13	26	-
L3TC [ 71 ]	12	✗	2	546	6	18	3	4	-
OmniZip-S	4.8M	✓	1	219	3	8	2	3	4

Table 10: Adjusted compression performance (bits/Byte) on text-like (natural language, gene sequence, database) and speech datasets.

*

denotes pretrained LLMs,

{\dagger}

indicates that some results are reprodeced by us. Other values are taken from their papers.

Lossless Data Compression

See all 26 papers

Feb 27, 2026

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder

Recent multimodal systems often rely on separate expert modality encoders which cause linearly scaling complexity and computational overhead with added modalities. While unified Omni-models address this via Mixture-of-Expert (MoE) architectures with specialized experts and routing, they still inflate parameter counts and introduce routing overhead. In this paper, we propose Omni-C (Omni-Compress), a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities--images, audio, and text--through unimodal contrastive pretraining on large-scale unaligned data. By maximizing parameter sharing in the backbone and using lightweight modality-specific projection heads, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing. This design supports efficient deployment on memory-constrained systems via sequential modality processing and low-memory inference, eliminating the need for parallel expert loading or specialized hardware. Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks, with modest zero-shot degradation on audio and text that is largely recovered through lightweight linear probing or parameter efficient fine-tuning. The unified architecture substantially reduces inference memory usage compared to multi-encoder baselines, advancing efficient and scalable multimodal learning.

Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1

City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom

Multimodal Learning Unified Multimodal Models Multimodal Transformer Cross-Modal Representation Learning Efficient Multimodal Inference Multimodal Contrastive Learning

Ask PDF

Aug 4, 2026

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.

Wanshun Su, Yang Shi, Feihu Liu +10

Northwestern Polytechnical University · Peking University · Alibaba Group +1

Efficient Multimodal Inference Audio Token Compression Multimodal Large Language Models Multimodal Token Compression Visual Token Compression

Ask PDF

Jul 25, 2026

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

Jinsen Su, Yongdong Luo, Yuexiao Ma +3

Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China +1

Efficient Multimodal Inference Audio Token Compression Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

Abstract

Lossless Data Compression

Explore similar work

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models