Lossy Compressive Text Autoencoders

Vinko Sabolčec; Angelos Katharopoulos; David Grangier

cs.CLOct 7, 2026

Lossy Compressive Text Autoencoders

Authors: Vinko Sabolčec, Angelos Katharopoulos, David Grangier

Organizations: Apple

Abstract

Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the multi-resolution autoencoder architecture.

Architecture	BLEU $\uparrow$	Judge Score $\uparrow$
Mean Pool AE	$0.12\pm 0.01$	$1.49\pm 0.11$
with Residual Projection	$0.77\pm 0.01$	$5.62\pm 0.21$
CLS Pool AE	$0.20\pm 0.01$	$1.83\pm 0.13$
with Residual Projection	$0.80\pm 0.01$	$\mathbf{5.90\pm 0.21}$
Multi-resolution AE	$0.77\pm 0.01$	$5.59\pm 0.21$
with Upscaler Loss	$\mathbf{0.91\pm 0.01}$	$\underline{5.78\pm 0.20}$

Table 1: BLEU and LLM judge score comparison of 0.6B parameter baseline and multi-resolution AE models trained on 128 sequence length, scaling factor of 128, and latent dimension 384.

Approach	Representation	Bits per Token $\downarrow$	BLEU $\uparrow$	Judge Score $\uparrow$
No Quantizer	Continuous	$48.00$	$0.91\pm 0.01$	$5.78\pm 0.20$
Narrow Bottleneck	Continuous	$12.00$	$0.10\pm 0.01$	$1.58\pm 0.11$
FSQ	Discrete	$9.00$	$\underline{0.44\pm 0.01}$	$\underline{2.41\pm 0.13}$
RVQ	Discrete	$9.00$	$0.07\pm 0.01$	$1.44\pm 0.10$
PRVQ	Discrete	$9.63$	$0.33\pm 0.01$	$2.13\pm 0.11$
CRVQ	Discrete	$9.63$	$\mathbf{0.76\pm 0.01}$	$\mathbf{3.25\pm 0.14}$

Table 2: BLEU and LLM judge score comparison of 0.6B parameter multi-resolution AE models trained at approximately 9 bits per token on 128 sequence length, scaling factor of 128, and latent dimension 384, except CRVQ which uses scaling factor 8 and latent dimension 24.

Figure 2: BLEU score comparison of multi-resolution AE model trained with stochastic CRVQ depth, uniformly sampled between 1 and 11 during training, at different inference-time depths, with the fixed-depth baselines (depths 3, 5, 7, and 9).

Loss Variants on Copy Dataset and No Mask
Supervision	BLEU $\uparrow$	Judge Score $\uparrow$	NumKept $\uparrow$	PNounKept $\uparrow$
Copy Dataset, CE Loss, and No Mask	$\underline{0.76\pm 0.01}$	$\underline{3.25\pm 0.14}$	$0.91\pm 0.01$	$0.68\pm 0.02$
BoundedCE log2	$0.45\pm 0.01$	$2.35\pm 0.08$	$0.51\pm 0.02$	$0.62\pm 0.02$
WeightedClass 10.0	$0.61\pm 0.01$	$2.75\pm 0.12$	$\mathbf{0.94\pm 0.01}$	$\mathbf{0.83\pm 0.01}$
$\text{CE}+0.1\cdot\text{CE}_{LM}$	$0.66\pm 0.01$	$2.62\pm 0.12$	$0.83\pm 0.02$	$0.60\pm 0.02$
Mask Variants on Copy Dataset and CE Loss

Table 3: Supervision evaluation comparison of 0.6B parameter CRVQ multi-resolution AE models trained at 9.63 bits per token on 128 sequence length, scaling factor of 8, and latent dimension 24.

Dataset	BLEU $\uparrow$	chrF $\uparrow$	Rouge-L $\uparrow$	NumKept $\uparrow$	PNounKept $\uparrow$	BERTScore $\uparrow$	Judge Score $\uparrow$
Copy	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$	$10.00\pm 0.00$
RTT	$\underline{0.58\pm 0.01}$	$\underline{0.79\pm 0.00}$	$\underline{0.81\pm 0.01}$	$\underline{0.89\pm 0.01}$	$0.76\pm 0.01$	$\mathbf{0.79\pm 0.01}$	$\underline{9.42\pm 0.06}$
Compress	$0.44\pm 0.02$	$0.66\pm 0.01$	$0.70\pm 0.01$	$0.82\pm 0.02$	$\underline{0.90\pm 0.01}$	$0.60\pm 0.02$	$\mathbf{9.74\pm 0.04}$
Corrupt	$\mathbf{0.72\pm 0.00}$	$\mathbf{0.81\pm 0.00}$	$\mathbf{0.87\pm 0.00}$	$\mathbf{0.99\pm 0.00}$	$\mathbf{0.99\pm 0.00}$	$\underline{0.72\pm 0.01}$	$2.03\pm 0.13$

Table 4: Comparison of evaluation metrics on four datasets: Copy (identical reference and hypothesis), RTT (round-trip translation), Compress (redundancy removal), and Corrupt (verb replacement). We use the Qwen3.6-35B-A3B model as the judge.

Figure 3: LLM judge scores from the Qwen3.5-27B judge of text summaries using different summary models and target summary ratios, and four datasets: Copy (identical reference and hypothesis), RTT (round-trip translation), Compress (redundancy removal), and Corrupt (verb replacement).

Method	Bits per Token $\downarrow$	Bits per Byte $\downarrow$	BLEU $\uparrow$	Judge Score $\uparrow$
Traditional Lossless
Zlib (Level 9)	$14.44$	$3.05$	$1.00\pm 0.00$	$10.00\pm 0.00$
Bzip2 (Level 9)	$11.36$	$2.40$	$1.00\pm 0.00$	$10.00\pm 0.00$
Zstandard (Level 3)	$13.91$	$2.94$	$1.00\pm 0.00$	$10.00\pm 0.00$
Zstandard (Level 22)	$10.50$	$2.22$	$1.00\pm 0.00$	$10.00\pm 0.00$
Neural Lossless

Table 5: Comparison of compression rates (Bits per Byte) and reconstruction performance (BLEU and judge score) across traditional lossless, neural lossless, and neural lossy text compression methods. Unless otherwise specified, neural-based approaches use 128 sequence length.

Model	Accuracy $\uparrow$
Same Passage Extraction	$0.64\pm 0.03$
SONAR	$0.55\pm 0.03$
Multi-resolution AE 0.6B	$\underline{0.63\pm 0.03}$
Multi-resolution AE 2.0B	$\mathbf{0.64\pm 0.03}$
Multi-resolution AE 4.3B	$\mathbf{0.64\pm 0.03}$

Table 6: Accuracy of trained multi-resolution AE models at 11 bits per token, and SONAR on the MS MARCO-based question-answering task at 128 sequence length.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: LLM judge score of 0.6B parameter multi-resolution AE models trained using the upscaler and the decoder loss at 128 sequence length, scaling factor of 128, hidden dimension 768, and varying latent bottleneck dimensions.

Figure 6: LLM judge score of 0.6B parameter multi-resolution AE models trained using the upscaler and the decoder loss, latent bottleneck dimension 48, scaling factor of 32, and varying sequence lengths.

Loss Weight	BLEU	Judge Score
0.75 Upscaler + 0.25 Decoder	$\mathbf{0.92\pm 0.01}$	$\underline{5.68\pm 0.18}$
0.50 Upscaler + 0.50 Decoder	$\underline{0.91\pm 0.01}$	$\mathbf{5.78\pm 0.20}$
0.25 Upscaler + 0.75 Decoder	$0.66\pm 0.02$	$4.15\pm 0.19$
0.00 Upscaler + 1.00 Decoder	$0.77\pm 0.01$	$5.59\pm 0.21$

Appendix

Table 7: BLEU and LLM judge score comparison of 0.6B parameter multi-resolution AE models trained on 128 sequence length, scaling factor of 128, and latent dimension 384 with varying upscaler and decoder loss weights.

Codebook Size	Num. Quantizers	Scaling Factor	Latent Dim.	BLEU	Judge Score
2048	7	8	24	$\mathbf{0.76\pm 0.01}$	$\mathbf{3.25\pm 0.14}$
2048	14	16	48	$\underline{0.60\pm 0.01}$	$\underline{2.61\pm 0.12}$
2048	28	32	96	$0.13\pm 0.01$	$1.56\pm 0.11$
2048	56	64	192	$0.11\pm 0.01$	$1.56\pm 0.11$

Appendix

Table 8: BLEU and LLM judge score comparison of 0.6B parameter multi-resolution AE models trained on 128 sequence length with varying chunked residual vector quantization (CRVQ) configurations at 9.63 bits per token.

Encoder Mask Ratio	BLEU	Judge Score
0%	$0.76\pm 0.01$	$3.25\pm 0.14$
5%	$\underline{0.78\pm 0.01}$	$3.49\pm 0.15$
10%	$\mathbf{0.79\pm 0.01}$	$\underline{3.68\pm 0.16}$
15%	$\mathbf{0.79\pm 0.01}$	$\mathbf{3.72\pm 0.16}$
20%	$\mathbf{0.79\pm 0.01}$	$\underline{3.68\pm 0.16}$

Appendix

Table 9: BLEU and LLM judge score comparison of 0.6B parameter CRVQ multi-resolution AE models trained on 128 sequence length, scaling factor of 8, and latent dimension 24, at 9.63 bits per token with varying encoder mask ratios at matched non-masked token count.

Figure 7: LLM judge scores from the Gemma4-31B-it judge of text summaries using different summary models and target summary ratios, and four datasets: Copy (identical reference and hypothesis), RTT (round-trip translation), Compress (redundancy removal), and Corrupt (verb replacement).

Explore similar work

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression
Aug 4, 2026Angelo Nardone, Paolo FerraginaDiffusion Language Model Inference Diffusion Language Models
University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy
When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression
May 31, 2026Riku Kisako, Hayato Tsukagoshi, Ryohei SasanoEmbedding Compression Dimensionality Reduction
Graduate School of Informatics, Nagoya University
Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction
May 27, 2026Yuchun Zou, Junhong Tong, Jun Li
CUNY Graduate Center · CUNY Queens College · CUNY Queens College & Graduate Center

Aug 4, 2026, cs.CLJ/K move · Enter open · S save

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

Angelo Nardone, Paolo Ferragina

University of Pisa, Department of Computer Science, Pisa, Italy · Scuola Superiore Sant’Anna, Pisa, Italy

Diffusion Language Model Inference Diffusion Language Models

Open paper AskPDF arXiv

We study the problem of lossless text compression, motivated by the rapid growth in the collection and storage of digital textual data - including plain text, source code, and structured formats such as XML - and by recent advances in neural language model-based compression. In particular, recent LLM-based approaches, whether built on symbol-ranking pipelines or paired with a statistical compressor, have demonstrated compression ratios significantly superior to general-purpose compressors such as zstd, gzip, or bzip on text and code. However, these neural approaches suffer from severe throughput limitations, making them not yet practically usable. For the first time in the context of lossless neural text compression, we introduce Diffusion Language Models (DLMs) as an alternative inference paradigm to autoregressive LLM-based approaches. We argue that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation. However, achieving these improvements requires addressing algorithmic challenges introduced by applying DLMs to lossless compression, where the architecture allows the number and positions of symbols encoded at each forward pass to be decided independently. We design efficient and effective strategies to solve these challenges and evaluate them experimentally against LLM-based and general-purpose compressors on enwik8, a well-established textual benchmark. Our results show that the newly proposed DLM-based framework advances the state of the art in lossless text compression. Moreover, as DLMs are still a relatively young paradigm, recent advances toward increasingly capable and efficient models suggest substantial room for further improvements.

Abstract

Explore similar work

Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression

Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction