Large Language Model as Token Compressor and Decompressor

Wenbing Li; Yiran Wang; Zikai Song; Jielei Zhang; Tianhao Zhao; Junkai Lin; Wei Yang

cs.CLMar 26, 2026

Large Language Model as Token Compressor and Decompressor

Authors: Wenbing Li, Yiran Wang, Zikai Song, Jielei Zhang, Tianhao Zhao, Junkai Lin, Wei Yang

Abstract

In this paper, we study whether an off-the-shelf LLM can be adapted into a discrete, variable-length token compressor and decompressor for long-context processing. To this end, we design a self-expressive autoencoding framework that fine-tunes a pretrained LLM with lightweight LoRA adapters to map long texts into compact sequences of learned latent codes, termed Z-tokens, and to decode them back into natural language or task outputs. The resulting representation is content-adaptive: less predictable or information-dense segments can receive more Z-tokens, while redundant regions can be represented more compactly through a budget-aware length regularizer. Our method is evaluated on long-context datasets such as Wikipedia, CNN/DailyMail, HotpotQA, and QuALITY, showing that it preserves reconstruction quality and downstream performance while reducing effective context length, generation-stage memory usage, and end-to-end latency. This simple design supports both direct decoding from compressed contexts and autoregressive generation in the Z-token space, providing a practical interface for efficient long-context inference.

Figures & tables

Figure 1: Our framework consists of three components: a compressor, an inference module, and a decompressor. It supports two usage paradigms. In the first, we directly feed the compressed representation to the decompressor to perform the downstream task. In the second, we first run the inference module in the Z-token space and then invoke the decompressor to recover task outputs at the token level.

Figure 2: The figure shows the flexibility of Z-tokens. On the left , the same sentence can be compressed into different Z-token sequences while preserving meaning after decompression. On the right , the same Z-tokens yield different outputs in different contexts. This suggests that Z-tokens are context-dependent rather than chaotic.

Context	PPL (Ori) $\downarrow$	PPL (Z) $\downarrow$	Gap
1024 $\rightarrow$ 512	29.96	30.12	+0.16
1024 $\rightarrow$ 256	30.42	30.65	+0.23
1024 $\rightarrow$ 128	29.90	30.34	+0.44

Table 4: Comparison of PPL in the continuation task.

Table 5

Method	F1 $\uparrow$
Qwen3-0.6B	18.44
ICAE	17.26
Ours	18.25

Table 7: QASPER sliding-window results.

Method	100	200	300	400	500
ICAE [ 7 ]	29.46	21.88	14.53	11.27	8.96
Ours	58.24	52.23	46.66	41.84	37.66

Table 8: Free-decoding BLEU-4 on Wikipedia under different input lengths.

Table 8

Input Length	Baseline			Ours (Compressed Context)
Input Length	Time (s)	Tok/s	Mem. (GB)	Time (s)	Tok/s	Mem. (GB)
10k	11.36	45.09	24.86	9.25	70.61	12.77
20k	16.11	34.06	41.82	12.39	56.28	19.17
30k	22.67	24.09	56.48	17.21	47.69	25.40

Table 12: On Qwen3-4B, we compare long-context generation with and without compression, reporting inference time, throughput, and peak GPU memory.

Dataset	Proxy	Pearson $r$	Spearman $\rho$
Wikipedia	Surprisal	0.54	0.57
HotpotQA	Surprisal	0.49	0.52
Wikipedia	Entity-rare density	0.36	0.39
HotpotQA	Entity-rare density	0.43	0.46

Table 13: Correlation between allocated Z-token count and semantic-density proxies.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Base Model	AutoCompressor	Ours
HumanEval	18.90	12.64	16.85

Appendix

Table 14: HumanEval results, reported with pass@1.

Dataset	Proxy	Pearson $r$	Spearman $\rho$
Wikipedia	Segment surprisal	0.54	0.57
HotpotQA	Segment surprisal	0.49	0.52
Wikipedia	Entity-rare density	0.36	0.39
HotpotQA	Entity-rare density	0.43	0.46

Appendix

Table 15: Correlation between allocated Z-token count and semantic-density proxies in the controlled diagnostic study.

Surprisal quintile	Mean allocated Z-tokens	Trend bar
Q1 (lowest)	37.1
Q2	41.0
Q3	46.4
Q4	52.2
Q5 (highest)	59.8

Appendix

Figure 3: Binned trend plot showing the relationship between surprisal quintile and allocated Z-token count. Higher-surprisal segments consistently receive more Z-tokens.

Dataset	Task	Train	Valid	Test	Output
CNN/DailyMail	Summarization	287,227	13,368	11,490	Summary
HotpotQA	Multi-hop QA	90,447	7,405	7,405	Answer
NarrativeQA	Long-form QA	32,747	3,461	10,557	Answer
QASPER	Scientific QA	2,593	1,006	1,451	Answer
QuALITY	Multiple-choice QA	265	265	265	Option

Appendix

Table 16: Statistics of the downstream long-context datasets used in our experiments. The reported sizes follow standard public splits; exact numbers may vary slightly depending on preprocessing.

Z=(z_{1},\ldots,z_{K}),\quad z_{t}\in V_{Z},

Appendix

Algorithm 1 Training the Z-token compressor, decompressor, and inferencer

Z^{p}\sim p_{\phi}(Z\mid X),\quad Z^{p}\subset V_{Z}.

Appendix

Algorithm 2 Inference with adapter switching

LLM Context Compression

See all 37 papers

End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · University of Maryland · Princeton University +3
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
Jul 31, 2026Maryam Haghifam, Jason Cong, Yizhou SunMemory-Augmented Language Models Long-Context QA
University of California, Los Angeles, USA
LatentPress: Context Compression Beyond Text and Vision
Sep 1, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +3LLM Inference Acceleration LLM Context Compression
Iowa State University · Cornell University

Jun 8, 2026J/K move · Enter open · S save

End-to-End Context Compression at Scale

Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra+11

New York University · University of Maryland · Princeton University+3

LLM Compression Long-Context Language Model Inference Language Modeling Context Compression LLM Context Compression

Open paper AskPDF arXiv

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt. Furthermore, many methods require the input to fit within the target model's context window, and are generally incompatible with modern production inference engines. Encoder-decoder compressors, which map a long token sequence to a shorter sequence of latent embeddings consumed by a decoder, are an appealing alternative in principle. However, existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier. In this work, we revisit encoder-decoder compression and close this gap. We first perform an architecture search, pre-training many variants from scratch to determine how best to design and train encoder-decoder compressors. Guided by our findings, we continually pre-train a family of 0.6B-encoder, 4B-decoder models on over 350B tokens each, at compression ratios of 1:4, 1:8, and 1:16. We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage. We demonstrate that LCLMs serve as efficient backbones for long-horizon agents, letting the agent skim through a compressed long context and adaptively expand relevant segments on demand.

Abstract

LLM Context Compression

Explore similar work

End-to-End Context Compression at Scale

SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

LatentPress: Context Compression Beyond Text and Vision