Long-context LLM agents often struggle with growing token, memory, and latency costs, making efficient context compression essential for practical deployment. Existing LLM-as-a-compressor methods remain noticeably inferior to using the full context. We find that this gap partly stems from their inability to preserve contextual information effectively. In this work, we revisit context compression from a structural perspective and identify two key bottlenecks in standard LLM-based compressors: limited coordination among compression tokens during information aggregation, and layerwise dilution that weakens useful signals from intermediate hidden states. To address these limitations, we propose ComprExIT, a new context compression framework based on explicit information transmission. ComprExIT adaptively selects features across frozen LLM layers, then allocates information from anchors to compression slots through a globally coordinated transport plan. Experiments on 12 datasets show that ComprExIT consistently outperforms strong soft-compression baselines, improving average F1 by up to 18.5%, while adding only ~1% trainable parameters and achieving more than 2x faster compression than the fastest baselines. The code will be released upon acceptance.
Figures & tables
Figure 1: Performance (Average F1 over six datasets, Llama-3.2-3B) of existing context compression methods and their ability to preserve contextual information (reconstruction negative log-likelihood).
Figure 2: A comparison between existing LLM-as-a-compressor methods (left) and ComprExIT (right). Existing methods introduce gist tokens that are iteratively encoded across layers by the self-attention in the LLMs. ComprExIT instead leverages the layerwise hidden states of the context tokens encoded in a forward pass. The hidden states are selectively aggregated into token anchors, which are then transmitted to the compression tokens through a coordinated transmission plan.
Figure 3: Two views of gist-token aggregation patterns in ICAE: Pearson correlation between gist-token attention distributions (the diagonal, which is always 1, is masked in white for readability) and the singular-value spectrum of the attention aggregation matrix.
Figure 4: Performance of using each LLM layer for compression.
Methods
SQuAD
NewsQA
TriviaQA
SearchQA
HotpotQA
NQ
Average
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Llama-3.2-1B
Prompt tuning [w/ context]
71.89
81.09
30.72
45.76
61.04
66.75
64.03
70.64
53.60
69.49
53.03
66.59
55.72
66.72
Zero-shot [w/ context]
16.59
38.76
9.97
26.99
35.50
49.31
5.83
12.39
22.51
39.24
18.14
38.90
18.09
34.27
Zero-shot [w/o context]
2.80
8.83
0.97
3.11
15.63
21.60
4.04
6.64
2.76
7.17
3.37
8.43
4.93
9.29
ICAE ( Ge et al., 2024 )
36.84
50.21
20.20
33.73
57.59
63.54
67.40
74.43
40.67
57.04
42.26
58.00
44.16
56.16
Table 1: Experimental results on six QA benchmark datasets. Best among compression methods is bold . Our method is colored in . The prompt-tuned uncompressed baseline is colored in .
Table 6
Analysis
Setting
Avg. EM
Avg. F1
Compression ratio
×8 - ICAE
38.72
50.07
×8 - ComprExIT
44.83
58.66
×16 - ICAE
37.23
48.82
×16 - ComprExIT
40.01
52.53
Model size
8B - Prompt tuning
65.03
75.92
8B - ICAE
59.24
70.41
Table 4: Scalability analysis experiments for ComprExIT on higher compression ratios, larger models, and longer context.
Figure 6: Information allocation analysis. (a) Pearson correlation (diagonal masked) and (b) singular-value spectrum of allocation patterns. (c) Question-relevant entity recall on HotpotQA by TF-IDF rank. Top-10 entities are selected per context, and violin plot shows retrieval consistency across examples.
Figure 7: Layer-selection analysis in ComprExIT. Left : depth-wise gating weights across layers. Right : position–layer heatmap showing token-wise layer preferences.
Method
NLL ↓
Hr↑
ICAE
4.294
0.635
500x
4.176
0.645
ComprExIT
0.639
0.946
Table 5: Context reconstruction by a decoder trained from scratch on the frozen compressed representations (Llama-3.2-3B, 4× compression).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Sinkhorn iterations
Segment T
10
30
50
100
64
4.83
5.18
5.52
6.37
128
4.81
5.45
6.10
7.69
512
5.46
7.28
9.12
13.65
Appendix
Table 6: Per-sample time (ms) of the Sinkhorn step with a batch size of 64. Our default setting is underlined .
Dataset
Average Question Length
Average Context Length
#Train
#Validation
SQuAD
11
137
86,588
10,507
NewsQA
8
599
74,160
4,212
TriviaQA
16
784
61,688
7,785
SearchQA
17
749
117,384
16,980
HotpotQA
22
232
72,928
5,904
Natural Questions
9
153
104,071
12,836
Appendix
Table 7: Statistics of the training and evaluation datasets used, including in-domain and out-of-domain datasets. #Train represents the number of training samples and #Validation represents the number of validation samples.
Table 13
Methods
RelationExtraction
BioASQ
TextbookQA
DuoRC
DROP
RACE
Average
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Llama-3.2-1B-Base
Prompt tuning [w/ context]
69.91
81.15
65.96
78.80
52.10
59.61
38.51
47.40
32.54
43.14
33.68
46.30
48.78
59.40
Zero-shot [w/ context]
27.48
47.87
6.98
28.84
18.16
29.97
17.46
30.54
18.43
32.30
10.53
22.71
16.51
32.04
Zero-shot [w/o context]
5.09
10.33
14.76
20.56
11.11
16.03
0.60
2.55
14.84
19.73
0.45
2.88
7.81
12.01
ICAE [ Ge et al., 2024 ]
62.28
75.97
42.62
54.09
28.21
34.60
12.59
18.80
24.15
33.23
9.50
16.95
29.89
38.94
Appendix
Table 10: Experimental results on six out-of-domain MRQA benchmark datasets. Out-of-domain: these datasets are not included in the training data. Best among compression methods is bold . Our method is colored in . The prompt-tuned uncompressed baseline is colored in . Note that ICAE and 500x have a smaller compression ratio on samples with contexts shorter than 512 tokens because they are trained with a fixed set of 128 compression tokens. In particular, on the RelationExtraction dataset, whose average context length is only 30, they use more compression tokens than context tokens.
Sinkhorn iter.
5
10
30
50
100
Avg. F1
61.32
65.49
66.55
66.58
66.58
Segment T
64
128
512
Avg. F1
64.07
66.55
66.00
Proj. dim.
64
256
512
1024
Avg. F1
65.64
66.55
65.89
65.62
Appendix
Table 11: Sensitivity of ComprExIT to OT hyperparameters on Llama-3.2-1B (Avg. F1 on the six in-domain datasets). Our default settings are underlined .
Method
16k
32k
Mem.
Time
Mem.
Time
Full Context
8.91
–
11.79
–
ICAE
15.47
1.12
23.44
2.75
ComprExIT
19.33
0.52
32.57
1.36
ComprExIT (windowed)
11.76
0.52
16.83
1.36
Appendix
Table 12: Peak GPU memory (Mem., GB) and compression time (Time, s) at long contexts. The windowed variant is only used to measure efficiency and is not trained.
Figure 8: Last-layer attention heatmap of gist tokens produced by ICAE (left) and width-wise information transmission plan in ComprExIT (right).
Figure 9: The training curves of baseline methods and ComprExIT under the next-token prediction task.
Item
ComprExIT
Run & setup
Dataset
SlimPajama-6B
Number of Tokens
1 Billion
Device
4 × NVIDIA-A100
Precision
Bfloat16
Sequence & compression
Appendix
Table 13: NTP Training configuration for ComprExIT.
Item
ICAE
500x
Run & setup
Dataset
SlimPajama-6B
SlimPajama-6B
Number of Tokens
1 Billion
1 Billion
Device
4 × NVIDIA-A100
4 × NVIDIA-A100
Precision
Bfloat16
Bfloat16
Sequence & compression
Appendix
Table 14: NTP Training configuration for ICAE and 500x.
Item
Activation Beacon
Run & setup
Dataset
SlimPajama-6B
Number of Tokens
1 Billion
Device
4 × NVIDIA-A100
Precision
Bfloat16
Sequence
Appendix
Table 15: NTP Training configuration for Activation Beacon.
Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt. Furthermore, many methods require the input to fit within the target model's context window, and are generally incompatible with modern production inference engines. Encoder-decoder compressors, which map a long token sequence to a shorter sequence of latent embeddings consumed by a decoder, are an appealing alternative in principle. However, existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier. In this work, we revisit encoder-decoder compression and close this gap. We first perform an architecture search, pre-training many variants from scratch to determine how best to design and train encoder-decoder compressors. Guided by our findings, we continually pre-train a family of 0.6B-encoder, 4B-decoder models on nearly 400B tokens each, at compression ratios of 1:4, 1:8, and 1:16. We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage. LCLMs also unlock new efficient downstream applications: they serve as backbones for long-horizon agents that skim a compressed long context and adaptively expand relevant segments on demand, and they can compact the model's own generated reasoning into latent memory during inference, which is nearly lossless on the full MATH test set while being highly efficient, reducing peak KV usage by up to 7.9×.
Ang Li, Sean McLeish, Haozhe Chen +12
New York University · Modal Labs · Harvard University +5
Context compression aims to shorten long context inputs with minimal information loss for LLM inference acceleration. While existing methods have shown promise, they typically rely on complex compression modules or compression-specific training, leaving the intrinsic capabilities of LLMs underexplored. In contrast, this work reveals that a thinking model itself can naturally compress long contexts by organizing task-relevant information. We thus derive Thinking as Compression (TaC), a new compression paradigm that treats thinking itself as compressed context. Without relying on specific dedicated compressor, TaC directly prompts the thinking model to generate thinking traces as the shortened context, already outperforming most representative compression methods. Further, given that raw thinking output may struggle with budget control and shortcut behaviors, we introduce Thinking as Compression Constrained (TaC-C), leveraging a simple reward-driven optimization framework to elicit intrinsic thinking as compact and controllable compressed context. Experiments across four long-context QA benchmarks demonstrate that TaC-C consistently outperforms existing baselines. At 4x and 8x compression ratios, it surpasses the strongest competitor by 17.4% and 23.4% in average F1, and by 15.7% and 21.7% in average Exact Match Score (EM), respectively.
Guoxin Ma, Yibing Liu, Chengzhengxu Li +7
Baidu Inc., Beijing, China · Xi’an Jiaotong University · City University of Hong Kong +1
Long-horizon LLM agents play an important role in solving complex and sustained tasks, with ever-growing interaction contexts. However, these accumulated contexts will raise substantial computational cost and obscure key decision-relevant evidence. Existing methods typically train external compressors or context-management agents to rewrite context as shorter hard-token sequences, but such external selection may not align with the LLM's evolving representation and is necessarily lossy. In this paper, we introduce Agentic Soft-Token Compression (ASTC), a training-free compression method using soft tokens within LLM agent for long-horizon tasks. Specifically, ASTC uses the model's native recurrent dynamics to determine the already absorbed information, leveraging the commonly-used hybrid Gated Delta Network (GDN)-attention. It then carries this information forward as agentic soft tokens while retaining exact values as hard tokens. The resulting compact sequence reduces later-layer computation and yields reusable KV caches, extending the savings across the agent's trajectory. Extensive experiments on long-horizon tasks show that ASTC matches or exceeds the uncompressed agent's accuracy with up to 59.84% fewer end-to-end Attention Pairs, while trained compressors require 12.3 to 194.9 times more Attention Pairs per successful task. These findings highlight agentic soft tokens as a model-native way to carry absorbed context forward as the task representation evolves.
Xuning He, Kai Fan, Runyu Zhang +4
Shanghai Jiao Tong University · Nanyang Technological University · Nankai University +2