Long-context LLM agents often struggle with growing token, memory, and latency costs, making efficient context compression essential for practical deployment. Existing LLM-as-a-compressor methods remain noticeably inferior to using the full context. We find that this gap partly stems from their inability to preserve contextual information effectively. In this work, we revisit context compression from a structural perspective and identify two key bottlenecks in standard LLM-based compressors: limited coordination among compression tokens during information aggregation, and layerwise dilution that weakens useful signals from intermediate hidden states. To address these limitations, we propose ComprExIT, a new context compression framework based on explicit information transmission. ComprExIT adaptively selects features across frozen LLM layers, then allocates information from anchors to compression slots through a globally coordinated transport plan. Experiments on 12 datasets show that ComprExIT consistently outperforms strong soft-compression baselines, improving average F1 by up to 18.5%, while adding only ~1% trainable parameters and achieving more than 2x faster compression than the fastest baselines. The code will be released upon acceptance.
Figures & tables
Figure 1: Performance (Average F1 over six datasets, Llama-3.2-3B) of existing context compression methods and their ability to preserve contextual information (reconstruction negative log-likelihood).
Figure 2: A comparison between existing LLM-as-a-compressor methods (left) and ComprExIT (right). Existing methods introduce gist tokens that are iteratively encoded across layers by the self-attention in the LLMs. ComprExIT instead leverages the layerwise hidden states of the context tokens encoded in a forward pass. The hidden states are selectively aggregated into token anchors, which are then transmitted to the compression tokens through a coordinated transmission plan.
Figure 3: Two views of gist-token aggregation patterns in ICAE: Pearson correlation between gist-token attention distributions (the diagonal, which is always 1, is masked in white for readability) and the singular-value spectrum of the attention aggregation matrix.
Figure 4: Performance of using each LLM layer for compression.
Methods
SQuAD
NewsQA
TriviaQA
SearchQA
HotpotQA
NQ
Average
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Llama-3.2-1B
Prompt tuning [w/ context]
71.89
81.09
30.72
45.76
61.04
66.75
64.03
70.64
53.60
69.49
53.03
66.59
55.72
66.72
Zero-shot [w/ context]
16.59
38.76
9.97
26.99
35.50
49.31
5.83
12.39
22.51
39.24
18.14
38.90
18.09
34.27
Zero-shot [w/o context]
2.80
8.83
0.97
3.11
15.63
21.60
4.04
6.64
2.76
7.17
3.37
8.43
4.93
9.29
ICAE ( Ge et al., 2024 )
36.84
50.21
20.20
33.73
57.59
63.54
67.40
74.43
40.67
57.04
42.26
58.00
44.16
56.16
Table 1: Experimental results on six QA benchmark datasets. Best among compression methods is bold . Our method is colored in . The prompt-tuned uncompressed baseline is colored in .
Table 6
Analysis
Setting
Avg. EM
Avg. F1
Compression ratio
×8 - ICAE
38.72
50.07
×8 - ComprExIT
44.83
58.66
×16 - ICAE
37.23
48.82
×16 - ComprExIT
40.01
52.53
Model size
8B - Prompt tuning
65.03
75.92
8B - ICAE
59.24
70.41
Table 4: Scalability analysis experiments for ComprExIT on higher compression ratios, larger models, and longer context.
Figure 6: Information allocation analysis. (a) Pearson correlation (diagonal masked) and (b) singular-value spectrum of allocation patterns. (c) Question-relevant entity recall on HotpotQA by TF-IDF rank. Top-10 entities are selected per context, and violin plot shows retrieval consistency across examples.
Figure 7: Layer-selection analysis in ComprExIT. Left : depth-wise gating weights across layers. Right : position–layer heatmap showing token-wise layer preferences.
Method
NLL ↓
Hr↑
ICAE
4.294
0.635
500x
4.176
0.645
ComprExIT
0.639
0.946
Table 5: Context reconstruction by a decoder trained from scratch on the frozen compressed representations (Llama-3.2-3B, 4× compression).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Sinkhorn iterations
Segment T
10
30
50
100
64
4.83
5.18
5.52
6.37
128
4.81
5.45
6.10
7.69
512
5.46
7.28
9.12
13.65
Appendix
Table 6: Per-sample time (ms) of the Sinkhorn step with a batch size of 64. Our default setting is underlined .
Dataset
Average Question Length
Average Context Length
#Train
#Validation
SQuAD
11
137
86,588
10,507
NewsQA
8
599
74,160
4,212
TriviaQA
16
784
61,688
7,785
SearchQA
17
749
117,384
16,980
HotpotQA
22
232
72,928
5,904
Natural Questions
9
153
104,071
12,836
Appendix
Table 7: Statistics of the training and evaluation datasets used, including in-domain and out-of-domain datasets. #Train represents the number of training samples and #Validation represents the number of validation samples.
Table 13
Methods
RelationExtraction
BioASQ
TextbookQA
DuoRC
DROP
RACE
Average
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Llama-3.2-1B-Base
Prompt tuning [w/ context]
69.91
81.15
65.96
78.80
52.10
59.61
38.51
47.40
32.54
43.14
33.68
46.30
48.78
59.40
Zero-shot [w/ context]
27.48
47.87
6.98
28.84
18.16
29.97
17.46
30.54
18.43
32.30
10.53
22.71
16.51
32.04
Zero-shot [w/o context]
5.09
10.33
14.76
20.56
11.11
16.03
0.60
2.55
14.84
19.73
0.45
2.88
7.81
12.01
ICAE [ Ge et al., 2024 ]
62.28
75.97
42.62
54.09
28.21
34.60
12.59
18.80
24.15
33.23
9.50
16.95
29.89
38.94
Appendix
Table 10: Experimental results on six out-of-domain MRQA benchmark datasets. Out-of-domain: these datasets are not included in the training data. Best among compression methods is bold . Our method is colored in . The prompt-tuned uncompressed baseline is colored in . Note that ICAE and 500x have a smaller compression ratio on samples with contexts shorter than 512 tokens because they are trained with a fixed set of 128 compression tokens. In particular, on the RelationExtraction dataset, whose average context length is only 30, they use more compression tokens than context tokens.
Sinkhorn iter.
5
10
30
50
100
Avg. F1
61.32
65.49
66.55
66.58
66.58
Segment T
64
128
512
Avg. F1
64.07
66.55
66.00
Proj. dim.
64
256
512
1024
Avg. F1
65.64
66.55
65.89
65.62
Appendix
Table 11: Sensitivity of ComprExIT to OT hyperparameters on Llama-3.2-1B (Avg. F1 on the six in-domain datasets). Our default settings are underlined .
Method
16k
32k
Mem.
Time
Mem.
Time
Full Context
8.91
–
11.79
–
ICAE
15.47
1.12
23.44
2.75
ComprExIT
19.33
0.52
32.57
1.36
ComprExIT (windowed)
11.76
0.52
16.83
1.36
Appendix
Table 12: Peak GPU memory (Mem., GB) and compression time (Time, s) at long contexts. The windowed variant is only used to measure efficiency and is not trained.
Figure 8: Last-layer attention heatmap of gist tokens produced by ICAE (left) and width-wise information transmission plan in ComprExIT (right).
Figure 9: The training curves of baseline methods and ComprExIT under the next-token prediction task.
Item
ComprExIT
Run & setup
Dataset
SlimPajama-6B
Number of Tokens
1 Billion
Device
4 × NVIDIA-A100
Precision
Bfloat16
Sequence & compression
Appendix
Table 13: NTP Training configuration for ComprExIT.
Item
ICAE
500x
Run & setup
Dataset
SlimPajama-6B
SlimPajama-6B
Number of Tokens
1 Billion
1 Billion
Device
4 × NVIDIA-A100
4 × NVIDIA-A100
Precision
Bfloat16
Bfloat16
Sequence & compression
Appendix
Table 14: NTP Training configuration for ICAE and 500x.
Item
Activation Beacon
Run & setup
Dataset
SlimPajama-6B
Number of Tokens
1 Billion
Device
4 × NVIDIA-A100
Precision
Bfloat16
Sequence
Appendix
Table 15: NTP Training configuration for Activation Beacon.