Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Authors: Yong Zhang, Heng Li, Yanwen Huang, Ning Cheng, Yang Guo, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
Organizations: Ping An Technology (Shenzhen) Co., Ltd., China · University of Science and Technology of China · University of Electronic Science and Technology of China
Retrieved contexts are often long, noisy, and redundant, motivating query-aware compression before answer generation. Native attention provides signals for context selection, but heterogeneous head behaviors complicate their direct aggregation into sentence-utility scores. We introduce Sentinel, a sentence-level compressor that learns to combine these signals through a signed linear readout across layers and heads. Only the readout is trained using existing QA evidence annotations, while the backbone remains frozen and sentence scoring requires no autoregressive generation. Across twelve LongBench tasks, Sentinel with a 0.5B backbone outperforms Full Context on multiple tasks under a 2,000-token budget with each of three readers: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and DeepSeek V4.1 Flash. Despite English-only QA supervision, the same readout transfers to Chinese tasks without task-specific retraining. Length-stratified profiling further shows lower compression latency and GPU memory usage than LLMLingua-2 and CPC.
Figures & tables
Figure 1: Sentinel Framework Overview. Sentinel learns a signed sentence-level readout over a frozen language model’s attention features. Retrieval-dependent QA selection provides a sample-level behavioral basis for training; existing answer spans and supporting facts supply positive sentence labels, with other sentences labeled negative. Only the probe is trained. At inference time, one non-autoregressive forward pass scores the sentences in a native context window for budgeted selection.
Method
Qwen2.5-7B-Instruct
DeepSeek V4.1 Flash
English
Chinese
Macro
English
Chinese
Macro
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
Reference conditions
Empty
9.65
19.53
9.74
12.98
8.76
12.61
7.60
9.66
12.15
12.01
31.50
9.93
17.81
22.00
15.86
7.56
15.14
17.15
Full Context
41.98
45.97
26.84
38.26
61.76
17.51
11.58
30.28
36.27
48.35
64.82
27.86
47.01
63.97
17.19
12.94
31.37
43.10
Metric-Based Context Compression
Table 1: LongBench results by task category and reader. SD, MD, and Sum denote Single-Document QA, Multi-Document QA, and Summarization. Avg. averages the nine English or three Chinese tasks equally; Macro averages all 12 tasks. Full Context is uncompressed. DeepSeek thinking is disabled. Best and second-best compression results in each column are bold and underlined, respectively.
Compressor query
Answer F1 (%)
Correct
29.62
Swapped
6.92
Generic
3.16
Table 2: Query ablation on 100 common valid Qasper QA examples with a 256-token compression budget and the Qwen2.5-7B-Instruct reader. The reader always answers the target question.
Task
Sentinel
Negative weights set to zero
NarrativeQA
22.73
17.03
Qasper
45.19
45.19
MultiFieldQA-en
51.07
50.14
HotpotQA
56.16
54.97
2WikiMQA
49.07
47.59
MuSiQue
33.12
33.06
Table 3: Effect of signed probe weights. Negative coefficients are set to zero without retraining. Both variants use the Qwen2.5-0.5B-Instruct compressor, Qwen2.5-7B-Instruct reader, and a 2K-token budget.
Configuration
Macro
Attention features
All-layer (336-d)
35.87
Last-layer (14-d)
34.47
Selected Heads (14-d)
34.82
All-layer training configurations
500
35.17
Table 4: Feature and training configurations with the 0.5B compressor, Qwen reader, and a 2K-token budget. Task-level scores are given in Appendix E .
Figure 2: LongBench macro scores across Instruct compressor families with the Qwen reader and a 2K-token budget. The highest-scoring checkpoint is labeled; the dashed line marks Full Context.
Figure 3: Twelve-task macro scores by target retained length. L is each example’s original context length; the ticks denote target budgets of 0.1L – 0.5L .
Figure 4: Compressor-only cost by input-document length. Left: mean compress() latency in seconds; the inset shows Sentinel’s latency in milliseconds. Right: P95 peak allocated GPU memory. All methods were profiled on the same NVIDIA RTX 5090 GPU. Bins reflect pre-compression document length, not necessarily the length attended by each method. Input-length bins are given in tokens; the edge bins are sparsely sampled. Appendix F gives the profiling protocol.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
Reader: Qwen2.5-7B-Instruct
Empty
3.77
11.00
14.19
25.24
24.77
8.58
6.31
13.56
9.36
8.76
12.61
7.60
12.98
9.66
12.15
Full Context
29.67
44.16
52.12
59.30
47.31
31.29
32.69
23.58
24.24
61.76
17.51
11.58
38.26
30.28
36.27
LLMLingua-1
10.52
33.05
45.41
34.55
36.37
17.79
29.86
19.57
24.37
50.23
17.52
13.14
27.94
26.96
27.70
LongLLMLingua
10.30
32.75
43.32
33.60
34.35
20.49
29.99
19.90
24.11
49.95
18.31
11.68
27.65
26.65
27.40
LLMLingua-2
11.96
35.53
40.52
35.02
38.50
17.95
29.79
21.71
23.74
49.32
17.87
12.99
28.30
26.73
27.91
Appendix
Table 5: Task-level results across three readers. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
Qwen2.5-0.5B
22.73
45.19
51.07
56.16
49.07
33.12
32.32
22.72
24.17
62.64
17.45
13.82
37.39
31.30
35.87
Qwen2.5-1.5B
22.17
46.80
52.07
58.52
46.89
32.97
32.85
23.42
24.31
59.96
16.55
13.31
37.78
29.94
35.82
Qwen2.5-3B
24.37
47.01
51.15
61.17
49.97
36.58
32.33
23.43
24.32
59.58
16.90
12.70
38.93
29.73
36.63
Qwen2.5-7B
25.20
44.79
52.45
60.45
47.59
35.86
32.94
23.93
24.09
60.65
17.26
13.04
38.59
30.32
36.52
Qwen3-0.6B
22.52
45.47
52.16
49.40
48.82
30.33
32.23
21.87
23.76
61.04
17.38
13.53
36.28
30.65
34.88
Qwen3-1.7B
23.09
44.50
51.62
58.65
52.16
33.24
32.32
23.45
24.04
61.46
17.07
13.09
38.12
30.54
36.22
Appendix
Table 6: Task-level Sentinel scores across Instruct compressor families with the Qwen reader. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
All-layer (336-d)
22.73
45.19
51.07
56.16
49.07
33.12
32.32
22.72
24.17
62.64
17.45
13.82
37.39
31.30
35.87
Feature configurations
Last-layer (14-d)
20.28
45.21
50.69
54.10
46.07
31.16
31.96
21.46
23.85
58.60
17.34
12.88
36.09
29.61
34.47
Selected Heads (14-d)
20.50
45.01
50.80
55.33
47.41
30.54
32.13
21.71
23.76
60.55
16.65
13.50
36.35
30.23
34.82
Training configurations (all-layer)
500
18.11
46.07
52.05
55.95
48.84
32.61
32.62
21.41
23.91
60.38
17.14
13.00
36.84
30.17
35.17
Appendix
Table 7: Attention feature and probe-training configurations with the Qwen reader. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Raw Attention
Sentinel
Task
0.1L
0.2L
0.3L
0.4L
0.5L
0.1L
0.2L
0.3L
0.4L
0.5L
NarrativeQA
14.03
15.79
19.68
23.47
20.81
13.79
21.92
21.44
23.24
28.01
Qasper
28.46
35.47
40.55
42.42
41.81
44.65
45.85
45.90
46.10
46.56
MultiFieldQA-en
37.21
45.31
47.03
47.91
48.65
48.20
48.71
49.21
49.57
51.20
HotpotQA
31.66
36.10
41.59
41.27
46.45
58.90
58.60
59.84
57.37
58.92
2WikiMQA
27.64
35.35
35.75
39.22
40.35
49.23
46.77
45.05
44.22
50.82
Appendix
Table 8: Task-level scores at target retained fractions of the original context. Macro is the unweighted mean of the same 12 tasks in each column.
Efficient context compression is critical for retrieval-augmented question answering in resource-constrained settings, where long retrieved contexts increase latency, memory use, and LLM reader cost. We propose a lightweight encoder-only framework for query-driven sentence pruning that preserves answer-critical evidence while aggressively reducing irrelevant context. Our method learns marginal contribution scores for sentences using counterfactual training signals and optimizes a contrastive ranking objective that separates critical evidence from noncritical context. Our approach scores all sentences from a single full-context encoding, enabling fast inference with low computational overhead. Experiments show that it maintains accuracy comparable to the strongest baseline while using 3.7× less peak memory and achieving nearly 3× lower compression latency, demonstrating an effective quality--efficiency trade-off for practical resource-constrained deployment.
Thao Do, Dinh Phu Tran, An Vo +2
Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea
Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key-value (KV) cache grows with the number of processed tokens. Larger context windows also do not ensure reliable evidence use. Context compression reduces this cost, but many soft-compression methods use LLMs as compressors and rely on compact memory tokens both to preserve information and to condition the decoder. We propose SeDeM, a selective decompression framework that decouples compact memory storage from decoder conditioning. SeDeM stores context as compact hidden-state memory blocks, selects query-relevant blocks, and decompresses only the selected blocks for decoder conditioning. Thus, the decoder avoids both full-context processing and direct generation from highly compressed memory slots. On four long-context QA benchmarks, SeDeM achieves higher QA scores than the compression baselines in our main comparison in both 1B and 3B same-backbone settings, and with the 3B backbone exceeds full-context fine-tuning on three datasets. SeDeM also provides favorable quality--efficiency trade-offs, achieving 1.74--2.46× lower online time-to-first-token and 1.08--1.10× higher autoregressive decoding throughput relative to ICAE while maintaining strong answer quality.
Long-context compression is essential for reducing the cost and latency of large language model inference. However, existing methods can fragment important evidence, require additional training or alignment, and often depend on the target model for effective compression. We introduce TopoCompress, a training-free and model-agnostic framework that compresses long contexts by selecting coherent semantic spans. TopoCompress first scores each span using dense and lexical query relevance together with semantic acceleration. It then constructs a hybrid graph that connects spans based on semantic similarity and sequential adjacency, and propagates the query-guided relevance scores over the graph. Across five long-context tasks-HotpotQA, 2WikiMQA, MuSiQue, Qasper, and MultiFieldQA-en-TopoCompress consistently outperforms strong compression baselines. Notably, TopoCompress achieves performance comparable to the strongest baseline while using a 4x smaller compression budget, and provides a 1.41x smaller compression time over the fastest baseline.