Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Authors: Yong Zhang, Heng Li, Yanwen Huang, Ning Cheng, Yang Guo, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
Organizations: Ping An Technology (Shenzhen) Co., Ltd., China · University of Science and Technology of China · University of Electronic Science and Technology of China
Retrieved contexts are often long, noisy, and redundant, motivating query-aware compression before answer generation. Native attention provides signals for context selection, but heterogeneous head behaviors complicate their direct aggregation into sentence-utility scores. We introduce Sentinel, a sentence-level compressor that learns to combine these signals through a signed linear readout across layers and heads. Only the readout is trained using existing QA evidence annotations, while the backbone remains frozen and sentence scoring requires no autoregressive generation. Across twelve LongBench tasks, Sentinel with a 0.5B backbone outperforms Full Context on multiple tasks under a 2,000-token budget with each of three readers: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and DeepSeek V4.1 Flash. Despite English-only QA supervision, the same readout transfers to Chinese tasks without task-specific retraining. Length-stratified profiling further shows lower compression latency and GPU memory usage than LLMLingua-2 and CPC.
Figures & tables
Figure 1: Sentinel Framework Overview. Sentinel learns a signed sentence-level readout over a frozen language model’s attention features. Retrieval-dependent QA selection provides a sample-level behavioral basis for training; existing answer spans and supporting facts supply positive sentence labels, with other sentences labeled negative. Only the probe is trained. At inference time, one non-autoregressive forward pass scores the sentences in a native context window for budgeted selection.
Method
Qwen2.5-7B-Instruct
DeepSeek V4.1 Flash
English
Chinese
Macro
English
Chinese
Macro
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
SD
MD
Sum
Avg.
Reference conditions
Empty
9.65
19.53
9.74
12.98
8.76
12.61
7.60
9.66
12.15
12.01
31.50
9.93
17.81
22.00
15.86
7.56
15.14
17.15
Full Context
41.98
45.97
26.84
38.26
61.76
17.51
11.58
30.28
36.27
48.35
64.82
27.86
47.01
63.97
17.19
12.94
31.37
43.10
Metric-Based Context Compression
Table 1: LongBench results by task category and reader. SD, MD, and Sum denote Single-Document QA, Multi-Document QA, and Summarization. Avg. averages the nine English or three Chinese tasks equally; Macro averages all 12 tasks. Full Context is uncompressed. DeepSeek thinking is disabled. Best and second-best compression results in each column are bold and underlined, respectively.
Compressor query
Answer F1 (%)
Correct
29.62
Swapped
6.92
Generic
3.16
Table 2: Query ablation on 100 common valid Qasper QA examples with a 256-token compression budget and the Qwen2.5-7B-Instruct reader. The reader always answers the target question.
Task
Sentinel
Negative weights set to zero
NarrativeQA
22.73
17.03
Qasper
45.19
45.19
MultiFieldQA-en
51.07
50.14
HotpotQA
56.16
54.97
2WikiMQA
49.07
47.59
MuSiQue
33.12
33.06
Table 3: Effect of signed probe weights. Negative coefficients are set to zero without retraining. Both variants use the Qwen2.5-0.5B-Instruct compressor, Qwen2.5-7B-Instruct reader, and a 2K-token budget.
Configuration
Macro
Attention features
All-layer (336-d)
35.87
Last-layer (14-d)
34.47
Selected Heads (14-d)
34.82
All-layer training configurations
500
35.17
Table 4: Feature and training configurations with the 0.5B compressor, Qwen reader, and a 2K-token budget. Task-level scores are given in Appendix E .
Figure 2: LongBench macro scores across Instruct compressor families with the Qwen reader and a 2K-token budget. The highest-scoring checkpoint is labeled; the dashed line marks Full Context.
Figure 3: Twelve-task macro scores by target retained length. L is each example’s original context length; the ticks denote target budgets of 0.1L – 0.5L .
Figure 4: Compressor-only cost by input-document length. Left: mean compress() latency in seconds; the inset shows Sentinel’s latency in milliseconds. Right: P95 peak allocated GPU memory. All methods were profiled on the same NVIDIA RTX 5090 GPU. Bins reflect pre-compression document length, not necessarily the length attended by each method. Input-length bins are given in tokens; the edge bins are sparsely sampled. Appendix F gives the profiling protocol.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
Reader: Qwen2.5-7B-Instruct
Empty
3.77
11.00
14.19
25.24
24.77
8.58
6.31
13.56
9.36
8.76
12.61
7.60
12.98
9.66
12.15
Full Context
29.67
44.16
52.12
59.30
47.31
31.29
32.69
23.58
24.24
61.76
17.51
11.58
38.26
30.28
36.27
LLMLingua-1
10.52
33.05
45.41
34.55
36.37
17.79
29.86
19.57
24.37
50.23
17.52
13.14
27.94
26.96
27.70
LongLLMLingua
10.30
32.75
43.32
33.60
34.35
20.49
29.99
19.90
24.11
49.95
18.31
11.68
27.65
26.65
27.40
LLMLingua-2
11.96
35.53
40.52
35.02
38.50
17.95
29.79
21.71
23.74
49.32
17.87
12.99
28.30
26.73
27.91
Appendix
Table 5: Task-level results across three readers. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
Qwen2.5-0.5B
22.73
45.19
51.07
56.16
49.07
33.12
32.32
22.72
24.17
62.64
17.45
13.82
37.39
31.30
35.87
Qwen2.5-1.5B
22.17
46.80
52.07
58.52
46.89
32.97
32.85
23.42
24.31
59.96
16.55
13.31
37.78
29.94
35.82
Qwen2.5-3B
24.37
47.01
51.15
61.17
49.97
36.58
32.33
23.43
24.32
59.58
16.90
12.70
38.93
29.73
36.63
Qwen2.5-7B
25.20
44.79
52.45
60.45
47.59
35.86
32.94
23.93
24.09
60.65
17.26
13.04
38.59
30.32
36.52
Qwen3-0.6B
22.52
45.47
52.16
49.40
48.82
30.33
32.23
21.87
23.76
61.04
17.38
13.53
36.28
30.65
34.88
Qwen3-1.7B
23.09
44.50
51.62
58.65
52.16
33.24
32.32
23.45
24.04
61.46
17.07
13.09
38.12
30.54
36.22
Appendix
Table 6: Task-level Sentinel scores across Instruct compressor families with the Qwen reader. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Method
NQA
Qasper
MF-en
Hotpot
2Wiki
MuSiQue
GovReport
QMSum
MultiNews
MF-zh
DuReader
VCSUM
EN Avg.
ZH Avg.
Macro
All-layer (336-d)
22.73
45.19
51.07
56.16
49.07
33.12
32.32
22.72
24.17
62.64
17.45
13.82
37.39
31.30
35.87
Feature configurations
Last-layer (14-d)
20.28
45.21
50.69
54.10
46.07
31.16
31.96
21.46
23.85
58.60
17.34
12.88
36.09
29.61
34.47
Selected Heads (14-d)
20.50
45.01
50.80
55.33
47.41
30.54
32.13
21.71
23.76
60.55
16.65
13.50
36.35
30.23
34.82
Training configurations (all-layer)
500
18.11
46.07
52.05
55.95
48.84
32.61
32.62
21.41
23.91
60.38
17.14
13.00
36.84
30.17
35.17
Appendix
Table 7: Attention feature and probe-training configurations with the Qwen reader. EN Avg. and ZH Avg. average the nine English and three Chinese tasks, respectively; Macro averages all 12 tasks equally.
Raw Attention
Sentinel
Task
0.1L
0.2L
0.3L
0.4L
0.5L
0.1L
0.2L
0.3L
0.4L
0.5L
NarrativeQA
14.03
15.79
19.68
23.47
20.81
13.79
21.92
21.44
23.24
28.01
Qasper
28.46
35.47
40.55
42.42
41.81
44.65
45.85
45.90
46.10
46.56
MultiFieldQA-en
37.21
45.31
47.03
47.91
48.65
48.20
48.71
49.21
49.57
51.20
HotpotQA
31.66
36.10
41.59
41.27
46.45
58.90
58.60
59.84
57.37
58.92
2WikiMQA
27.64
35.35
35.75
39.22
40.35
49.23
46.77
45.05
44.22
50.82
Appendix
Table 8: Task-level scores at target retained fractions of the original context. Macro is the unweighted mean of the same 12 tasks in each column.