BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection

Zhengpei Hu; Kai Li; Dapeng Fu; Chang Zeng; Yue Li; Yuanhao Tang; Jianqiang Huang

cs.CLMar 20, 2026

BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection

Authors: Zhengpei Hu, Kai Li, Dapeng Fu, Chang Zeng, Yue Li, Yuanhao Tang, Jianqiang Huang

Organizations: School of Computer Technology and Application, Qinghai University · Tsinghua University · Ant Group Security and Intelligence Laboratory (SIL)

Abstract

The exponential expansion of context windows in LLMs has unlocked capabilities for long-document understanding but introduced severe bottlenecks in inference latency and information utilization. Existing compression methods often suffer from high training costs or semantic fragmentation due to aggressive token pruning. In this paper, we propose BEAVER, a novel training-free framework that shifts compression from linear token removal to structure-aware hierarchical selection. BEAVER maximizes hardware parallelism by mapping variable-length contexts into dense page-level tensors via dual-path pooling, and preserves discourse integrity through a hybrid planner combining semantic and lexical dual-branch selection with sentence smoothing. Extensive evaluations on four long-context benchmarks demonstrate that BEAVER achieves comparable performance to state-of-the-art (SOTA) methods like LongLLMLingua. Notably, on the RULER benchmark, BEAVER maintains high fidelity in multi-needle retrieval where baselines deteriorate. Regarding efficiency, BEAVER reduces latency by 26.4x on 128k contexts, offering a scalable solution for high-throughput applications. Our code is available at https://cslikai.cn/BEAVER/.

Figures & tables

Figure 1: Comparison of different prompt compression paradigms for long-context LLMs.

Figure 2: Overall pipeline of BEAVER . It comprises a Segmenter, a PageEncoder, and a QueryPlanner.

Figure 3: Illustration of sentence-level smoothing. To ensure syntactic completeness, the initially retrieved span (orange solid box) is extended outward to the nearest sentence boundaries (dashed boxes).

Methods	LongBench											ZeroSCROLLS
Methods	SingleDoc	MultiDoc	Summ.	FewShot	Synth.	Code	AVG	Tokens	$1/\tau$	Lat.	Spd.	AVG	Tokens	$1/\tau$	Lat.	Spd.
2,000-token constraint
(1) Unsupervised Statistical Methods
Selective-Context	16.2	34.8	24.4	15.7	8.4	49.2	24.8	1,925	5x	47.1	0.3x	19.4	1,865	5x	47.5	0.3x
LongLLMLingua	39.0	42.2	27.4	69.3	53.8	56.6	48.0	1,809	6x	6.1	2.6x	32.7	1,753	6x	5.2	2.3x
(2) Supervised and Specialized Learning Methods

Table 1: Performance comparison on LongBench and ZeroSCROLLS benchmarks with a 2,000 and 3,000-token budget. Best results are bolded, and second best are underlined. "Tokens" reports the average number of tokens remaining after compression, and "

1/\tau

" is the average compression ratio. "Lat." and "Spd." denote Latency and Speedup, respectively.

Method	Single			Multi						FWE	QA		AVG	Avg Tokens
Method	S-1	S-2	S-3	Key-1	Key-2	Key-3	Val	Qry	VT	Freq	QA-1	QA-2	AVG	Avg Tokens
LLMLingua	100.0	5.0	4.0	7.0	6.0	12.0	6.0	4.8	59.4	90.7	15.0	37.0	28.9	3,230
LongLLMLingua	100.0	6.0	4.0	9.0	2.0	3.0	7.8	7.3	60.4	91.3	19.0	36.0	28.8	3,179
LLMLingua-2	27.0	86.0	27.0	72.0	11.0	2.0	80.3	82.3	3.4	95.3	45.0	43.0	47.9	3,132
BEAVER (ours)	100.0	100.0	100.0	99.0	88.0	41.0	99.8	99.8	78.2	93.7	50.0	55.0	83.7	3,198

Table 2: Performance of BEAVER and baselines on RULER (16k context, 3k budget). "S": Single-needle; "Multi": Multi-needle (Keys/Values) & Variable Tracking (VT); "FWE": Frequent Word Extraction; "QA": Question Answering.

Methods	Coursera QA	QuALITY	SFictionQA	TPO	LongFQA	Legal Contract QA	AVG	Tokens	$1/\tau$
LLMLingua	58.9	50.0	60.9	65.1	34.3	21.9	48.5	2,122	4 $\times$
LongLLMLingua	62.2	51.0	65.6	71.0	37.7	21.3	51.5	2,196	4 $\times$
LLMLingua-2	64.4	55.0	66.4	73.2	45.0	23.7	54.6	2,110	4 $\times$
Simply-BEAVER	63.1	52.9	74.2	72.5	39.7	27.2	54.9	2,123	4 $\times$
BEAVER (ours)	64.4	56.9	76.6	74.0	44.7	28.8	57.6	2,180	4 $\times$

Table 3: Out-of-domain evaluation on general long-context scenarios with a 2,000-token budget.

Figure 4: Comparison of inference latency versus context length for different compression methods.

Configuration	S-Doc	M-Doc	Avg.	$\Delta$
BEAVER	40.7	37.6	39.2	–
Segmenter
Page Size $M$ =32	38.5	36.3	37.4	-1.8
Page Size $M$ =128	37.5	36.1	36.8	-2.4
PageEncoder
w/o Max-Pooling	39.2	33.7	36.5	-2.7

Table 4: Ablation study on LongBench QA (2K budget).

\Delta

denotes the performance drop relative to BEAVER .

Figure 5: Scalability and robustness analysis on RULER (16k context). Performance is normalized against the Dense upper bound (dashed line represents 100%).

Task Category	# Subsets	Language	Avg. Length (tokens)	Metric	# Samples	Representative Datasets
Single-Doc QA	4	3 En, 1 Zh	3.6k – 18.4k	F1	750	NarrativeQA, Qasper, MultiFieldQA
Multi-Doc QA	4	3 En, 1 Zh	4.9k – 15.8k	F1 / Rouge-L	800	HotpotQA, DuReader, MuSiQue
Summarization	5	4 En, 1 Zh	2.1k – 15.4k	Rouge-L	1,000	GovReport, VCSUM, SAMSum
Few-Shot Learning	3	2 En, 1 Zh	5.2k – 22.3k	Acc. / F1	600	TREC, LSHT, TriviaQA
Synthetic Tasks	3	2 En, 1 Zh	6.7k – 11.1k	Accuracy (EM)	600	PassageCount, PassageRetrieval
Code Completion	2	Multi-Code	1.2k – 4.2k	Edit Sim.	1,000	LCC, RepoBench-P

Table 5: Statistics of LongBench six major task categories. Here, "En" denotes English datasets, and "Zh" denotes Chinese datasets.

Task Category	# Subsets	Language	Avg. Length (words)	Metric	# Samples	Representative Datasets
Summarization	4	En	4.9k – 10.8k	Rouge	1,378	GovReport, QMSum, SQuALITY
Question Answering	4	En	1.7k – 49.4k	F1 / Accuracy	2,000	NarrativeQA, Qasper, MuSiQue
Aggregation	2	En	5.5k – 6.8k	ES / Cidx	1,000	SpaceDigest, BookSumSort
Total	10	En	1.7k – 49.4k	—	4,378	—

Table 6: Statistics of ZeroSCROLLS task categories. The benchmark consists of 10 datasets across 3 major categories. "ES" denotes Exponential Similarity, and "Cidx" denotes Concordance Index.

Task Category	# Subsets	Language	Length (tokens)	Metric	# Samples	Representative Datasets
Retrieval (NIAH)	8	En	4k – 128k	Accuracy	Variable	S-NIAH, MK-NIAH, KV-Ret
Multi-hop Tracing	1	En	4k – 128k	Accuracy	Variable	Variable Tracking (VT)
Aggregation	2	En	4k – 128k	Accuracy	Variable	CWE, FWE
Question Answering	2	En	4k – 128k	EM / F1	Variable	SQuAD, HotpotQA
Total	13	En	4k – 128k	—	Variable	—

Table 7: Statistics of RULER task categories. RULER contains 13 configurable tasks across 4 categories. The context length is synthetic and typically evaluated from 4k to 128k tokens. "NIAH" denotes Needle In A Haystack tasks.

Task Category	# Subsets	Language	Avg. Length (tokens)	Metric	# Samples	Representative Datasets
Summarization	7	En	7.3k – 20.0k	Rouge	386	GovReport, QMSum, SPACE
Open-Ended QA	6	En	3.9k – 62.3k	Rouge / F1	764	NarrativeQA, NQ, Qasper
Multiple Choice / Exam	6	En	3.9k – 16.4k	Accuracy	957	TOEFL, QuALITY, Coursera
Code Understanding	1	En	31.6k	Accuracy	90	CodeU
Total	20	En	3.9k – 62.3k	—	2,197	—

Table 8: Statistics of L-Eval Task Categories. The suite comprises 20 datasets categorized by task type. "En" denotes English language tasks.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	ZeroSCROLLS Datasets											Efficiency
Methods	GvRp	SSFD	QMsm	SQAL	QALT	Nrtv	Qspr	MuSQ	SpDg	BkSS	AVG	Tokens	$1/\tau$	Lat.	Spd.
2,000-token constraint
(1) Unsupervised Statistical Methods
Selective-Context	19.0	8.4	9.7	12.4	47.0	12.5	21.6	11.5	41.2	11.0	19.4	1,865	5x	47.5	0.3x
LongLLMLingua	20.1	12.4	14.9	16.5	65.1	27.7	30.7	23.6	68.5	47.2	32.7	1,753	6x	5.2	2.3x
(2) Supervised and Specialized Learning Methods

Appendix

Table 9: Detailed evaluation on ZeroSCROLLS datasets. We report the performance on individual sub-tasks: GovReport (GvRp), SummScreenFD (SSFD), QMSum (QMsm), SQuALITY (SQAL), Quality (QALT), NarrativeQA (Nrtv), Qasper (Qspr), MuSiQue (MuSQ), SpaceDigest (SpDg), and BookSumSort (BkSS). "Lat." and "Spd." denote Latency and Speedup, and "-" indicates that the results were not available from the original paper.

Methods	Qwen3-8B							Tokens $\downarrow$	$1/\tau$ $\uparrow$
Methods	Coursera QA	QuALITY	SFictionQA	TPO	LongFQA	Legal Contract QA	AVG	Tokens $\downarrow$	$1/\tau$ $\uparrow$
LLMLingua	24.9	48.0	60.9	66.6	13.6	11.2	37.5	2,122	4 $\times$
LongLLMLingua	30.2	49.0	69.5	72.9	15.2	10.9	41.3	2,196	4 $\times$
LLMLingua-2	25.3	53.0	74.2	78.4	16.9	12.8	43.5	2,110	4 $\times$
BEAVER (ours)	28.3	57.4	73.4	80.3	17.7	13.4	45.1	2,180	4 $\times$

Appendix

Table 10: Out-of-domain evaluation on general long-context scenarios (L-Eval) using Qwen3-8B. All methods operate under a 2,000-token input budget constraint. Highest scores are bolded , and second highest are underlined .

Figure 6: Visualization of Single-Needle Retrieval Accuracy.

Figure 7: Visualization of Multi-Needle Retrieval Accuracy.

Method	Single			Multi						FWE	QA		AVG
Method	S-1	S-2	S-3	Key-1	Key-2	Key-3	Val	Qry	VT	Freq	QA-1	QA-2	AVG
Qwen3-0.6b
Dense	100.0	100.0	100.0	95.0	84.0	45.0	82.8	94.0	86.4	90.7	34.0	28.0	78.3
LLMLingua	100.0	5.0	4.0	7.0	5.0	10.0	6.0	4.8	45.2	91.0	3.0	20.0	25.1
LongLLMLingua	100.0	6.0	4.0	10.0	1.0	3.0	7.5	7.0	56.6	92.0	3.0	21.0	25.9
LLMLingua-2	27.0	69.0	24.0	57.0	7.0	2.0	44.0	45.8	0.6	92.0	10.0	17.0	32.9

Appendix

Table 11: Main Results on RULER benchmark (16k context). Highest scores are bolded , and second highest are underlined .

Figure 8: Qualitative comparison on Financial QA under a 500 token budget.

Figure 9: Qualitative comparison on GovReport summarization under a 500 token budget.

Figure 10: Qualitative comparison on GSM100 under a 1500 token budget.

Figure 11: Qualitative comparison on CodeU under a 1000 token budget.

LLM Context Compression

See all 37 papers

From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors
Apr 25, 2026Yitian Zhou, Chaoning Zhang, Jiaquan Zhang +6LLM Compression Long-Context Language Model Inference
School of Computer Science and Engineering University of Electronic Science and Technology of China · School of Information and Software Engineering, University of Electronic Science and Technology of China · Department of Computer Science and Engineering, Kyung Hee University +3
TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
Aug 31, 2026Daniel Agyei Asante, Yang LiLLM Compression Long-Context QA
1Iowa State University, United States
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · University of Maryland · Princeton University +3

Apr 25, 2026J/K move · Enter open · S save

From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors

Yitian Zhou, Chaoning Zhang, Jiaquan Zhang, Zhenzhen Huang+5

School of Computer Science and Engineering University of Electronic Science and Technology of China · School of Information and Software Engineering, University of Electronic Science and Technology of China · Department of Computer Science and Engineering, Kyung Hee University+3

LLM Compression Long-Context Language Model Inference LLM Context Compression

Open paper AskPDF arXiv

Long-context large language models remain computationally expensive to run and often fail to reliably process very long inputs, which makes context compression an important component of many systems. Existing compression approaches typically rely on trained compressors, dense retrieval-style selection, or heuristic trimming, and they often struggle to jointly preserve task relevance, topic coverage, and cross-sentence coherence under a strict token budget. To address this, we propose a training-free and model-agnostic compression framework that selects a compact set of sentences guided by structural graph priors. Our method constructs a sparse hybrid sentence graph that combines mutual k-NN semantic edges with short-range sequential edges, extracts a topic skeleton via clustering, and ranks sentences using an interpretable score that integrates task relevance, cluster representativeness, bridge centrality, and a cycle coverage cue. A budgeted greedy selection with redundancy suppression then produces a readable compressed context in original order. Experimental results on four datasets show that our approach is competitive with strong extractive and abstractive baselines, demonstrating larger gains on long-document benchmarks.

Abstract

LLM Context Compression

Explore similar work

From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors

TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories

End-to-End Context Compression at Scale