KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Ziqing Man; Di Wu; Jia-Chen Gu; Junyi Zhang; Kai-Wei Chang; Nanyun Peng

cs.CLOct 19, 2026

KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Authors: Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang, Kai-Wei Chang, Nanyun Peng

Organizations: University of California, Los Angeles

Abstract

Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost. These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.

Figures & tables

Figure 1: Comparison of indirect evaluation and direct evidence evaluation on a simplified KEEP-Bench case. The upper branch shows indirect evaluation through downstream QA, which considers only the final answer and may therefore falsely accept an unfaithful compressed context. The lower branch shows KEEP-Bench ’s direct evidence evaluation, which independently checks whether each reference evidence proposition is entailed by the compressed context, thereby detecting the preservation failure.

Benchmark	Direct Eval.	Cases	Challenges
Benchmark	Direct Eval.	Cases	Multi-hop	Fictional	Parallel	Condition	Imbalance
MuSiQue-Ans ( Trivedi et al. 2022 )	✗	24,814	✓	✗	✗	✗	✗
ConditionalQA ( Sun, Cohen, and Salakhutdinov 2022 )	✗	3,427	✓	✗	✗	✓	✗
CompressionBench ( Zakazov et al. 2025 )	✗	400	✓	✗	✗	✗	✗
Information Preservation Eval. ( Łajewska et al. 2025 )	✓	900	✓	✗	✗	✗	✗
DeepAmbigQA ( Ji et al. 2025 )	✗	3,600	✓	✗	✓	✗	✗

Table 1: Comparison of KEEP-Bench with representative benchmarks. Direct Eval. indicates whether the compressed context is evaluated directly, rather than only through downstream task performance. ✓ and ✗ indicate whether a property is present in at least part of a benchmark. Fictional denotes systematic entity fictionalization. Parallel , Condition , and Imbalance correspond to the three diagnostic settings described in the main text. Cases reports the number of evaluation instances per benchmark.

Figure 2: Illustrative examples of the diagnostic settings in KEEP-Bench . Parallel evidence requires preserving evidence for two valid interpretations; conditional evidence requires preserving an explicit condition; and imbalanced evidence requires preserving dominant and minority evidence.

Setting	Cases	Avg. P.	Avg. T.
Parallel	900	22.8	2,799
Conditional	1,364	21.0	2,544
Imbalanced	1,838	9.8	3,577

Table 2: Dataset statistics for the three settings in KEEP-Bench . Avg. P. denotes the average number of paragraphs, and Avg. T. denotes the average source context length measured using the GPT-5 tokenizer.

Figure 3: Computation cost and evidence preservation under the default compression budget. Overall All-Rec versus average TFLOPs per case on a log scale. Circle area represents model size; the dashed line marks the Pareto frontier.

Figure 4: All-Rec under different compression budgets across the three evidence settings. Middle points correspond to the default budget in the main results; left and right points use

0.5\times

and

2\times

its target sentence count, and the x-axis shows the resulting average output tokens. Shown are the three strongest and three weakest general LLMs by All-Rec at the default budget and the three dedicated compressors with explicit length control (dashed lines). Panels share the same y-axis.

Model	Measure	Adjacent	Middle	Distant
(a) Evidence distance:
Qwen3-8B	Para-Rec / All-Rec	53.3 / 43.5	51.2 / 41.0	48.5 / 38.4
Qwen3-8B	$\Delta$ vs. Adjacent	–	$-2.1$ / $-2.5$	$-4.8$ / $-5.1$
Llama-3.1-8B-Instruct	Para-Rec / All-Rec	50.8 / 31.1	47.4 / 27.1	41.8 / 26.2
Llama-3.1-8B-Instruct	$\Delta$ vs. Adjacent	–	$-3.4$ / $-4.0$	$-9.0$ / $-4.9$
Gemma-4-E4B-it	Para-Rec / All-Rec	65.5 / 51.0	57.0 / 42.3	67.5 / 52.1

Table 4: Effects of evidence position on preservation. Panel (a) reports Para-Rec and All-Rec, with changes from Adjacent shown in the

\Delta

rows. Panel (b) reports recall of the first proposition in the earlier and later chains, with

\Delta=\text{Earlier Rec}-\text{Later Rec}

. Results are averaged over all applicable absolute placements and block orders within each distance condition.

	Evidence propositions missed (%)
Model	1	2	3	$\geq 4$
Gemma-4-E2B-it	15.8	31.8	27.7	24.6
Gemma-4-E4B-it	36.2	32.4	15.1	16.2
Gemma-4-12B-it	51.2	29.4	9.1	10.4
Gemma-4-31B-it	61.0	27.4	5.9	5.7

Table 5: Residual failure scope across the Gemma-4 family under the default budget, using all 991 cases with All-Rec

=0

for all four models (290 parallel, 302 conditional, and 399 imbalanced). Columns report the percentage of cases missing exactly one, two, three, or at least four evidence propositions.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Context	Answer 1	Answer 2	Both
Original	196/200 (98.0%)	191/200 (95.5%)	188/200 (94.0%)
Faithful compressed	195/200 (97.5%)	193/200 (96.5%)	192/200 (96.0%)
Unfaithful compressed	80/200 (40.0%)	54/200 (27.0%)	23/200 (11.5%)

Appendix

Table 6: Downstream performance on the 200-query filtered parallel pilot subset. “Answer 1” and “Answer 2” report semantic matches to the respective gold answers. “Both” is the primary accuracy measure and requires matching both answers.

Model

ICL

Thinking

Parallel

Conditional

Imbalanced

Science

Policy

Rate

Para-Rec

All-Rec

Rate

Cond-Rec

All-Rec

Rate

All-Rec

Rate

All-Rec

Gemma-4-E2B-it

✓

22.4

\times

8.2%

2.8%

30.2

\times

32.6%

7.9%

29.6

\times

5.0%

20.6

\times

6.8%

✓

✗

23.4

\times

6.6%

2.3%

30.7

\times

31.7%

8.3%

29.2

\times

5.9%

22.0

\times

4.1%

✗

✓

18.7

\times

27.9%

14.8%

22.2

\times

35.9%

22.9%

17.2

\times

28.9%

17.6

\times

7.6%

✗

23.4

\times

12.6%

3.6%

26.1

\times

25.4%

11.5%

21.4

\times

15.8%

23.9

\times

7.0%

Appendix

Table 7: Results of Gemma-4 models with and without ICL and thinking under the default compression budget. For each model, bold marks the best configuration on each preservation metric. The configuration with ICL and without thinking is the one used in the main experiments (Table 3 ).

Method	$\mathbf{p_{1}}$	$\mathbf{p_{2}}$	$\mathbf{p_{3}}$	$\mathbf{p_{4}}$	Retained	Para-Rec	All-Rec
BRIEF-Pro	yes	no	no	yes	2/4	no	no
RECOMP (extractive)	no	yes	no	yes	2/4	no	no
EXIT	no	no	yes	yes	2/4	no	no
Llama-3.2-1B-Instruct	no	no	no	no	0/4	no	no
Llama-3.2-3B-Instruct	no	yes	no	yes	2/4	no	no
Llama-3.1-8B-Instruct	no	yes	no	no	1/4	no	no

Appendix

Table 8: Proposition-level judgements for the parallel case study under the default budget. A yes indicates that the corresponding proposition is preserved. Para-Rec is positive when both

p_{1}

and

p_{3}

, which distinguish the two candidate referents, are preserved; All-Rec is positive only when all four propositions are preserved.

Dedicated compressors
Method	$A_{m}$ (%)	$\overline{F}_{m}$ (TFLOPs)
RECOMP (abstractive)	0.5	6.21
BRIEF-Pro	6.7	21.94
RECOMP (extractive)	4.8	1.49
EXIT	6.3	59.96
Open-weight LLMs

Appendix

Table 9: Overall All-Rec

A_{m}

and average inference computation

\overline{F}_{m}

per case under the default compression budget (RECOMP (abstractive) without length control), with prefix amortization for methods using ICL. These two quantities give the EPR values in Table 3 . Proprietary API models are excluded.

Method	Parallel			Conditional			Imbalanced
Method	0.5 $\times$	1 $\times$	2 $\times$	0.5 $\times$	1 $\times$	2 $\times$	0.5 $\times$	1 $\times$	2 $\times$
BRIEF-Pro	0.0 (133)	0.0 (202)	0.9 (332)	8.2 (143)	19.9 (201)	63.9 (332)	0.2 (186)	0.2 (327)	1.4 (586)
RECOMP (extractive)	0.0 (111)	1.4 (219)	6.6 (431)	0.5 (104)	2.5 (207)	11.6 (410)	1.5 (169)	8.1 (308)	22.1 (587)
EXIT	0.1 (109)	3.0 (217)	14.9 (430)	1.1 (99)	4.5 (200)	16.3 (402)	2.8 (191)	9.2 (342)	21.2 (633)
Llama-3.2-1B-Instruct	0.0 (106)	0.0 (171)	0.0 (306)	0.2 (66)	0.1 (120)	2.4 (237)	0.2 (147)	0.6 (230)	1.6 (353)
Llama-3.2-3B-Instruct	3.0 (119)	2.8 (202)	2.9 (331)	3.7 (72)	6.7 (124)	12.7 (232)	4.0 (112)	4.7 (173)	8.1 (274)

Appendix

Table 10: All-Rec in percent, with the average output length in tokens in parentheses, at 0.5

\times

, 1

\times

, and 2

\times

the default sentence budget. The 1

\times

columns correspond to Table 3 ; imbalanced results pool the science and policy cases.

Setting	Decisions	Exact Matches	Agreement
Parallel	100	95	95.0%
Conditional	100	91	91.0%
Imbalanced	100	93	93.0%
Overall	300	279	93.0%

Appendix

Table 11: Decision-level exact agreement between the automatic judge and manual validation judgements.

LLM Context Compression

See all 51 papers

PReM: Learning What to Preserve and When to Refresh for Context Compression
Jul 15, 2026Bohan Yu, Lei Shen, Chenxi Zhou +5LLM Inference Acceleration Long-Context Language Model Inference
Alibaba Group
MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression
Sep 23, 2026Ke Wan, Yifan Wang, Liheng Lai +1Retrieval-Augmented Generation Evidence Retrieval
Department of Computer Science University of Virginia Charlottesville, VA, USA · AfterQuery
Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
Aug 3, 2026Mengting Ai, Jingrui He, Yue GuoCoT Faithfulness Reasoning Evaluation
University of Illinois Urbana-Champaign

Jul 15, 2026J/K move · Enter open · S save

PReM: Learning What to Preserve and When to Refresh for Context Compression

Bohan Yu, Lei Shen, Chenxi Zhou, Chen Han+4

Alibaba Group

LLM Inference Acceleration Long-Context Language Model Inference KV-Cache Compression Context Compression Memory-Augmented Language Models LLM Context Compression+1

Open paper AskPDF arXiv

Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds. However, existing compression-oriented approaches, such as key-value (KV) cache compression and context compression, often either make an early decision about which contextual information to keep or rely on an external compressor. Such designs make it difficult to adapt the compressed context to the evidence needed by later reasoning steps. This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains the long context as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM uses a dedicated memory layer to make memory-selection decisions, and a special memory token <m> to trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes. Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, while maintaining a favorable balance between answer quality and inference efficiency.

Abstract

LLM Context Compression

Explore similar work

PReM: Learning What to Preserve and When to Refresh for Context Compression

MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression

Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression