Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost. These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.
Figures & tables
Figure 1: Comparison of indirect evaluation and direct evidence evaluation on a simplified KEEP-Bench case. The upper branch shows indirect evaluation through downstream QA, which considers only the final answer and may therefore falsely accept an unfaithful compressed context. The lower branch shows KEEP-Bench ’s direct evidence evaluation, which independently checks whether each reference evidence proposition is entailed by the compressed context, thereby detecting the preservation failure.
Benchmark
Direct Eval.
Cases
Challenges
Multi-hop
Fictional
Parallel
Condition
Imbalance
MuSiQue-Ans ( Trivedi et al. 2022 )
✗
24,814
✓
✗
✗
✗
✗
ConditionalQA ( Sun, Cohen, and Salakhutdinov 2022 )
✗
3,427
✓
✗
✗
✓
✗
CompressionBench ( Zakazov et al. 2025 )
✗
400
✓
✗
✗
✗
✗
Information Preservation Eval. ( Łajewska et al. 2025 )
✓
900
✓
✗
✗
✗
✗
DeepAmbigQA ( Ji et al. 2025 )
✗
3,600
✓
✗
✓
✗
✗
Table 1: Comparison of KEEP-Bench with representative benchmarks. Direct Eval. indicates whether the compressed context is evaluated directly, rather than only through downstream task performance. ✓ and ✗ indicate whether a property is present in at least part of a benchmark. Fictional denotes systematic entity fictionalization. Parallel , Condition , and Imbalance correspond to the three diagnostic settings described in the main text. Cases reports the number of evaluation instances per benchmark.
Figure 2: Illustrative examples of the diagnostic settings in KEEP-Bench . Parallel evidence requires preserving evidence for two valid interpretations; conditional evidence requires preserving an explicit condition; and imbalanced evidence requires preserving dominant and minority evidence.
Setting
Cases
Avg. P.
Avg. T.
Parallel
900
22.8
2,799
Conditional
1,364
21.0
2,544
Imbalanced
1,838
9.8
3,577
Table 2: Dataset statistics for the three settings in KEEP-Bench . Avg. P. denotes the average number of paragraphs, and Avg. T. denotes the average source context length measured using the GPT-5 tokenizer.
Figure 3: Computation cost and evidence preservation under the default compression budget. Overall All-Rec versus average TFLOPs per case on a log scale. Circle area represents model size; the dashed line marks the Pareto frontier.
Figure 4: All-Rec under different compression budgets across the three evidence settings. Middle points correspond to the default budget in the main results; left and right points use 0.5× and 2× its target sentence count, and the x-axis shows the resulting average output tokens. Shown are the three strongest and three weakest general LLMs by All-Rec at the default budget and the three dedicated compressors with explicit length control (dashed lines). Panels share the same y-axis.
Model
Measure
Adjacent
Middle
Distant
(a) Evidence distance:
Qwen3-8B
Para-Rec / All-Rec
53.3 / 43.5
51.2 / 41.0
48.5 / 38.4
Δ vs. Adjacent
–
−2.1 / −2.5
−4.8 / −5.1
Llama-3.1-8B-Instruct
Para-Rec / All-Rec
50.8 / 31.1
47.4 / 27.1
41.8 / 26.2
Δ vs. Adjacent
–
−3.4 / −4.0
−9.0 / −4.9
Gemma-4-E4B-it
Para-Rec / All-Rec
65.5 / 51.0
57.0 / 42.3
67.5 / 52.1
Table 4: Effects of evidence position on preservation. Panel (a) reports Para-Rec and All-Rec, with changes from Adjacent shown in the Δ rows. Panel (b) reports recall of the first proposition in the earlier and later chains, with Δ=Earlier Rec−Later Rec . Results are averaged over all applicable absolute placements and block orders within each distance condition.
Evidence propositions missed (%)
Model
1
2
3
≥4
Gemma-4-E2B-it
15.8
31.8
27.7
24.6
Gemma-4-E4B-it
36.2
32.4
15.1
16.2
Gemma-4-12B-it
51.2
29.4
9.1
10.4
Gemma-4-31B-it
61.0
27.4
5.9
5.7
Table 5: Residual failure scope across the Gemma-4 family under the default budget, using all 991 cases with All-Rec =0 for all four models (290 parallel, 302 conditional, and 399 imbalanced). Columns report the percentage of cases missing exactly one, two, three, or at least four evidence propositions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Context
Answer 1
Answer 2
Both
Original
196/200 (98.0%)
191/200 (95.5%)
188/200 (94.0%)
Faithful compressed
195/200 (97.5%)
193/200 (96.5%)
192/200 (96.0%)
Unfaithful compressed
80/200 (40.0%)
54/200 (27.0%)
23/200 (11.5%)
Appendix
Table 6: Downstream performance on the 200-query filtered parallel pilot subset. “Answer 1” and “Answer 2” report semantic matches to the respective gold answers. “Both” is the primary accuracy measure and requires matching both answers.
Model
ICL
Thinking
Parallel
Conditional
Imbalanced
Science
Policy
Rate
Para-Rec
All-Rec
Rate
Cond-Rec
All-Rec
Rate
All-Rec
Rate
All-Rec
Gemma-4-E2B-it
✓
✓
22.4 ×
8.2%
2.8%
30.2 ×
32.6%
7.9%
29.6 ×
5.0%
20.6 ×
6.8%
✓
✗
23.4 ×
6.6%
2.3%
30.7 ×
31.7%
8.3%
29.2 ×
5.9%
22.0 ×
4.1%
✗
✓
18.7 ×
27.9%
14.8%
22.2 ×
35.9%
22.9%
17.2 ×
28.9%
17.6 ×
7.6%
✗
✗
23.4 ×
12.6%
3.6%
26.1 ×
25.4%
11.5%
21.4 ×
15.8%
23.9 ×
7.0%
Appendix
Table 7: Results of Gemma-4 models with and without ICL and thinking under the default compression budget. For each model, bold marks the best configuration on each preservation metric. The configuration with ICL and without thinking is the one used in the main experiments (Table 3 ).
Method
p1
p2
p3
p4
Retained
Para-Rec
All-Rec
BRIEF-Pro
yes
no
no
yes
2/4
no
no
RECOMP (extractive)
no
yes
no
yes
2/4
no
no
EXIT
no
no
yes
yes
2/4
no
no
Llama-3.2-1B-Instruct
no
no
no
no
0/4
no
no
Llama-3.2-3B-Instruct
no
yes
no
yes
2/4
no
no
Llama-3.1-8B-Instruct
no
yes
no
no
1/4
no
no
Appendix
Table 8: Proposition-level judgements for the parallel case study under the default budget. A yes indicates that the corresponding proposition is preserved. Para-Rec is positive when both p1 and p3 , which distinguish the two candidate referents, are preserved; All-Rec is positive only when all four propositions are preserved.
Method
Am (%)
Fm (TFLOPs)
Dedicated compressors
RECOMP (abstractive)
0.5
6.21
BRIEF-Pro
6.7
21.94
RECOMP (extractive)
4.8
1.49
EXIT
6.3
59.96
Open-weight LLMs
Appendix
Table 9: Overall All-Rec Am and average inference computation Fm per case under the default compression budget (RECOMP (abstractive) without length control), with prefix amortization for methods using ICL. These two quantities give the EPR values in Table 3 . Proprietary API models are excluded.
Method
Parallel
Conditional
Imbalanced
0.5 ×
1 ×
2 ×
0.5 ×
1 ×
2 ×
0.5 ×
1 ×
2 ×
BRIEF-Pro
0.0 (133)
0.0 (202)
0.9 (332)
8.2 (143)
19.9 (201)
63.9 (332)
0.2 (186)
0.2 (327)
1.4 (586)
RECOMP (extractive)
0.0 (111)
1.4 (219)
6.6 (431)
0.5 (104)
2.5 (207)
11.6 (410)
1.5 (169)
8.1 (308)
22.1 (587)
EXIT
0.1 (109)
3.0 (217)
14.9 (430)
1.1 (99)
4.5 (200)
16.3 (402)
2.8 (191)
9.2 (342)
21.2 (633)
Llama-3.2-1B-Instruct
0.0 (106)
0.0 (171)
0.0 (306)
0.2 (66)
0.1 (120)
2.4 (237)
0.2 (147)
0.6 (230)
1.6 (353)
Llama-3.2-3B-Instruct
3.0 (119)
2.8 (202)
2.9 (331)
3.7 (72)
6.7 (124)
12.7 (232)
4.0 (112)
4.7 (173)
8.1 (274)
Appendix
Table 10: All-Rec in percent, with the average output length in tokens in parentheses, at 0.5 × , 1 × , and 2 × the default sentence budget. The 1 × columns correspond to Table 3 ; imbalanced results pool the science and policy cases.
Setting
Decisions
Exact Matches
Agreement
Parallel
100
95
95.0%
Conditional
100
91
91.0%
Imbalanced
100
93
93.0%
Overall
300
279
93.0%
Appendix
Table 11: Decision-level exact agreement between the automatic judge and manual validation judgements.
Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds. However, existing compression-oriented approaches, such as key-value (KV) cache compression and context compression, often either make an early decision about which contextual information to keep or rely on an external compressor. Such designs make it difficult to adapt the compressed context to the evidence needed by later reasoning steps. This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains the long context as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM uses a dedicated memory layer to make memory-selection decisions, and a special memory token <m> to trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes. Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, while maintaining a favorable balance between answer quality and inference efficiency.
Retrieval-augmented generation often relies on multiple retrieved contexts that contain substantial redundancy, motivating context compression to preserve useful information under limited input budgets. Likelihood-based compressors can account for cross-context redundancy through sequential scoring, but this makes evidence scores dependent on context order. We show that permuting the same contexts under an unchanged compressor can substantially change which supporting evidence survives compression. We attribute this sensitivity to information preemption: earlier, partially relevant contexts can absorb credit for shared information, reducing the incremental scores of later, stronger evidence and increasing its risk of removal. Controlled pair-swap interventions provide direct empirical support for this mechanism by showing that placing stronger evidence before overlapping, partially relevant contexts can improve its survival. Based on this insight, we introduce MORSE, a compression-aware method for evidence-preserving context ordering. MORSE uses reverse query likelihood to construct an evidence-first anchor and to evaluate compressed candidate outputs, enabling compression-aware selection among alternative permutations. Across multi-hop Question Answering (QA) benchmarks, compression procedures, budgets, and scoring models, MORSE improves evidence retention over reverse ordering and generally outperforms matched random search, with downstream QA gains. Our code is available at https://github.com/tbn5pj/MORSE_code
Ke Wan, Yifan Wang, Liheng Lai +1
Department of Computer Science University of Virginia Charlottesville, VA, USA · AfterQuery
KV cache compression is commonly evaluated by final-answer accuracy, implicitly assuming that preserving the answer also preserves the reasoning that supports it. We test this assumption for large reasoning models and show that it can fail: under compression, correct answers and the validity of their visible supporting rationales can be preserved at different rates. We study this failure with a controlled fixed-trace replay protocol, which holds reasoning content fixed and isolates whether compression preserves usable information from an already available trace. We evaluate ten token-eviction KV compression methods and one quantization method on three models across mathematical reasoning, scientific QA, clinical calculation, and long-context retrieval. We measure final accuracy, answer-chain consistency, and perturbation faithfulness. Across tasks, token-eviction methods can preserve competitive final-answer accuracy while substantially degrading chain support or perturbation faithfulness. We call this the answer-evidence gap. A coverage-preserving quantization control is substantially less affected, suggesting that the failure is tied less to KV memory reduction itself than to losing access to parts of the reasoning trace. Code is available at https://github.com/famous-blue-raincoat/Safe_KV_Compress.