While context compression can mitigate the growing inference costs of Large Language Models (LLMs) by shortening contexts, existing methods that specify a target compression ratio or length suffer from unpredictable performance degradation, hindering their reliable deployment. We introduce a paradigm shift to Performance-oriented Context Compression (PoC), where developers specify an acceptable performance floor instead of a compression ratio. PoC employs a lightweight performance predictor to automatically find the most aggressive compression ratio that satisfies this constraint before steering an off-the-shelf compressor. We design and compare two predictor variants: a simple context-agnostic predictor and a more sophisticated context-aware one that considers the input's inherent compressibility. On both question-answering and summarization benchmarks, the context-aware predictor consistently achieves lower performance prediction error than the context-agnostic predictor, while the resulting context-aware PoC attains a superior overall performance. Our work paves the way for a more reliable, efficient, and performance-aware deployment of context compression for LLMs.
Figures & tables
Figure 1: Overview of Performance-oriented Context Compression (PoC) . PoC is a novel context compression paradigm that moves away from budget-driven compression toward performance-driven compression. Unlike Budget-oriented Context Compression (BoC), PoC compresses context to the minimum length required to meet a given performance constraint. This approach mitigates the unpredictable performance degradation inherent in BoC, thus enhancing the reliability and applicability of context compression.
Figure 2: Overview of the PoC pipeline: (1) A performance predictor first estimates the performance-compression curve for the input context. (2) Given a target performance retention, PoC then searches for the minimum compression ratio that meets the constraint. (3) Finally, an off-the-shelf compressor compresses the context using the selected ratio.
Figure 3: Context-aware predictor. It takes the context and multiple compression ratios as input to simultaneously predict multiple performance retentions. This parallel prediction approach enables the prediction of multiple performance retentions in a single forward pass, significantly accelerating the prediction speed.
Method
SearchQA
TriviaQA
Natural Questions
HotpotQA
SQuAD
PPE ↓
F1@R ↑
EM@R ↑
PPE ↓
F1@R ↑
EM@R ↑
PPE ↓
F1@R ↑
EM@R ↑
PPE ↓
F1@R ↑
EM@R ↑
PPE ↓
F1@R ↑
EM@R ↑
context-agnostic
0.156
62.3
47.4
0.122
49.2
32.0
0.168
61.9
42.0
0.162
66.3
47.0
0.148
62.8
41.8
context-aware
0.136 ( ↓12.8% )
64.7
50.4
0.113 ( ↓7.4% )
67.5
53.1
0.156 ( ↓7.1% )
62.2
42.3
0.159 ( ↓1.9% )
67.0
47.5
0.144 ( ↓2.7% )
63.0
42.4
Table 1: Performance on Question Answering benchmarks. We report the Performance Prediction Error (PPE, lower is better) and Performance at Ratio (P@R) for F1 and EM scores (higher is better). The percentage in parentheses indicates the relative reduction in PPE compared to the context-agnostic baseline.
Method
GovReport
SummScreenFD
PPE ↓
R1@R ↑
R2@R ↑
RL@R ↑
PPE ↓
R1@R ↑
R2@R ↑
RL@R ↑
context-agnostic
0.023
34.0
13.2
18.1
0.059
29.5
5.8
16.2
context-aware
0.018 ( ↓21.7% )
34.0
13.2
18.1
0.039 ( ↓33.9% )
30.3
5.8
16.5
Table 2: Performance on Summarization benchmarks. We report the Performance Prediction Error (PPE) and Performance at Ratio (P@R) for ROUGE-1/2/L scores. The percentage in parentheses shows the relative reduction in PPE for our context-aware model compared to the baseline.
Figure 4: The performance-compression curve interpolated by the context-agnostic predictor. To illustrate the sample-specific variability, we randomly plot the performance-compression curves of 3 samples. (a)–(f) present the performance-compression curves for the QA and summarization tasks, while (h) shows the unnormalized performance-compression curve (where the y-axis represents the F1 score) on TriviaQA. The unnormalized performance-compression curves for the other datasets are provided in Appendix A .
Component
Ratio
Latency (ms)
Std. Dev. (ms)
Predictor
-
1.53
0.03
Compressor
0.02
27.41
0.36
0.10
27.51
0.69
0.20
27.60
0.42
0.50
27.68
0.62
0.80
27.72
0.48
Table 3: Latency evaluation of the framework components. The predictor and compressor latencies represent the overhead, measured on 512-token chunks. The Reader latency demonstrates the inference speedup on a 32,768-token context compressed at different ratios.
Figure 5: The unnormalized performance-compression curves interpolated by the context-agnostic predictor. In subfigures (a)–(e), the y-axis represents the F1 score, while in (f) and (g), the y-axis represents the geometric mean of ROUGE-1, ROUGE-2, and ROUGE-L.
Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA · Fuqua School of Business Duke University Durham, USA · Department of Engineering Carnegie Mellon University Pittsburgh, USA +1