EffComp: Efficient Prompt Compression via Hybrid Reinforcement & Supervised Learning

Romtham Tungsununtham; Piyawat Lertvittayakumjorn; Atiwong Suchato; Peerapon Vateekul

cs.CLNov 2, 2026

EffComp: Efficient Prompt Compression via Hybrid Reinforcement & Supervised Learning

Authors: Romtham Tungsununtham, Piyawat Lertvittayakumjorn, Atiwong Suchato, Peerapon Vateekul

Organizations: Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Thailand · Google, United States

Abstract

Prompt compression is aimed at reducing input prompt lengths to enable cheaper, faster, and potentially higher-quality LLM predictions. However, existing prompt compression methods are often limited by modest compression gains, a risk of hallucination, and/or high compression latency. We propose EffComp, an Efficient prompt Compression framework for RAG-based question answering (QA). EffComp employs BERT-style document reranker and sentence selector models to allow fast extractive prompt compression at the sentence level. Its extractive nature prevents hallucinations in the compressed prompts. Additionally, the training process using a hybrid reinforcement and supervised learning approach is designed to optimize the compression ratio while preserving LLM accuracy. Experiments on various QA datasets demonstrate that EffComp outperforms state-of-the-art prompt compression methods achieving the best trade-off between compression ratio and accuracy with minimal latency. This makes it practical for real-world applications.

Figures & tables

Figure 1: The inference workflow of EffComp . The top row in the figure illustrates where EffComp is in the prediction pipeline. In Step 1, retrieved documents are filtered, reordered, and concatenated. In Step 2, they are decomposed into sentences, from which necessary ones are selected and concatenated to form the compressed context for LLM prediction.

Figure 2: A single epoch of the final finetuning (Phase II). In each iteration, we perform reinforcement learning, followed by supervised learning. Reward normalization encourages higher compression when some predictions are correct and lower compression otherwise.

Method	HQA				2WikiQA				NQ				TQA
	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Raw Documents	1.0x	34.6	0.00	0.00	1.0x	29.1	0.00	0.00	1.0x	30.1	0.00	0.00	1.0x	77.4	0.00	0.00
Reranker-only methods
BGE reranker	3.0x	35.2	0.39	0.46	3.0x	23.8	0.27	0.35	3.0x	29.6	0.33	0.41	3.0x	79.9	0.77	0.73
Jina reranker	3.0x	34.6	0.38	0.46	3.0x	24.4	0.28	0.36	3.0x	30.6	0.34	0.42	3.0x	80.1	0.77	0.73
GTE reranker	3.0x	34.7	0.38	0.46	3.0x	23.9	0.27	0.35	3.0x	30.6	0.34	0.42	3.0x	80.4	0.77	0.73

Table 1: Results on four QA benchmarks for Gemma-2-9B-IT. We report compression ratio (

1/\tau

), accuracy, and

\mathrm{C}_{\beta}

for

\beta\in\{0.5,1\}

. Higher is better. Best values are in bold.

Multi-hop
Method	HQA					2WikiQA					MuSiQue
	$1/\tau$	Acc	F1	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	F1	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	F1	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Raw Documents	1.0x	36.5	46.6	0.00	0.00	1.0x	40.5	43.7	0.00	0.00	1.0x	20.5	26.5	0.00	0.00
DAC	6.5x	25.0	34.9	0.29	0.39	5.4x	24.0	28.4	0.28	0.37	6.7x	7.0	11.8	0.09	0.13
LLMLingua2	5.4x	33.0	42.0	0.37	0.47	5.7x	27.0	30.3	0.31	0.41	5.5x	16.0	21.8	0.19	0.27
LongLLMLingua	3.5x	32.0	41.9	0.36	0.44	3.5x	38.5	43.9	0.42	0.5	3.5x	15.5	21.1	0.18	0.25

Table 2: Results on LongBench QA datasets (out of distribution) using Gemma-2-9B-IT as the target LLM. The baselines are evaluated at a 5x compression ratio. We report compression ratio (

1/\tau

), accuracy, token level F1 score, and

\mathrm{C}_{\beta}

. Higher is better, and best values are in bold.

Panel of Figure 3 — Figure 3: Ablation study of different training strategies for EffComp on HQA. The plots compare (left) validation accuracy and (right) compression rate $1-\tau=1-|\mathbb{C}|/|\mathbb{D}|$ during validation across ten epochs. For both metrics, the higher values are better.

Target LLM: Claude Sonnet 4.5
Method	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Raw Documents	1.0x	28.0	0.00	0.00
LLMLingua2	5.4x	40.2	0.45	0.54
DAC	5.1x	40.2	0.45	0.54
CPC	5.6x	42.4	0.47	0.56
LongLLMLingua	5.1x	43.6	0.48	0.57

Table 3: Performance of prompt compression methods with commercial LLMs on HQA.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Input and output of the sentence selector. The input begins with the [CLS] token and the query

q

, then followed by the candidate sentences. Each sentence is prepended with a special [SEN] token. The [SEP] tokens were inserted after the query

q

and after the last token of the last candidate sentence. The sentence selector produces token-level outputs, but only predictions at the [SEN] positions are used to determine whether each corresponding sentence is retained or discarded.

Base datasets
Dataset	Train	Validation	Test
HotpotQA	90,447	–	7,405
2WikiMultiHopQA	167,454	–	12,576
Natural Questions	152,148	–	5,499
TriviaQA	138,384	–	17,944
Intermediate finetuning

Appendix

Table 4: Number of examples in each dataset before and after filtering. The base dataset refers to the raw data, while the intermediate finetuning and final finetuning stages indicate the dataset sizes used in each phase.

Question (from HQA): Stein had a part in the 1994 superhero fantasy film directed by whom
Ground-truth answer : Charles Russell
Method	$1/\tau$ $\uparrow$	$\|\mathbb{C}\|$ $\downarrow$	Compressed Context	LLM Prediction
Long-LLMLingua	5.01x	758	The Mask (film) The Mask is a 1994 American fantasy superhero comedy film directed by Charles Russell, produced by Bob Engelman […] Ben Stein Benjamin Jeremy (born November 25, 1944) is an American writer, […] He is well- screen econom teacher "Ferris Bueller’s Day Off" (1986) and as Dr. Arthur Neuman in "The Mask" (1994)Son […]	Charles Russell ✓
Recomp-Abstractive	189.75x	20	Ben Stein had a part in the 1994 superhero fantasy film "The Shadow" directed by Russell Mulcahy.	Russell Mulcahy ✗
CompAct	49.93x	76	The Shadow is a 1994 American superhero film […] directed by Russell Mulcahy, that stars Alec Baldwin. The film co-stars John Lone, Penelope Ann Miller, […] It was released to theaters on July 1, 1994 and received generally mixed reviews.	Russell Mulcahy ✗
EffComp (BGE)	43.13x	88	He is most well-known on screen as the economics teacher in "Ferris Bueller’s Day Off" (1986) and as Dr. Arthur Neuman in "The Mask" (1994) and "Son of The Mask (film) The Mask is a 1994 American fantasy superhero comedy film directed by Charles Russell, produced by Bob Engelman […]	Charles Russell ✓

Appendix

Table 5: Examples of compressed contexts and predictions from LongLLMLingua, Recomp-Abstrctive, CompAct, and EffComp (with BGE reranker).

|\mathbb{C}|

denotes the number of tokens of the compressed context. ✓ indicates a correct prediction, while ✗ indicates an incorrect prediction.

Method	2WikiQA				NQ				TQA
	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Gemma-2-9B-IT
No reranker (in-distribution)	16.1x	29.5	0.34	0.45	23.6x	31.7	0.37	0.48	48.4x	78.9	0.82	0.87
No reranker (out-of-distribution)	11.7x	28.9	0.34	0.44	9.8x	31.1	0.36	0.46	32.8x	78.9	0.82	0.87
BGE reranker (in-distribution)	30.3x	25.4	0.30	0.40	36.4x	29.5	0.34	0.45	59.5x	80.1	0.83	0.88
BGE reranker (out-of-distribution)	19.4x	25.0	0.29	0.40	14.1x	29.0	0.34	0.44	49.6x	79.5	0.83	0.88

Appendix

Table 6: Cross-dataset generalization of EffComp using Gemma-2-9B-IT and Llama-3.1-8B-Instruct as target models, evaluated both with and without the document reranker. We report the mean compression ratio (

1/\tau

), accuracy (Acc), and

\mathrm{C}_{\beta}

. The in-distribution rows correspond to training on each dataset individually, while the out-of-distribution rows correspond to training only on HQA and directly applying the model to other datasets. For all of these metrics, higher values are better. Bold numbers indicate the best performance in each setting.

Method	HQA				2WikiQA				NQ				TQA
	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Raw Documents	1.0x	24.9	0.00	0.00	1.0x	16.7	0.00	0.00	1.0x	25.3	0.00	0.00	1.0x	69.7	0.00	0.00
Reranker-only methods
BGE	3.0x	29.6	0.33	0.41	3.0x	19.2	0.22	0.30	3.0x	26.6	0.30	0.38	3.0x	75.3	0.73	0.71
Jina	3.0x	29.2	0.33	0.41	3.0x	19.3	0.22	0.30	3.0x	28.1	0.32	0.40	3.0x	75.6	0.74	0.71
GTE	3.0x	29.0	0.33	0.40	3.0x	19.2	0.22	0.30	3.0x	28.4	0.32	0.40	3.0x	75.9	0.74	0.71

Appendix

Table 7: Results on four QA benchmarks. We report the mean of compression ratio (

1/\tau

), accuracy (Acc),

\mathrm{C}_{\beta}

for Llama-3.1-8B-Instruct as the target model. For all of these metrics, higher values are better. The best results in each column are marked in bold.

Method	GovReport		Multi-News		QMSum
	$1/\tau$	ROUGE-L	$1/\tau$	ROUGE-L	$1/\tau$	ROUGE-L
Raw documents	1.0x	18.8	1.0x	18.2	1.0x	19.2
LLMLingua2	5.2x	16.8	5.7x	16.8	5.5x	17.6
DAC	6.0x	17.4	4.9x	17.1	6.3x	17.2
CPC	5.1x	17.2	4.7x	16.3	5.2x	17.7
LongLLMLingua	4.6x	19.0	3.1x	18.1	3.8x	19.0

Appendix

Table 8: Generalization of EffComp to long-generation tasks from LongBench. The EffComp model is trained on HQA and directly applied to GovReport, Multi-News, and QMSum without task-specific training. We report the mean compression ratio (

1/\tau

) and ROUGE-L score. Higher values are better for both metrics. Bold numbers indicate the best performance in each setting.

Retriever	Step 1 (BGE)	Step 2	$1/\tau$	Acc	$\mathbf{C}_{0.5}$	$\mathbf{C}_{1}$
Gemma-2-9B-IT
Contriever-MS MARCO (Top 30) ∗	Top 10	✓	41.1x	35.7	0.41	0.52
Voyage 4 Lite (Top 6)	–	–	5.0x	29.3	0.34	0.43
Voyage 4 Lite (Top 6)	–	✓	37.3x	30.2	0.35	0.46
Voyage 4 Lite (Top 30)	Top 6	✓	51.6x	35.3	0.40	0.52
F2LLM-v2-8B (Top 6)	–	–	5.0x	32.6	0.37	0.46

Appendix

Table 9: Performance of recent retrieval methods with and without EffComp on 1,000 HQA examples. Step 1 denotes BGE reranking, where “Top

k

” indicates the number of documents retained by the reranker. Step 2 denotes the sentence selector, where ✓indicates that sentence selection is applied. ∗ denotes the retrieval configuration mainly used throughout our paper.

Model	Method	Entailment	Neutral	Contradiction
	CompAct	81.2%	15.8%	3.0%
Gemini 2.5 Flash Lite	Recomp-Abstractive	45.7%	35.4%	18.9%
	EffComp (BGE)	86.0%	11.4%	2.6%
	CompAct	50.6%	1.3%	48.1%
GPT-4.1 nano	Recomp-Abstractive	43.7%	0.6%	55.8%
	EffComp (BGE)	61.8%	0.2%	38.0%

Appendix

Table 10: NLI label counts on HQA (7,405 examples). Higher entailment and lower neutral/contradiction indicate better faithfulness and fewer hallucinations.

Method	Comp (s)	Read (s)	Total (s)
Raw documents	–	1.84	1.84
LLMLingua2	0.28	0.84	1.12
DAC	1.46	0.79	2.25
CPC	0.26	0.81	1.07
LongLLMLingua	2.31	0.90	3.21
Recomp-Extractive	0.19	0.74	0.93

Appendix

Table 11: Compression latency, LLM read latency, and total latency per example averaged over 500 HQA examples using Gemma-2-9B-IT. Bold values denote the lowest metric in each column.

Method	Compression TFLOPs
LLMLingua2	3.3
DAC	35.2
CPC	8.4
LongLLMLingua	252.2
Recomp Extractive	2.7
CompAct	65.3

Appendix

Table 12: Compression computational cost in TFLOPs averaged over 500 HQA examples with a batch size of 1. Lower values indicate lower computational cost.

Method	Compressor Memory (MB)	LLM Inference Memory (MB)
Raw documents	–	11,874.37
LLMLingua2	2,588.44	7,158.68
DAC	23,161.87	6,890.60
CPC	2,969.61	6855.91
LongLLMLingua	16,129.06	6,935.18
Recomp-Extractive	945.46	6,619.45

Appendix

Table 13: Peak GPU memory usage (in MB) for each compressor and the downstream LLM (Gemma-2-9B-IT). Memory is measured during the model forward pass. For Recomp-Abstractive, the peak across 500 examples is reported due to the lack of per-sample measurement.

Method	HQA				NQ
	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$	$1/\tau$	Acc	$\mathrm{C}_{0.5}$	$\mathrm{C}_{1}$
Raw Documents	1.0x	34.6	0.00	0.00	1.0x	30.1	0.00	0.00
Reranker-only methods
BGE reranker	3.0x	35.2	0.39	0.46	3.0x	29.6	0.33	0.41
Jina reranker	3.0x	34.6	0.38	0.46	3.0x	30.6	0.34	0.42
GTE reranker	3.0x	34.7	0.38	0.46	3.0x	30.6	0.34	0.42

Appendix

Table 14: Results on two QA benchmarks for Gemma-2-9B-IT. We report compression ratio (

1/\tau

), accuracy, and

\mathrm{C}_{\beta}

. Higher is better. Best values are in bold.

Method	Acc	$1-\tau$	$\mathbf{C}_{0.5}$	$\mathbf{C}_{1}$
Phase I*	82.3	86.4 (%)	0.83	0.84
Phase I* + Phase II	88.6	83.3 (%)	0.87	0.86
Phase I + Phase II	91.6	75.6 (%)	0.88	0.83

Appendix

Table 15: Comparison of different Phase I labeling strategies on the validation set. Phase I* uses gold supporting sentences to construct the intermediate finetuning labels. We report accuracy (Acc), compression rate, and

\mathrm{C}_{\beta}

. Higher values are better, and best values are in bold.

Method	Acc	$1-\tau$	$\mathbf{C}_{0.5}$	$\mathbf{C}_{1}$
[CLS]	93.5	67.4 (%)	0.87	0.78
[SEN]	91.6	75.6 (%)	0.88	0.83

Appendix

Table 16: Comparison between [CLS] and [SEN] tokens for sentence-level representation on the validation set. We report accuracy (Acc), compression rate, and

\mathrm{C}_{\beta}

. Higher values are better, and best values are in bold.

Panel of Figure 5 — Figure 5: Ablation study of different training strategies for EffComp on three QA datasets. The plots compare (left) validation accuracy and (right) compression rate $1-\tau=1-|\mathbb{C}|/|\mathbb{D}|$ during validation across five epochs. For both metrics, the higher values are better.

Prompt Compression

Prompt compression reduces the tokens sent to language models while trying to preserve the information needed for accurate, efficient responses.

See all 36 papers

Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
Apr 3, 2026Cornelius Kummer, Lena Jurkschat, Michael Färber +1Prompt Compression LLM Inference Efficiency
ScaDS.AI Dresden/Leipzig, CIDS, TU Dresden
PC-SubMax: Efficient Prompt Compression via Regularized Submodular Maximization
Sep 26, 2026Ziyi Zhang, Shuang Cui, Haotian Zhang +1Prompt Compression Efficient Language Model Inference
Soochow University
Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices
Apr 27, 2026Zihuai Xu, Ruofei Hou, Yang Xu +3Retrieval-Augmented Generation On-Device Language Model Inference
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123

Apr 3, 2026J/K move · Enter open · S save

Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference

Cornelius Kummer, Lena Jurkschat, Michael Färber, Sahar Vahdati

ScaDS.AI Dresden/Leipzig, CIDS, TU Dresden

Prompt Compression LLM Inference Efficiency LLM Inference Acceleration

Open paper AskPDF arXiv

With the wide adoption of language models for IR -- and specifically RAG systems -- the latency of the underlying LLM becomes a crucial bottleneck, since the long contexts of retrieved passages lead large prompts and therefore, compute increase. Prompt compression, which reduces the size of input prompts while aiming to preserve performance on downstream tasks, has established itself as a cost-effective and low-latency method for accelerating inference in large language models. However, its usefulness depends on whether the additional preprocessing time during generation is offset by faster decoding. We present the first systematic, large-scale study of this trade-off, with thousands of runs and 30,000 queries across several open-source LLMs and three GPU classes. Our evaluation separates compression overhead from decoding latency while tracking output quality and memory usage. LLMLingua achieves up to 18% end-to-end speed-ups, when prompt length, compression ratio, and hardware capacity are well matched, with response quality remaining statistically unchanged across summarization, code generation, and question answering tasks. Outside this operating window, however, the compression step dominates and cancels out the gains. We also show that effective compression can reduce memory usage enough to offload workloads from data center GPUs to commodity cards, with only a 0.3s increase in latency. Our open-source profiler predicts the latency break-even point for each model-hardware setup, providing practical guidance on when prompt compression delivers real-world benefits.

Abstract

Prompt Compression

Explore similar work

Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference

PC-SubMax: Efficient Prompt Compression via Regularized Submodular Maximization

Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices