Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactions

Payal Fofadiya; Sunil Tiwari

cs.CVMar 31, 2026

Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactions

Authors: Payal Fofadiya, Sunil Tiwari

Organizations: LinkedIn · eBay

Abstract

Large Language Models (LLMs) often experience performance degradation during long-running interactions due to increasing context length, memory saturation, and computational overhead. This paper presents an adaptive context compression framework that integrates importance-aware memory selection, coherence-sensitive filtering, and dynamic budget allocation to retain essential conversational information while controlling context growth. The approach is evaluated on LOCOMO, LOCCO, and LongBench benchmarks to assess answer quality, retrieval accuracy, coherence preservation, and efficiency. Experimental results demonstrate that the proposed method achieves consistent improvements in conversational stability and retrieval performance while reducing token usage and inference latency compared with existing memory and compression-based approaches. These findings indicate that adaptive context compression provides an effective balance between long-term memory preservation and computational efficiency in persistent LLM interactions

Figures & tables

Fig. 1: Architecture of the proposed adaptive context compression framework for long-running LLM interactions.

Method	Answer Acc	Retrieve Acc	Recall Acc	Coherence
MemoryBank	0.624	0.640	–	0.927
SCM	0.849	0.934	–	0.943
MemoChat	0.609	0.512	–	0.932
ChatRsum	0.486	–	–	0.921
ILSTMA [ 10 ]	0.884	0.938	0.663	0.948
Ours	0.89–0.91	0.94–0.95	0.68–0.71	0.95–0.96

TABLE I: Reported performance on long-running conversational benchmarks

Fig. 2: Comparison of answer accuracy, retrieval accuracy, recall accuracy, and coherence scores across long-running conversational benchmarks. The proposed adaptive context compression method achieves slightly improved performance compared with existing memory-based approaches.

Ref	Dataset	ACCU/ F1	Recall / Retrieval	Effici/ Scale	Metrics
[ 14 ]	LOCCO, LOCCO-L	Consistency = 4.40 $\pm$ 0.52	–	2,981 dialogue QA pairs retained	Coherence = 4.45 $\pm$ 0.78; Participation = 4.58 $\pm$ 0.86; Overall = 4.47; Consistency model validation accuracy = 98%
[ 15 ]	LOCOMO	Overall QA F1 = 51.6	Retrieval F1 = 41.0; Recall@k = 76.7	Average tokens per conversation = 16,618.1	Baseline without retrieval overall QA F1 = 22.4; Average sessions = 27.2; Average turns = 588.2
[ 22 ]	LongBench	–	–	4,750 test instances	21 tasks across 6 categories; Average length = 6,711 English words and 13,386 Chinese characters
[ 10 ]	MemoryBank, SCM	Answer accuracy = 0.884	Retrieve accuracy = 0.938; Recall accuracy = 0.663	Execution time reduction = 21.45%	Coherence = 0.948
[ 20 ]	HotpotQA, MSMARCO, SQuAD	F1 = 80.23 (HotpotQA), 53.30 (MSMARCO), 70.52 (SQuAD)	–	Compression ratio = 23.81 $\times$ –27.39 $\times$	EM = 65.49 (HotpotQA); EM = 52.10 (SQuAD)
[ 18 ]	LongBench, Ruler, InfiniteBench	LongBench average = 36.74	RepoBench-P = 48.92	Decoding speedup $>$ 9 $\times$	Extra computation = 0.01%; Extra memory usage = 0.6%

TABLE II: Quantitative comparison of datasets and reported evaluation results for long-context memory and context management research

Method	Efficiency Metric	Reported Value
ILSTMA [ 10 ]	Execution time reduction	21.45% (5.17s $\rightarrow$ 4.06s)
ATACompressor [ 20 ]	Compression ratio	23.81 $\times$ – 27.39 $\times$
LAVA [ 18 ]	Decoding speedup	$>$ 9 $\times$ (extra computation 0.01%)
Ours	Token reduction	25% – 55%
	Latency improvement	10% – 35%

TABLE III: Efficiency comparison of long-context compression and memory management methods

Fig. 3: Efficiency comparison across long-context compression methods using reported efficiency values. The proposed adaptive context compression method achieves higher overall efficiency through adaptive token reduction and latency improvement.

Method	Consistency	Coherence
LOCCO benchmark [ 14 ]	4.40 $\pm$ 0.52	4.45 $\pm$ 0.78
Ours	4.45–4.55	4.50–4.60

TABLE IV: Coherence and memory stability comparison on LOCCO

LLM Context Compression

See all 37 papers

CompLLM: Compression for Long Context Q&A
Sep 23, 2025Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran +1LLM Compression Long-Context Language Model Inference
Amazon · Center For Research in Computer Vision, University of Central Florida
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · University of Maryland · Princeton University +3
Large Language Model as Token Compressor and Decompressor
Mar 26, 2026Wenbing Li, Yiran Wang, Zikai Song +4LLM Context Compression LLM Compression

Sep 23, 2025J/K move · Enter open · S save

CompLLM: Compression for Long Context Q&A

Gabriele Berton, Jayakrishnan Unnikrishnan, Son Tran, Mubarak Shah

Amazon · Center For Research in Computer Vision, University of Central Florida

LLM Compression Long-Context Language Model Inference Long-Context QA Context Compression LLM Context Compression

Open paper AskPDF arXiv

Large Language Models (LLMs) face significant computational challenges when processing long contexts due to the quadratic complexity of self-attention. While soft context compression methods, which map input text to smaller latent representations, have shown promise, their real-world adoption is limited. Existing techniques typically compress the context as a single unit, which leads to quadratic compression complexity and an inability to reuse computations across queries with overlapping contexts. In this work, we introduce CompLLM, a soft compression technique designed for practical deployment. Instead of processing the context holistically, CompLLM divides it into segments and compresses each one independently. This simple design choice yields three critical properties: efficiency, as the compression step scales linearly with the context length; scalability, enabling models trained on short sequences (e.g., 1k tokens) to generalize to contexts of 100k tokens; and reusability, allowing compressed segments to be cached and reused across different queries. Our experiments show that with a 2x compression rate, at high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x and reduces the KV cache size by 50%. Furthermore, CompLLM achieves performance comparable to that obtained with the uncompressed context, and even surpasses it on very long sequences, demonstrating its effectiveness and practical utility.

Abstract

LLM Context Compression

Explore similar work

CompLLM: Compression for Long Context Q&A

End-to-End Context Compression at Scale

Large Language Model as Token Compressor and Decompressor