From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Chaogui Gou

cs.CLOct 15, 2026

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Authors: Chaogui Gou

Organizations: USTB

Abstract

Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.

Figures & tables

Survey	Organizing perspective	KV / serving	Long-term / agent memory	Explicit cost model	State transition / injection	Relation to this survey
Long-context LLM surveys ( Liu et al., 2025a ; Liu et al., 2025b )	Architecture, infrastructure, training, evaluation	●	○	◐	○	Background on long-context modeling (Sec. 2 )
Prompt compression ( Li et al., 2025c )	Hard / soft prompt compression	○	○	◐	○	Admission control for active memory (Secs. 4.1 – 4.2 )
KV-cache management ( Li et al., 2024a )	Token / model / system level	●	○	◐	○	System-level basis (Secs. 4.3 – 4.5 , 6.3 )
Agent memory mechanisms ( Zhang et al., 2024b )	Memory sources, forms, operations	○	●	○	◐	Writing and maintenance of external memory (Sec. 5 )
From Human Memory to AI Memory ( Wu et al., 2025b )	Object $\times$ form $\times$ time	◐	●	○	○	Cognitive analogy; we adopt a cost view
Cognitive Memory in LLMs ( Shan et al., 2025 )	Sensory / short-term / long-term	◐	●	○	◐	Covers KV memory without modeling transition cost

Table 1: Comparison with representative related surveys. ●: systematically discussed; ◐: partially discussed; ○: largely not discussed. Ratings reflect the authors’ reading of each survey’s structure and coverage.

Figure 1: Memory-centric lifecycle view of long-context compression. Information flows between compute-active and compute-inactive memory: writing externalizes information from the inference path, while retrieval, compression, and injection reactivate external memory. Offloaded KV caches, persistent prefix caches, and materialized KV states are intermediate states on the activation path that can be recalled directly. In-place operations compress, reorganize, or discard content within a state; per-unit holding cost increases from bottom to top.

Operator	Transition	Typical representation change	Representative methods	Main cost / risk	Section
Write / externalize	A $\to$ I (copy)	Text $\to$ summaries, structured facts, or embeddings	MemoryBank ( Zhong et al., 2023 ) , Mem0 ( Chhikara et al., 2025 ) , A-MEM ( Xu et al., 2025b )	Extraction and indexing cost; erroneous or sensitive content may be fixed	5.1 – 5.2
Offload	A $\to$ S	KV $\to$ KV (optionally quantized or low-rank)	InfiniGen ( Lee et al., 2024 ) , ShadowKV ( Sun et al., 2025a )	Host–device bandwidth and storage	6.3
Recall	S $\to$ A	KV $\to$ KV	InfiniGen ( Lee et al., 2024 ) , RetrievalAttention ( Liu et al., 2024b ) , MagicPIG ( Chen et al., 2025 )	Retrieval precision and transfer latency	6.3
Retrieve-and-inject	I $\to$ A	Text / structure $\to$ text, latent, or KV	RAG ( Lewis et al., 2020 ) , LongLLMLingua ( Jiang et al., 2024 ) , HippoRAG ( Jimenez Gutierrez et al., 2024 )	Retrieval, compression, and re-prefill cost	5.3 , 6.1 – 6.2
Materialize	I $\to$ S	Text $\to$ KV or latent	Prefix-cache precomputation, TurboRAG ( Lu et al., 2024 ) , Cartridges ( Eyuboglu et al., 2025 )	Offline compute and storage; tied to model version	6.3
Evict	A $\to$ $\mathord{\emptyset}$	—	H2O ( Zhang et al., 2023 ) , SnapKV ( Li et al., 2024b ) , StreamingLLM ( Xiao et al., 2024b )	Irrecoverable information loss	4.3

Table 2: State-transition operators. A: compute-active; S: intermediate state on the activation path; I: compute-inactive.

Method	Operator	Repr.	Selection signal / mechanism	Training	Access	Reuse
Compute-active memory (Section 4 )
Selective Context	A $\to$ A (prune)	Text	Self-information of lexical units	None	Black-box	–
LLMLingua / LongLLMLingua	A $\to$ A (prune)	Text	Small-LM perplexity under a budget; question-aware variant	None	Black-box	–
Gist Tokens	A $\to$ A (encode)	Latent	Learned gist tokens	Fine-tune	White-box	✓
AutoCompressors	A $\to$ A (encode)	Latent	Segment-wise summary vectors	Fine-tune	White-box	✓
Activation Beacon	A $\to$ A (encode)	KV	Learned activation compression	Fine-tune	White-box	–

Table 3: Representative methods coded along the analytical dimensions of Section 3 . Operators follow Table 2 (A: compute-active; S: intermediate; I: compute-inactive). Access: black-box methods work with closed-model APIs; white-box methods require model weights or internal states; system methods operate in the serving runtime. Reuse: whether the compressed or stored state is reused across requests or sessions.

Benchmark	Setting	Localization	Multi-evid. reasoning	Cross-session memory	Update / conflict	Compression-aware	Agent / tool	Cost report
NIAH / Passkey ( Yu et al., 2025b ; Munkhdalai et al., 2024 )	Synthetic long sequences	●	○	○	○	○	○	○
RULER ( Hsieh et al., 2024 )	Controlled lengths	●	◐	○	○	○	○	○
NoLiMa ( Modarressi et al., 2025 )	Non-literal matching	●	◐	○	○	○	○	○
BABILong ( Kuratov et al., 2024 )	Facts in distractor text	◐	●	○	○	○	○	○
LongBench / v2 ( Bai et al., 2024a ; Bai et al., 2024b )	Natural long documents	◐	●	○	○	○	○	○
InfiniteBench ( Zhang et al., 2024a )	Beyond 100K tokens	◐	●	○	○	○	○	○

Table 4: Coverage of representative benchmarks across lifecycle capabilities. ●: primary focus; ◐: partially covered; ○: not covered. No benchmark jointly covers KV-level compression, memory updating, and systematic cost reporting.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 250 papers

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference Efficiency Long-Context Language Model Inference
Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
Jul 31, 2026Maryam Haghifam, Jason Cong, Yizhou SunMemory-Augmented Language Models Long-Context QA
University of California, Los Angeles, USA
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · Modal Labs · Harvard University +5

Sep 22, 2026J/K move · Enter open · S save

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia, Yutaka Watanobe+1

Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

LLM Inference Efficiency Long-Context Language Model Inference Memory-Efficient Inference KV-Cache Compression Context Compression LLM Context Compression

Open paper AskPDF arXiv

Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.

Abstract

LLM Context Management

Explore similar work

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

End-to-End Context Compression at Scale