From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Chaogui Gou

cs.CLOct 15, 2026

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Authors: Chaogui Gou

Organizations: USTB

Abstract

Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.

Figures & tables

Survey	Organizing perspective	KV / serving	Long-term / agent memory	Explicit cost model	State transition / injection	Relation to this survey
Long-context LLM surveys ( Liu et al., 2025a ; Liu et al., 2025b )	Architecture, infrastructure, training, evaluation	●	○	◐	○	Background on long-context modeling (Sec. 2 )
Prompt compression ( Li et al., 2025c )	Hard / soft prompt compression	○	○	◐	○	Admission control for active memory (Secs. 4.1 – 4.2 )
KV-cache management ( Li et al., 2024a )	Token / model / system level	●	○	◐	○	System-level basis (Secs. 4.3 – 4.5 , 6.3 )
Agent memory mechanisms ( Zhang et al., 2024b )	Memory sources, forms, operations	○	●	○	◐	Writing and maintenance of external memory (Sec. 5 )
From Human Memory to AI Memory ( Wu et al., 2025b )	Object $\times$ form $\times$ time	◐	●	○	○	Cognitive analogy; we adopt a cost view
Cognitive Memory in LLMs ( Shan et al., 2025 )	Sensory / short-term / long-term	◐	●	○	◐	Covers KV memory without modeling transition cost

Table 1: Comparison with representative related surveys. ●: systematically discussed; ◐: partially discussed; ○: largely not discussed. Ratings reflect the authors’ reading of each survey’s structure and coverage.

Figure 1: Memory-centric lifecycle view of long-context compression. Information flows between compute-active and compute-inactive memory: writing externalizes information from the inference path, while retrieval, compression, and injection reactivate external memory. Offloaded KV caches, persistent prefix caches, and materialized KV states are intermediate states on the activation path that can be recalled directly. In-place operations compress, reorganize, or discard content within a state; per-unit holding cost increases from bottom to top.

Operator	Transition	Typical representation change	Representative methods	Main cost / risk	Section
Write / externalize	A $\to$ I (copy)	Text $\to$ summaries, structured facts, or embeddings	MemoryBank ( Zhong et al., 2023 ) , Mem0 ( Chhikara et al., 2025 ) , A-MEM ( Xu et al., 2025b )	Extraction and indexing cost; erroneous or sensitive content may be fixed	5.1 – 5.2
Offload	A $\to$ S	KV $\to$ KV (optionally quantized or low-rank)	InfiniGen ( Lee et al., 2024 ) , ShadowKV ( Sun et al., 2025a )	Host–device bandwidth and storage	6.3
Recall	S $\to$ A	KV $\to$ KV	InfiniGen ( Lee et al., 2024 ) , RetrievalAttention ( Liu et al., 2024b ) , MagicPIG ( Chen et al., 2025 )	Retrieval precision and transfer latency	6.3
Retrieve-and-inject	I $\to$ A	Text / structure $\to$ text, latent, or KV	RAG ( Lewis et al., 2020 ) , LongLLMLingua ( Jiang et al., 2024 ) , HippoRAG ( Jimenez Gutierrez et al., 2024 )	Retrieval, compression, and re-prefill cost	5.3 , 6.1 – 6.2
Materialize	I $\to$ S	Text $\to$ KV or latent	Prefix-cache precomputation, TurboRAG ( Lu et al., 2024 ) , Cartridges ( Eyuboglu et al., 2025 )	Offline compute and storage; tied to model version	6.3
Evict	A $\to$ $\mathord{\emptyset}$	—	H2O ( Zhang et al., 2023 ) , SnapKV ( Li et al., 2024b ) , StreamingLLM ( Xiao et al., 2024b )	Irrecoverable information loss	4.3

Table 2: State-transition operators. A: compute-active; S: intermediate state on the activation path; I: compute-inactive.

Method	Operator	Repr.	Selection signal / mechanism	Training	Access	Reuse
Compute-active memory (Section 4 )
Selective Context	A $\to$ A (prune)	Text	Self-information of lexical units	None	Black-box	–
LLMLingua / LongLLMLingua	A $\to$ A (prune)	Text	Small-LM perplexity under a budget; question-aware variant	None	Black-box	–
Gist Tokens	A $\to$ A (encode)	Latent	Learned gist tokens	Fine-tune	White-box	✓
AutoCompressors	A $\to$ A (encode)	Latent	Segment-wise summary vectors	Fine-tune	White-box	✓
Activation Beacon	A $\to$ A (encode)	KV	Learned activation compression	Fine-tune	White-box	–

Table 3: Representative methods coded along the analytical dimensions of Section 3 . Operators follow Table 2 (A: compute-active; S: intermediate; I: compute-inactive). Access: black-box methods work with closed-model APIs; white-box methods require model weights or internal states; system methods operate in the serving runtime. Reuse: whether the compressed or stored state is reused across requests or sessions.

Benchmark	Setting	Localization	Multi-evid. reasoning	Cross-session memory	Update / conflict	Compression-aware	Agent / tool	Cost report
NIAH / Passkey ( Yu et al., 2025b ; Munkhdalai et al., 2024 )	Synthetic long sequences	●	○	○	○	○	○	○
RULER ( Hsieh et al., 2024 )	Controlled lengths	●	◐	○	○	○	○	○
NoLiMa ( Modarressi et al., 2025 )	Non-literal matching	●	◐	○	○	○	○	○
BABILong ( Kuratov et al., 2024 )	Facts in distractor text	◐	●	○	○	○	○	○
LongBench / v2 ( Bai et al., 2024a ; Bai et al., 2024b )	Natural long documents	◐	●	○	○	○	○	○
InfiniteBench ( Zhang et al., 2024a )	Beyond 100K tokens	◐	●	○	○	○	○	○

Table 4: Coverage of representative benchmarks across lifecycle capabilities. ●: primary focus; ◐: partially covered; ○: not covered. No benchmark jointly covers KV-level compression, memory updating, and systematic cost reporting.

LLM Context Management

Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.

See all 250 papers

Sep 22, 2026

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.

Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2

Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

LLM Inference Efficiency Long-Context Language Model Inference Memory-Efficient Inference KV-Cache Compression Context Compression LLM Context Compression

Ask PDF

Jul 31, 2026

SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key-value (KV) cache grows with the number of processed tokens. Larger context windows also do not ensure reliable evidence use. Context compression reduces this cost, but many soft-compression methods use LLMs as compressors and rely on compact memory tokens both to preserve information and to condition the decoder. We propose SeDeM, a selective decompression framework that decouples compact memory storage from decoder conditioning. SeDeM stores context as compact hidden-state memory blocks, selects query-relevant blocks, and decompresses only the selected blocks for decoder conditioning. Thus, the decoder avoids both full-context processing and direct generation from highly compressed memory slots. On four long-context QA benchmarks, SeDeM achieves higher QA scores than the compression baselines in our main comparison in both 1B and 3B same-backbone settings, and with the 3B backbone exceeds full-context fine-tuning on three datasets. SeDeM also provides favorable quality--efficiency trade-offs, achieving 1.74--2.46

\times

lower online time-to-first-token and 1.08--1.10

\times

higher autoregressive decoding throughput relative to ICAE while maintaining strong answer quality.

Maryam Haghifam, Jason Cong, Yizhou Sun

University of California, Los Angeles, USA

Memory-Augmented Language Models Long-Context QA Long-Context Inference Context Compression LLM Context Compression

Ask PDF

Jun 8, 2026

End-to-End Context Compression at Scale

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt. Furthermore, many methods require the input to fit within the target model's context window, and are generally incompatible with modern production inference engines. Encoder-decoder compressors, which map a long token sequence to a shorter sequence of latent embeddings consumed by a decoder, are an appealing alternative in principle. However, existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier. In this work, we revisit encoder-decoder compression and close this gap. We first perform an architecture search, pre-training many variants from scratch to determine how best to design and train encoder-decoder compressors. Guided by our findings, we continually pre-train a family of 0.6B-encoder, 4B-decoder models on nearly 400B tokens each, at compression ratios of 1:4, 1:8, and 1:16. We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage. LCLMs also unlock new efficient downstream applications: they serve as backbones for long-horizon agents that skim a compressed long context and adaptively expand relevant segments on demand, and they can compact the model's own generated reasoning into latent memory during inference, which is nearly lossless on the full MATH test set while being highly efficient, reducing peak KV usage by up to

7.9\times

Ang Li, Sean McLeish, Haozhe Chen +12

New York University · Modal Labs · Harvard University +5

LLM Compression Long-Context Language Model Inference Language Modeling Context Compression LLM Context Compression

Ask PDF

Abstract

LLM Context Management

Explore similar work

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering

End-to-End Context Compression at Scale