cs.CLSep 30, 2026

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

Authors: Zhentao Tan, Jingyi Shen, Yanbo Li, Yao Liu, Yue Wu, Jieping Ye

Organizations: Alibaba Token Hub, Alibaba Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogeneous mechanism composition. This survey analyzes these developments as model-internal contextual memory. We introduce a five-dimensional lens---Memory Representation, Memory Update, Access, Readout, and Integration---describing what is represented, how it changes, what is query-eligible, how it is read, and how readouts form outputs. This lens compares overlapping research lines without imposing one computational model. We reconstruct mechanism-level developments and architectural adoption using 59 release-level records from 14 major model lineages and 11 high-performing open-weight endpoints. First, explicit-memory and recurrent-state methods retain distinct interfaces but increasingly control overlapping memory functions. Second, heterogeneous architectures increasingly coordinate across network depth: layer-wise composition distributes complementary memory processing across representational stages, while cross-layer reuse carries selected memory and routing artifacts forward. Depth thus becomes a dimension along which contextual memory is constructed and managed. Third, these developments motivate a stateful multidimensional memory-routing hypothesis: persistent memory is organized across temporal scope, network depth, substrate type, and representation granularity, while coordinated Sparse Write and Sparse Read determine what is maintained and what contributes to each query. Overall, efficient sequence architecture design increasingly concerns the organization, lifecycle, and selective use of contextual memory rather than an isolated Attention operator.

Figures & tables

Survey perspectivePrimary organizing principle
General / efficient Transformers [ 86 , 154 ]Architecture families, computational complexity, and approximation patterns.
Long-context LLMs [ 65 , 164 , 89 ]Context extension, length extrapolation, training, retrieval, and application strategies.
Efficient attention [ 149 ]Sparse and linear attention algorithms, efficiency, and deployment.
State space models [ 122 ]SSM lineage, state organization and dynamics, applications, and benchmarks.
KV-cache management [ 80 ]Cache selection, compression, eviction, quantization, and execution efficiency.
Memory-centric LLM architectures [ 193 ]Representation form, update dynamics, and persistence across a broad range of implicit and explicit memory mechanisms.
Table 1: Representative survey perspectives relevant to this work and their primary organizing principles. The table summarizes the dominant perspective of each survey category rather than its complete coverage.
Figure 1: An architectural primer for the five-dimensional memory-centric view. The left panel illustrates Memory Representation, Memory Update, Access, Readout, and Integration through typical explicit-memory and recurrent-state operations. The center and right panels map these dimensions onto representative Linear Attention and Sparse Attention blocks, respectively, while the top row places the two mechanisms in a schematic layer-wise hybrid model. In the Linear Attention example, a delta-style update maintains a fixed-size recurrent state that is read through state contraction; in the Sparse Attention example, candidate selection restricts access to a token KV cache before the core attention readout. Gates and output projections illustrate Integration. The displayed update rule, gates, and block ratio are representative design choices rather than universal properties of these families, and the colored labels denote analytical roles rather than a mandatory execution sequence.
Term or symbolMeaning in this survey
Core terminology and abbreviations
Contextual memoryModel-internal, input-dependent information retained or made available while processing the current sequence; it excludes pretrained parameters and external retrieval corpora.
KVKey–value representation associated with a token, compressed entry, or other explicit memory unit.
Softmax AttentionThe broader explicit-memory lineage in which contextual information is retained as separately addressable representations and read through normalized query–key interactions.
Sparse AttentionA research line centered on restricting the eligible token-, block-, or entry-level candidate set; most methods retain a Softmax Readout over that set.
Linear AttentionA family that represents history through one or more recurrent associative states rather than retaining the complete token-wise KV history.
Table 2: Core terminology, abbreviations, and framework-level notation used throughout the survey. Chapter-specific symbols are defined locally in the corresponding sections.
Research lineMemory substrateRep.Upd.Acc.Read.Int.
Softmax AttentionExplicit memory unitsPPSSS
Sparse AttentionExplicit token/block KVSSPSS
Linear AttentionAssociative recurrent statePPSSS
State Space ModelsStructured recurrent statePPSSS
Hybrid ArchitecturesHeterogeneous memory substratesPSSSS
Table 3: Qualitative emphasis of the surveyed research lines across the five analytical dimensions. P denotes a dimension commonly treated as a direct design target in the selected works, while S denotes a dimension inherited from the underlying mechanism or modified in particular subfamilies and extensions. The labels are intended as an interpretive guide rather than an exhaustive quantitative coding of the literature.
Figure 2: Head-sharing continuum from multi-head attention (MHA) to grouped-query attention (GQA) and multi-query attention (MQA). Each box represents one head; the number of query heads is fixed while the number of independently stored KV heads decreases.
Figure 3: Attention-map comparison between dense and sparse access. Dense Attention retains every causal pair. Structure Sparse Attention combines a fixed four-token sliding window with a sink connection to the first token. Routing Sparse Attention retains the causal lower-triangular domain, selects the diagonal by default, and chooses up to three additional tokens per row. Self-routing from the main attention and auxiliary-proxy routing through an indexer are alternative ways to produce the same type of selected support; all selected routing cells therefore share one visual style, while unselected causal cells remain unfilled.
Figure 5: Representative evidence constructions for self-routing Sparse Attention: dynamic token grouping, block aggregation, and in-sequence summary-token encoding.
Figure 6: Auxiliary-proxy Sparse Attention within one layer. Shared hidden states feed the Main Attention and Indexer paths; token- or block-level index selections determine which K/V entries reach Sparse Attention.
Figure 7: Indexer structures of DSA, MSA, and QSA, which differ mainly in where aggregation enters the index-score computation. The in-figure formulas are simplified; Eqs. ( 29 )–( 31 ) give the full definitions.
Retain categoryPeriodUnified formRepresentative methods
No decay2020–2025αth=1\alpha_{t}^{h}=1Linear Transformer, DeltaNet, DeltaProduct
Fixed head-wise scalar2023–2024αh∈R\alpha^{h}\in\mathbb{R}RetNet, Lightning Attention-2
Fixed channel-wise vector2024αh∈Rdk\alpha^{h}\in\mathbb{R}^{d_{k}}RWKV-5
Input-dependent head-wise scalar2024αth=fα(xt)∈R\alpha_{t}^{h}=f_{\alpha}(x_{t})\in\mathbb{R}GDN
Input-dependent channel-wise vector2023–2026αth=fα(xt)∈Rdk\alpha_{t}^{h}=f_{\alpha}(x_{t})\in\mathbb{R}^{d_{k}}GLA, RWKV-6/7, HGRN2, KDA, GDN2, EDA
Table 4: Evolution of Retain / Decay mechanisms in Linear Attention.
Erase categoryPeriodUnified formRepresentative methods
No explicit erase2020–2024mte=0, Et=0m_{t}^{e}=0,\ E_{t}=0Linear Transformer, RetNet, GLA, RWKV-5/6, HGRN2, Lightning Attention-2
Coupled corrective erase2021–2025mte=1, Ute=βtkt, Rte=ktm_{t}^{e}=1,\ U_{t}^{e}=\beta_{t}k_{t},\ R_{t}^{e}=k_{t}DeltaNet, GDN, KDA, DeltaProduct
Decoupled erase2025–2026UteU_{t}^{e} and RteR_{t}^{e} need not be aligned and/or mte>1m_{t}^{e}>1RWKV-7, GDN2, EDA
Table 5: Evolution of Erase / Correct mechanisms in Linear Attention.
Write categoryPeriodUnified formRepresentative methods
Direct write2020–2024Atw=kt, Ctw=vtA_{t}^{w}=k_{t},\ C_{t}^{w}=v_{t}Linear Transformer, RetNet, GLA, RWKV-5/6, Lightning Attention-2
Corrective write2021–2025Atw=βtkt, Ctw=vtA_{t}^{w}=\beta_{t}k_{t},\ C_{t}^{w}=v_{t}DeltaNet, GDN, KDA, DeltaProduct
Erase–write decoupling2025–2026Separate control of erase and write address or contentRWKV-7, GDN2, EDA
Table 6: Evolution of Write / Commit mechanisms in Linear Attention.
MethodDecayEraseWriteYearUnified update rule
Linear Transformer [ 74 ]No decayNo explicit eraseDirect2020St=St−1+ktvt⊤S_{t}=S_{t-1}+k_{t}v_{t}^{\top}
DeltaNet [ 140 , 179 ]No decayCoupled correctiveCorrective2021St=(I−βtktkt⊤)St−1+βtktvt⊤S_{t}=(I-\beta_{t}k_{t}k_{t}^{\top})S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}
RetNet [ 148 ]Fixed head-wiseNo explicit eraseDirect2023St=αhSt−1+ktvt⊤S_{t}=\alpha^{h}S_{t-1}+k_{t}v_{t}^{\top}
GLA [ 178 ]Input-dependent channel-wiseNo explicit eraseDirect2023St=DtSt−1+ktvt⊤S_{t}=D_{t}S_{t-1}+k_{t}v_{t}^{\top} , Dt=Diag⁡(fα(xt))D_{t}=\operatorname{Diag}(f_{\alpha}(x_{t}))
RWKV-5 [ 124 ]Fixed channel-wiseNo explicit eraseDirect2024St=Diag⁡(αh)St−1+ktvt⊤S_{t}=\operatorname{Diag}(\alpha^{h})S_{t-1}+k_{t}v_{t}^{\top}
RWKV-6 [ 124 ]Input-dependent channel-wiseNo explicit eraseDirect2024St=DtSt−1+ktvt⊤S_{t}=D_{t}S_{t-1}+k_{t}v_{t}^{\top} , Dt=Diag⁡(fα(xt))D_{t}=\operatorname{Diag}(f_{\alpha}(x_{t}))
Table 7: Representative recurrent update rules.
MethodPer-token objective Lt\mathcal{L}_{t}Resulting update
Linear Transformer [ 74 ]−⟨St−1⊤kt,vt⟩-\left\langle S_{t-1}^{\top}k_{t},v_{t}\right\rangleSt=St−1+ktvt⊤S_{t}=S_{t-1}+k_{t}v_{t}^{\top}
RetNet [ 148 ]−⟨St−1⊤kt,vt⟩+12∥1−αh St−1∥F2-\left\langle S_{t-1}^{\top}k_{t},v_{t}\right\rangle+\frac{1}{2}\left\|\sqrt{1-\alpha^{h}}\,S_{t-1}\right\|_{F}^{2}St=αhSt−1+ktvt⊤S_{t}=\alpha^{h}S_{t-1}+k_{t}v_{t}^{\top}
GLA [ 178 ]−⟨St−1⊤kt,vt⟩+12∥I−Dt St−1∥F2-\left\langle S_{t-1}^{\top}k_{t},v_{t}\right\rangle+\frac{1}{2}\left\|\sqrt{I-D_{t}}\,S_{t-1}\right\|_{F}^{2}St=DtSt−1+ktvt⊤S_{t}=D_{t}S_{t-1}+k_{t}v_{t}^{\top}
HGRN2 [ 127 ]−⟨St−1⊤(1−αt),it⟩+12∥I−Dt St−1∥F2-\left\langle S_{t-1}^{\top}(\mathbf{1}-\alpha_{t}),i_{t}\right\rangle+\frac{1}{2}\left\|\sqrt{I-D_{t}}\,S_{t-1}\right\|_{F}^{2}St=DtSt−1+(1−αt)it⊤S_{t}=D_{t}S_{t-1}+(\mathbf{1}-\alpha_{t})i_{t}^{\top}
DeltaNet [ 140 , 179 ]βt2∥St−1⊤kt−vt∥22\frac{\beta_{t}}{2}\left\|S_{t-1}^{\top}k_{t}-v_{t}\right\|_{2}^{2}St=(I−βtktkt⊤)St−1+βtktvt⊤S_{t}=(I-\beta_{t}k_{t}k_{t}^{\top})S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}
GDN [ 177 ]βt2∥S~t−1⊤kt−vt∥22\frac{\beta_{t}}{2}\left\|\widetilde{S}_{t-1}^{\top}k_{t}-v_{t}\right\|_{2}^{2} , optimized at S~t−1=αtSt−1\widetilde{S}_{t-1}=\alpha_{t}S_{t-1}St=αt(I−βtktkt⊤)St−1+βtktvt⊤S_{t}=\alpha_{t}(I-\beta_{t}k_{t}k_{t}^{\top})S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}
Table 8: Survey-derived equivalent per-token objectives corresponding to representative update rules.
AspectSSESDM
Unified stateSt∈RN×dvS_{t}\in\mathbb{R}^{N\times d_{v}} , N=MdkN=Md_{k} ; each row is a dvd_{v} -dimensional memory unit
State organizationMM fixed groups with dkd_{k} rows per groupFlat row table without group boundaries
Write selectionGroup Top- KgK_{g} plus within-group coefficientsIndependent PKM write router selecting Top- WW rows
Read selectionRead over selected groups and row supportIndependent PKM read router selecting Top- RR rows
Read/write relationShared group structure; strongly related supportsIndependent read and write routers
Capacity expansionIncrease the number of groupsDirectly increase the number of flat rows
Table 9: Comparison of SSE and SDM under a unified row-memory view.
Figure 8: Comparison of temporal grouping strategies in Log-Linear Attention, DLA, and MHLA. Log-Linear Attention deterministically organizes history into multiscale intervals, DLA forms irregular content-adaptive segments under a bounded state budget, and MHLA preserves a flat collection of fixed-granularity token blocks with query-block-conditioned mixing.
AspectLog-Linear AttentionDLAMHLA
Segmentation basisPosition-dependent Fenwick or power-of-two scheduleContent-dependent State Information ScoreFixed token-block partition
Segment formMultiscale contiguous intervalsUnequal-length semantic intervalsFlat, usually equal-granularity blocks
Number of statesMt=O( ⁣log⁡t)M_{t}=O(\!\log t)Mt≤KM_{t}\leq KDetermined by the number of retained blocks
State managementHierarchical binary-carry mergingDynamic creation and information-based mergingIndependent block summaries without intrinsic merging
Temporal resolutionFine for recent and coarse for distant historyAllocated according to semantic changeApproximately uniform across blocks
Cross-state IntegrationQuery-dependent integration of temporal-scale readoutsIntegration of readouts from dynamically constructed segment statesQuery-block-dependent integration of block-summary readouts
Table 10: Comparison of representative Temporal Expansion mechanisms.
Figure 9: A state-space update at one sequence position. The new state supplies both the current readout and the memory carried to the next position.
Figure 10: Shared and input-conditioned operators applied to the same input sequence. Both systems update their state at every position. The upper lane reuses learned transition, write and read operators; the lower lane generates control quantities from each input using fixed generator weights. In Mamba [ 55 ] , the generated step size modulates the effective discrete transition. Solid paths carry content or state, and dashed orange paths indicate control dependencies.
Figure 11: Two views of the SSD kernel in Equations ( 57 )–( 58 ) [ 28 ] . The highlighted recurrent path and matrix entry M42M_{42} identify the same contribution, from input u2u_{2} to readout r4r_{4} .
Kernel choiceState controlComputational interface
Learned time-invariant dynamics: S4, S4D, S5 [ 56 , 57 , 146 ]Shared operators govern input writing, temporal propagation and readout.Convolution or structured scans process sequences; recurrent decoding applies the same learned dynamics to the evolving state.
Selective dynamics: Mamba [ 55 ]Input-generated step sizes and write/read maps regulate the effective operators.A selective scan processes the input-dependent recurrence.
SSD kernel: Mamba-2 [ 28 ]A head-wise scalar transition combines with input-dependent write and read directions.Recurrent and structured-matrix views support block computation and state-based decoding.
Table 11: Representative SSM dynamics and computational interfaces. The Mamba-2 SSD kernel is a selective parameterization with a head-wise scalar transition.
Figure 12: Four ways to refine an SSM layer: (a) enlarge the state, (b) add read–write directions, (c) combine shared and expert temporal components as in HADES [ 144 ] , and (d) condition input-injection directions as in MuonSSM [ 114 ] . Panels (c,d) are conceptual sketches of the mechanisms described in the text.
Design questionSSM mechanismComputational or state effect
How should the state propagate and adapt?Structured dynamics and input-dependent controls.Temporal propagation, selective writing and readout.
How should the recurrence be computed?SSD recurrent and matrix views; block computation.Structured sequence computation and a fixed-size decoding state.
How should the state communicate?Multichannel interfaces: S5, Mamba-3 MIMO and MIMOMamba.Richer input–output interaction with the retained state.
How should heads and updates be organized?HADES head organization; MuonSSM update geometry.Complementary temporal roles; conditioned input-injection directions.
Table 12: SSM design questions, mechanisms and roles. The fixed-size decoding state assumes a fixed head count and fixed state dimensions.
Figure 13: Four composition granularities of Hybrid Architecture and their typical intervention scopes under the five-dimensional memory-centric view. The schematics show where heterogeneous mechanisms are combined. The five-cell columns correspond to Memory Representation, Memory Update, Access, Readout, and Integration. Dark, medium, and light blocks denote dimensions commonly modified directly by hybrid composition, affected in a method-dependent manner, or primarily inherited from the constituent mechanisms, respectively. The profiles are qualitative summaries rather than quantitative literature counts.
Layer allocationTraining strategyPeriodRepresentative methods
Predefined allocationFrom-scratch training2024–2026Griffin [ 29 ] ; Jamba [ 85 ] ; Samba [ 138 ] ; Zamba [ 51 ] ; Zamba2 [ 50 ] ; HySparse [ 43 ] ; HySparse2 [ 165 ]
Direct diagnosis selectionPretrained-model transfer2024–2026LightTransfer [ 191 ] ; Priming [ 20 ]
Alignment-based selectionPretrained-model transfer2025–2026Jet-Nemotron / PostNAS [ 59 ] ; HypeNet / HALO [ 23 ]
Iterative distillation-guided selectionPretrained-model transfer2025KL-Guided Layer Selection [ 83 ]
Table 13: Layer-wise Hybrid designs grouped by layer-allocation policy.
Head allocationPeriodAllocation characteristicRepresentative methods
Fixed head allocation2024–2025A predefined ratio is used across Hybrid layers.Hymba [ 39 ] ; Falcon-H1 [ 194 ]
Function-aware head selection2024–2026Heads with retrieval or long-range functions retain full attention, while the remaining heads use streaming, sparse, or recurrent paths.DuoAttention [ 169 ] ; HydraHead [ 151 ]
Input-adaptive head routing2026A test-time router assigns heads to full-attention or sparse-attention modes according to the input, allowing the overall sparsity ratio to vary dynamically.Elastic Attention [ 153 ]
Depth-adaptive head allocation2026The hybrid ratio is varied across depth.Head-wise Hybridization [ 143 ]
Table 14: Head-wise Hybrid designs grouped by the policy used to determine the head allocation.
Fusion interfaceUnified formPeriodRepresentative methods
Readout-level branch fusionrthyb=Fuse⁡r({rt(p)})r_{t}^{\mathrm{hyb}}=\operatorname{Fuse}_{r}(\{r_{t}^{(p)}\})2022–2026Block-Recurrent Transformer [ 67 ] ; Memorizing Transformer [ 166 ] ; Infini-attention [ 113 ] ; DART [ 126 ]
Output-level branch fusionothyb=Fuse⁡o({ot(p)})o_{t}^{\mathrm{hyb}}=\operatorname{Fuse}_{o}(\{o_{t}^{(p)}\})2025Titans-MAG [ 13 ]
Table 15: Branch-wise Hybrid designs grouped by the interface at which heterogeneous memory paths are combined.
Token allocationUnified selection rulePeriodRepresentative methods
Temporal-boundary allocationzi=ftime(t−i,w)z_{i}=f_{\mathrm{time}}(t-i,w) : deterministic allocation by token age, position, or local-window boundary2024–2025LoLCATs [ 190 ] ; Native Hybrid Attention [ 40 ]
Content-score allocationzi∈{KVrecent,KVsalient,Scompressed}z_{i}\in\{\mathrm{KV}_{\mathrm{recent}},\mathrm{KV}_{\mathrm{salient}},S_{\mathrm{compressed}}\} : recent tokens remain local; departing tokens enter salient historical KV or a compressed state according to self-recall error or self-saliency2025–2026LoLA [ 94 ] ; STILL [ 98 ]
Learned operation allocationzi=arg⁡max⁡o∈Opθ(o∣xi,Hi)z_{i}=\arg\max_{o\in\mathcal{O}}p_{\theta}(o\mid x_{i},\mathcal{H}_{i}) : a learned or search-derived policy assigns a Softmax or Linear/Recurrent operation2026NAtS-L [ 35 ]
Table 16: Token-wise Hybrid designs grouped by the signal used to determine token allocation.
Figure 14: Attention designs across the classifiable architecture inventory. For natively multimodal systems, only the autoregressive language-model backbone is classified. Fill colors encode attention composition, split fills denote heterogeneous designs, and outlines indicate broad context-length tiers. The increasing concentration of split-fill nodes in recent years indicates the growing use of Hybrid architectures; the figure is descriptive and does not imply performance rankings or causal relationships.
PeriodRecordsSingle-familyHybridCross-layer
2022–202366 (100%)00
2024–20252516 (64%)9 (36%)0
2026 *2710 (37%)17 (63%)5 (19%)
Table 17: Temporal distribution of Single-family, Hybrid, and cross-layer-coordinated architectures in the publicly documented model inventory. Single-family and Hybrid are mutually exclusive labels; cross-layer coordination is an overlapping attribute.
Model endpointReleaseModel sizeAttention structure
MiMo-V2.6-Pro2026-091.02T (42B active)60 SWA-GQA + 10 global-GQA layers [ 91 ]
DeepSeek-V4.1-Flash (max)2026-09552B (8B active prefill; 16B active decode)CSA2 with SWA; cross-layer KV/index/top- kk reuse
K2-Horizon-375B-A23B2026-09375B (23B active)full GQA
GLM-5.3-Flash2026-08320B (18B active)34 KDA + 11 compressed-indexer DSA layers
GLM-5.3 (max)2026-08744B (40B active)MLA-based DSA; IndexShare/IndexCache
Qwen3.8-Flash-Next2026-08176B; 125B main (6B active)3 GDN : 1 QSA
Table 18: Attention structures of the eleven selected high-performing open-weight model endpoints in the Artificial Analysis snapshot captured on September 22, 2026. Rows are ordered by release date in reverse chronological order. Model size reports total parameters; disclosed active parameters are shown in parentheses [ 7 ] .
Figure 15: Mechanism-level synthesis of the expanding control scopes of explicit-memory and state-based methods. For the five functional dimensions, the upper-right mini-bars represent 2017–2018, 2019–2020, 2021–2022, 2023–2024, and 2025–2026 from left to right; taller and darker bars indicate greater qualitative activity among the representative methods discussed in Sections 3 – 6 . The 2026 coverage extends through September.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ReleaseModel lineageRelease recordAttention / mixer compositionHybridSource
2026-09MiMoMiMo-V2.6-Pro-RLSWA-GQA + global GQA (6:1)Yes[ 91 ]
2026-09DeepSeekDeepSeek-V4.1-FlashSWA + CSA2Yes[ 34 ]
2026-09MiniCPMMiniCPM5-2B familyFull GQANo[ 120 ]
2026-09K2K2-Horizon-375B-A23BFull GQANo[ 69 ]
2026-08QwenQwen3.8-2.4T-A95BGated DeltaNet + full GQA (3:1)YesModel card ; config
2026-08QwenQwen3.8-27BGated DeltaNet + gated full GQA (3:1)Yes[ 134 ]
Appendix
Table S1: The 59 publicly documented release-level architecture records analyzed in Section 8 , with the lineage and Hybrid labels used to derive Table 17 .
More in this topic

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 513 papers

Explore similar work

CardsList
Jul 28, 2026

Memory for Large Language Models

Memory has evolved into a foundational architectural dimension in large language models (LLMs), shifting from an implicit byproduct of computation to a spectrum of explicit, controllable mechanisms. While recent advances introduce diverse strategies---spanning transient attention, recurrent state dynamics, parameter-efficient adaptations, and scalable lookup storage---this rapid evolution has led to a highly fragmented research landscape. In this survey, we present a systematic, architecture-centric taxonomy of memory in LLMs. Our framework characterizes memory along three orthogonal axes: representation (implicit versus explicit), update dynamics (offline versus online), and persistence (short-term versus long-term). We further formalize the granular mechanisms dictating memory writing, routing, state transitions, and consolidation. This unified perspective elucidates the conceptual boundaries between computation-coupled and independently addressable memory, effectively bridging disparate architectural paradigms. Additionally, we critically analyze hybrid memory architectures, system-level efficiency trade-offs, and multi-dimensional evaluation methodologies. By consolidating these scattered advancements into a cohesive framework, this survey charts the trajectory of memory-centric LLM design and provides a principled foundation for future innovations in scalable and adaptive language modeling.

Sining Zhoubian, Dan Zhang, Evgeny Kharlamov +1

Tsinghua University · National University of Singapore · Bosch AI

Memory-Augmented Language ModelsLLM Memory
AskPDF
Oct 15, 2026

From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

Long-context capability has become essential for large language models (LLMs) to handle complex tasks, yet continuously expanding the context window does not remove the computational, storage, and information-selection pressures incurred at inference time. Existing work on prompt compression, KV-cache management, efficient serving, retrieval augmentation, and agent memory is usually discussed in isolation, leaving the flow of information between storage and computation insufficiently understood. This survey reframes long-context compression as a memory-management problem for allocating inference-time resources. We distinguish compute-active memory, which directly participates in model computation, from compute-inactive memory, which lies outside the computation path but can be stored, retrieved, updated, or forgotten, and we characterize offloaded KV caches, persistent prefix caches, and precomputed KV states as intermediate states on the activation path. On this basis, we formulate writing, offloading, recall, retrieval-and-injection, materialization, eviction, consolidation, and forgetting as state transitions within an information lifecycle, and introduce a budgeted cost model that places these operations in a single cost account. The resulting framework unifies short-term context compression, long-term memory management, and the transitions between them, and shows that the central challenge is not merely reducing information volume but deciding which information should participate in costly inference, when, and in what form. We further summarize evaluation dimensions, benchmark gaps, and open challenges, and argue that future research should move from static, pointwise compression toward dynamic, governable, lifecycle-aware memory management.

Chaogui Gou

USTB

LLM Context ManagementMemory-Efficient InferenceLLM CompressionPersistent Memory for Language ModelsKV-Cache Management
AskPDF
May 18, 2026

Context Memorization for Efficient Long Context Generation

Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effective, it incurs two structural limitations: i) the prefix's influence fades as generation proceeds, and ii) attention computation over the prefix scales linearly with its length. Existing approaches either keep the prefix in attention while compressing it, or internalize it into model parameters through gradient-based training. The former still attends to the prefix at inference, while the latter is training-intensive and ill-suited to prefix updates. To address these issues, we propose attention-state memory, a training-free approach that externalizes the prefix into a lightweight, lookup-based memory of precomputed attention states between prefix and query tokens. On ManyICLBench with LLaMA-3.1-8B, our method improves accuracy over in-context learning at 1K-8K memory budgets while reducing attention latency by 1.36x at 8K, and surpasses full-attention RAG performance on NBA benchmark using only 20% of its memory footprint.

Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3

Institute of Science Tokyo, Japan · Imperial College London, UK

Memory-Augmented Language ModelsLong-Context Language Model InferenceLLM Inference AccelerationIn-Context Learning
AskPDF