Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

Ying Xie

cs.AIMar 15, 2026

Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

Authors: Ying Xie

Organizations: Kennesaw State University

Abstract

Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.

Figures & tables

Panel of Figure 1 — Figure 1: Simplified overview of SleepGate . During the wake phase, the model processes input tokens while maintaining a tagged KV cache. A sleep trigger decides whether to continue normal inference or enter a sleep micro-cycle. In the soft biasing variant used in our experiments, the cache passes through decay and the forgetting gate, and the resulting retention scores are converted to additive attention biases (Eq. 10 ) for a second forward pass. In the hard eviction variant, entries additionally pass through consolidation before eviction.

Component	Parameters	Description
Base transformer	793,344	Embedding + 4 layers + LM head
Temporal tagger	16,576	Semantic signature projection
Forgetting gate	74,241	2-layer MLP (retention scorer)
Consolidation	33,152	Cross-attention compression
Total	917,313
Sleep overhead	15.6%

Table 1: Model parameter breakdown.

Figure 2: Retrieval accuracy vs. PI depth (number of prior updates). SleepGate (solid red) maintains near-perfect accuracy through $n=10$ before degrading at higher depths. All baselines cluster near zero across all depths.

Method	$n{=}1$	$n{=}2$	$n{=}5$	$n{=}10$	$n{=}15$	$n{=}20$	$n{=}30$
SleepGate	82.5 / 0.0	99.0 / 0.5	99.5 / 0.0	97.0 / 3.5	73.5 / 23.5	30.5 / 55.5	16.5 / 62.0
StreamingLLM	17.5 / 0.0	18.0 / 9.0	10.0 / 28.5	6.0 / 19.5	6.0 / 22.5	4.0 / 31.5	5.5 / 29.0
Sliding Window	12.5 / 0.0	12.5 / 7.0	8.0 / 23.5	8.5 / 23.0	6.5 / 23.0	4.0 / 22.5	2.0 / 30.0
Decay Only	9.5 / 0.0	12.5 / 7.0	7.0 / 22.0	7.5 / 23.0	8.5 / 25.0	3.5 / 23.5	4.5 / 22.5
Full KV Cache	8.0 / 0.0	9.0 / 7.5	3.5 / 21.0	2.0 / 17.0	3.0 / 20.0	1.5 / 18.5	2.5 / 19.5
H 2 O	6.5 / 0.0	7.5 / 5.5	1.0 / 7.5	4.0 / 11.0	3.0 / 11.5	0.0 / 10.0	0.5 / 9.0

Table 2: Retrieval accuracy and stale retrieval rate (%) across PI depths. Each cell reports accuracy / stale%.

N=200

episodes per depth per method. SleepGate uses post-sleep (soft-biased) evaluation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value	Description
Gate hidden dim	$d_{h}$	128	Hidden dimension of $G_{\theta}$
Semantic sig. dim	$d_{s}$	64	Dimension of semantic signatures
Local pool window	$w$	4	Window size for local pooling
Conflict threshold	$\delta$	0.85	Cosine similarity for conflict
Keep threshold	$\alpha_{k}$	0.7	Retention score for keeping
Evict threshold	$\alpha_{e}$	0.3	Retention score for eviction

Appendix

Table 3: Hyperparameters used for SleepGate in the proof-of-concept experiments.

Biological Mechanism	SleepGate Module	Functional Correspondence
Synaptic homeostasis (global downscaling during SWS)	Key decay	Log-scale reduction of key magnitudes preserves relative importance while reducing absolute interference
Hippocampal-neocortical replay (selective consolidation)	Consolidation module	Cross-attention compression transfers important information into compact representations
Dopaminergic active forgetting	Forgetting gate $G_{\theta}$	Learned, content-dependent eviction of stale entries
Sleep spindles / sharp-wave ripples (oscillatory coordination)	Adaptive sleep trigger	Entropy and conflict signals coordinate when consolidation occurs
REM sleep (pattern separation)	Semantic signatures	Explicit representation of “what slot” each entry occupies enables disambiguation

Appendix

Table 4: Mapping between biological sleep mechanisms and SleepGate components.

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 199 papers

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1Memory-Augmented Language Models Long-Context Language Model Inference
Carnegie Mellon University · University of Maryland
SCM: Sleep-Consolidated Memory with Algorithmic Forgetting for Large Language Models
Apr 22, 2026Saish Sachin ShindeMemory-Augmented Language Models Memorization in Language Models
Clyrai IP Studio
LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models
Feb 27, 2026Sourav Chattaraj, Kanak RajLLM Reliability Belief Updating in LLMs
Thomson Reuters, Bengaluru, India

May 25, 2026J/K move · Enter open · S save

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti

Carnegie Mellon University · University of Maryland

Memory-Augmented Language Models Long-Context Language Model Inference Persistent Memory for Language Models Recurrent Transformers LLM Inference Acceleration

Open paper AskPDF arXiv

Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During sleep, the model performs

N

offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration

N

for our models improves performance, with the largest gains on examples that require deeper reasoning.

Abstract

KV-Cache Management

Explore similar work

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

SCM: Sleep-Consolidated Memory with Algorithmic Forgetting for Large Language Models

LLMs Remember First, Forget Last: Dual-Process Interference in Large Language Models