Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.
Figures & tables
Figure 1: Simplified overview of SleepGate . During the wake phase, the model processes input tokens while maintaining a tagged KV cache. A sleep trigger decides whether to continue normal inference or enter a sleep micro-cycle. In the soft biasing variant used in our experiments, the cache passes through decay and the forgetting gate, and the resulting retention scores are converted to additive attention biases (Eq. 10 ) for a second forward pass. In the hard eviction variant, entries additionally pass through consolidation before eviction.
Component
Parameters
Description
Base transformer
793,344
Embedding + 4 layers + LM head
Temporal tagger
16,576
Semantic signature projection
Forgetting gate
74,241
2-layer MLP (retention scorer)
Consolidation
33,152
Cross-attention compression
Total
917,313
Sleep overhead
15.6%
Table 1: Model parameter breakdown.
Figure 2: Retrieval accuracy vs. PI depth (number of prior updates). SleepGate (solid red) maintains near-perfect accuracy through n=10 before degrading at higher depths. All baselines cluster near zero across all depths.
Method
n=1
n=2
n=5
n=10
n=15
n=20
n=30
SleepGate
82.5 / 0.0
99.0 / 0.5
99.5 / 0.0
97.0 / 3.5
73.5 / 23.5
30.5 / 55.5
16.5 / 62.0
StreamingLLM
17.5 / 0.0
18.0 / 9.0
10.0 / 28.5
6.0 / 19.5
6.0 / 22.5
4.0 / 31.5
5.5 / 29.0
Sliding Window
12.5 / 0.0
12.5 / 7.0
8.0 / 23.5
8.5 / 23.0
6.5 / 23.0
4.0 / 22.5
2.0 / 30.0
Decay Only
9.5 / 0.0
12.5 / 7.0
7.0 / 22.0
7.5 / 23.0
8.5 / 25.0
3.5 / 23.5
4.5 / 22.5
Full KV Cache
8.0 / 0.0
9.0 / 7.5
3.5 / 21.0
2.0 / 17.0
3.0 / 20.0
1.5 / 18.5
2.5 / 19.5
H 2 O
6.5 / 0.0
7.5 / 5.5
1.0 / 7.5
4.0 / 11.0
3.0 / 11.5
0.0 / 10.0
0.5 / 9.0
Table 2: Retrieval accuracy and stale retrieval rate (%) across PI depths. Each cell reports accuracy / stale%. N=200 episodes per depth per method. SleepGate uses post-sleep (soft-biased) evaluation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Symbol
Value
Description
Gate hidden dim
dh
128
Hidden dimension of Gθ
Semantic sig. dim
ds
64
Dimension of semantic signatures
Local pool window
w
4
Window size for local pooling
Conflict threshold
δ
0.85
Cosine similarity for conflict
Keep threshold
αk
0.7
Retention score for keeping
Evict threshold
αe
0.3
Retention score for eviction
Appendix
Table 3: Hyperparameters used for SleepGate in the proof-of-concept experiments.
Biological Mechanism
SleepGate Module
Functional Correspondence
Synaptic homeostasis (global downscaling during SWS)
Key decay
Log-scale reduction of key magnitudes preserves relative importance while reducing absolute interference
Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During sleep, the model performs N offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration N for our models improves performance, with the largest gains on examples that require deeper reasoning.
Sangyun Lee, Sean McLeish, Tom Goldstein +1
Carnegie Mellon University · University of Maryland
We present SCM (Sleep-Consolidated Memory), a research preview of a memory architecture for large language models that draws on neuroscientific principles to address a fundamental limitation in current systems: the absence of persistent, structured, and biologically plausible memory. Existing approaches rely on truncating context windows, growing vector databases without bound, or tiered storage systems that lack consolidation and forgetting mechanisms. SCM implements five core components inspired by human memory: a limited-capacity working memory, multi-dimensional importance tagging, offline sleep-stage consolidation with distinct NREM and REM phases, intentional value-based forgetting, and a computational self-model enabling introspection. Across a standardized benchmark suite of eight tests, the prototype achieves perfect recall accuracy over ten-turn conversations while reducing memory noise by 90.9% through adaptive forgetting. Memory search latency remains below one millisecond even with hundreds of stored concepts. This work establishes the architectural foundations for memory systems that consolidate, prioritize, and forget, offering a testable platform for advancing LLM memory research.
Large language models can process millions of tokens, yet how they handle conflicting information within context remains poorly understood. From patient health logs tracking evolving vital signs to legal documents with superseding clauses, real-world applications routinely require models to retrieve specific values from streams of semantically similar, competing updates. We adapt classical interference paradigms from cognitive psychology to compare retroactive interference (RI; recalling initial values after updates) and proactive interference (PI; recalling recent values despite competing prior encodings) across 39 LLMs spanning 1B to 2.5T parameters. Every model exhibits the same pattern: PI causes substantially greater degradation than RI (d = 1.73), the opposite of the typical human finding where new information more readily disrupts old. Four lines of evidence indicate that RI and PI engage distinct mechanisms: model size predicts RI resistance but not PI; the two scores correlate only weakly; error analysis reveals qualitatively different failure profiles, with RI failures reflecting passive retrieval failure and PI failures reflecting active primacy intrusion; and hallucination rates remain below 1% in both conditions. An adaptive bootstrap design further shows that RI decays smoothly while PI collapses bimodally at high interference, consistent with gradual dilution versus winner-take-all attention dynamics. These findings characterize a systematic primacy bias in transformer attention, with implications for reliable deployment in settings where in-context information evolves over time.