Some Memories Must Fade: Persistence and Decay of User Memories in Model Parameters
Organizations: Samsung Research America
Abstract
Parametric memory allows Large Language Models to internalize knowledge directly into their weights, offering a potentially powerful alternative to retrieval-based memory (RAG). However, existing approaches to parametric memory are difficult to deploy as they largely mirror the Continual Learning (CL) objective of retaining information indefinitely. This is neither sustainable nor sufficient for a memory system, which must govern knowledge across its lifecycle by retaining stable facts, reinforcing recurring observations, updating changed beliefs, and allowing transient information to fade. We propose a system for parametric memory management consisting of a memory substrate (handling information storage and forgetting) and a learned policy (what to remember, how deeply, and for how long). Our substrate is a fixed-capacity LoRA adapter that consists of k-hot rank combinations which store facts without catastrophic interference and maintain an independently controlled temporal decay, enabling outdated information to fade while long-term facts persist. On this substrate, we train a lightweight controller that governs each incoming user event to ensure the memory system reflects the user's true, updated belief state. To evaluate how memory systems manage user memory, we introduce a controlled dataset of long-horizon user trajectories with varying persistence of facts based on changing belief states and user-behavioral shifts, and probe queries that test recall while information is valid and abstention after it expires. Our policy learns to effectively manage user memory over its entire lifecycle, balancing recall and abstention to achieve 86% of oracle performance, unlike CL and RAG baselines that primarily default to one of the two behaviors. By providing the first system to manage memory directly in model weights, this work establishes a foundation for long-horizon parametric memory.
Figures & tables
| Belief | Expiry | Reinforcement | User-dynamics | Lifecycle GT | |
|---|---|---|---|---|---|
| ForgetBench ( Gu et al., 2026 ) | ◐ | ✗ | ✗ | ✗ | ✗ |
| LoCoMo ( Maharana et al., 2024 ) | ◐ | ✗ | ✗ | ✗ | ✗ |
| PersonaMem ( Jiang et al., 2025 ) | ✓ | ✗ | ✗ | ◐ | ◐ |
| MemoryAgentBench ( Hu et al., 2026 ) | ✓ | ✗ | ✗ | ✗ | ✗ |
| LongMemEval ( Wu et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ◐ |
| Memora ( Uddin et al., 2026 ) | ✓ | ✗ | ◐ | ◐ | ✓ |
| Method | CB | VAL | ABS |
|---|---|---|---|
| Naive LoRA | 0.23 | 0.51 | 0.00 |
| EWC | 0.22 | 0.50 | 0.00 |
| PCGrad | 0.42 | 0.82 | 0.00 |
| Ours (PMM-PR) | 0.60 | 0.68 | 0.59 |
| System | CB | STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT |
|---|---|---|---|---|---|---|
| Metadata oracle | ||||||
| PCGrad (best CL, ) | ||||||
| RAG ( B) | ||||||
| RAG-MM ( B) | ||||||
| RAG-MM ( B) | ||||||
| RAG-MM ( B, trained) |
| Controller | Layout | CB | STABLE VAL | STABLE ABS | MB/user |
|---|---|---|---|---|---|
| Oracle | 32 shared | .65 | .53 | .74 | 11 |
| Oracle | 64 shared | .70 | .73 | .58 | 22 |
| Oracle | 32 + 32 dedicated | .71 | .82 | .68 | 22 |
| PMM-PR | 32 shared | .60 | .33 | .81 | 11 |
| PMM-PR | 32 + 32 dedicated | .63 | .70 | .67 | 22 |
| PMM-LR | 32 shared | .56 | .34 | .78 | 11 |
| Variant | CB acc | Notable effect |
|---|---|---|
| PMM-PR (full) | baseline | |
| No history | ONESHOT VAL | |
| No substrate | ROUTINE ABS | |
| No priors | overall pt | |
| No semantic | overall pt |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT | |||||||
| Config | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS | CB |
| Fixed , varying | |||||||||||
| 0.58 | 0.70 | 0.33 | 0.83 | 0.42 | 0.53 | 0.26 | 0.89 | 0.62 | 0.77 | 0.61 | |
| 0.64 | 0.70 | 0.76 | 0.74 | 0.85 | 0.27 | 0.46 | 0.80 | 0.97 | 0.46 | 0.69 | |
| 0.57 | 0.75 | 0.97 | 0.58 | 1.00 | 0.15 | 0.92 | 0.41 | 0.97 | 0.62 | 0.66 | |
| 0.49 | 0.75 | 0.95 | 0.40 | 1.00 | 0.09 | 0.97 | 0.21 | 0.92 | 0.48 | 0.57 | |
| Depth | 10 | 15 | 20 | 30 | 40 | 60 | 80 | 110 | 150 |
|---|---|---|---|---|---|---|---|---|---|
| Perfect router | 0.00 | 0.00 | 0.00 | 0.05 | 0.13 | 0.61 | 0.78 | 0.89 | 0.97 |
| Learnt router | 0.00 | 0.00 | 0.01 | 0.15 | 0.49 | 0.84 | 0.90 | 0.94 | 0.98 |
| (perfect) | 0.00 | 0.29 | 0.93 | 0.97 |
|---|---|---|---|---|
| (learnt) | 0.01 | 0.45 | 0.95 | 0.98 |
| Challenger depth | 20 | 40 | 80 | 110 |
|---|---|---|---|---|
| perfect router | 0.55 | 1.00 | 1.00 | 1.00 |
| learnt router | 0.92 | 1.00 | 1.00 | 1.00 |
| PMM | RAG-MM | |
|---|---|---|
| Read (per query) | 94 ms | 360 ms |
| Write (per fact) | 1.8 s ( ) – 13.1 s ( ) | 2.4 s |
| Full user trajectory | 702 s | 617 s |
| Depth | 10 | 20 | 30 | 40 | 60 | 80 | 110+ |
|---|---|---|---|---|---|---|---|
| 0.55 | 0.38 | 0.30 | 0.22 | 0.12 | 0.05 | 0.00 | |
| Implied | 0.10 | 0.20 | 0.32 | 0.45 | 0.70 | 0.85 | 0.95+ |
| STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT | ||||||
| System | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS |
| Metadata oracle | 0.53 | 0.74 | 0.75 | 0.65 | 0.84 | 0.24 | 0.52 | 0.81 | 0.93 | 0.46 |
| Naive | 0.17 | 0.00 | 0.72 | 0.00 | 0.63 | 0.00 | 0.76 | 0.00 | 0.26 | 0.00 |
| EWC | 0.19 | 0.00 | 0.73 | 0.00 | 0.64 | 0.00 | 0.76 | 0.01 | 0.19 | 0.00 |
| PCGrad | 0.70 | 0.00 | 0.96 | 0.00 | 0.89 | 0.00 | 0.91 | 0.00 | 0.64 | 0.00 |
| Unmanaged | 0.64 | 0.09 | 0.90 | 0.01 | 0.87 | 0.06 | 0.77 | 0.22 | 0.55 | 0.27 |
| System | CB | STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT |
|---|---|---|---|---|---|---|
| System | CB | STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT |
|---|---|---|---|---|---|---|
| Naive | ||||||
| EWC | ||||||
| Oracle RAG (1.7B) | ||||||
| Oracle RAG (122B) |
| Head | Initialized bias values |
|---|---|
| Store | (empirical write rate ) |
| Depth | over |
| over |
| Pattern | |||||||
|---|---|---|---|---|---|---|---|
| STABLE | |||||||
| ROUTINE | |||||||
| CONTESTED | |||||||
| TRANSIENT | |||||||
| ONESHOT |
| Hyperparameter | PMM-PR | PMM-LR |
|---|---|---|
| base weight | ||
| boost duration (steps) | ||
| class weighting | inverse | inverse |
| Success ( ) | Failure ( ) | |
|---|---|---|
| Prior writes (mean) | ||
| Last write depth (mean) | ||
| Accumulated depth (mean) | ||
| Days since last write (mean) | ||
| Self steps at query | ||
| Foreign steps at query |
| STABLE | ROUTINE | CONTESTED | TRANSIENT | ONESHOT | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| System | CB | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS | VAL | ABS |
| PMM-PR | 0.60 | 0.33 | 0.81 | 0.80 | 0.64 | 0.81 | 0.34 | 0.66 | 0.58 | 0.82 | 0.58 |
| No History | 0.58 | 0.33 | 0.91 | 0.67 | 0.68 | 0.67 | 0.32 | 0.74 | 0.68 | 0.58 | 0.76 |
| No Substrate | 0.58 | 0.34 | 0.95 | 0.77 | 0.55 | 0.76 | 0.28 | 0.70 | 0.55 | 0.65 | 0.72 |
| No Priors | 0.52 | 0.42 | 0.80 | 0.66 | 0.57 | 0.73 | 0.20 | 0.78 | 0.41 | 0.69 | 0.44 |
| No Semantic | 0.47 | 0.28 | 0.88 | 0.70 | 0.33 | 0.63 | 0.28 | 0.61 | 0.35 | 0.49 | 0.60 |
| Metric | Value |
|---|---|
| Total Users | |
| Query Horizon | Avg days (Min: , Max: ) |
| Total Writes | |
| Total Queries | |
| Query Target Split | VALUE ABSTAIN |
| Temporal Motif | Users |
|---|---|
| NORMAL DRIFT: Slow, mundane, almost imperceptible changes | |
| THE MICRO-VARIATION: Highly stable daily routine | |
| THE SEASONAL SHIFT: 2-to-3 month block behavior | |
| THE PHASED TRANSITION: Gradual shift over 2 months | |
| THE VACATION/TRIP: Stable routine changes for 2 weeks | |
| THE INSTANT CORRECTION: Realizes mistake minutes/hours later |
| Day | Event | Pattern | Ledger Entry / Query |
|---|---|---|---|
| Day 0 | INTRODUCE | STABLE | I work as a [JOB_8] |
| Day 2 | INTRODUCE | ROUTINE | I ordered my morning coffee from [LOC_CAFE_4] |
| Day 5 | INTRODUCE | TRANSIENT | I grabbed a quick lunch with Sarah at [LOC_BISTRO_9] |
| Day 12 | ABS QUERY | TRANSIENT | I grabbed a quick lunch with Sarah at [MASK] |
| Day 14 | INTRODUCE | CONTESTED | My current priority project is [PROJ_2] |
| Day 15 | UPDATE | ROUTINE | I ordered my morning coffee from [LOC_CAFE_4] |
| decay factor at query time | ||||
| depth | ||||
| 0.23 | 0.12 | 0.00 | 0.00 | |
| 0.83 | 0.56 | 0.06 | 0.00 | |
| 0.96 | 0.88 | 0.08 | 0.00 | |
| 1.00 | 0.94 | 0.17 | 0.00 | |
| 1.00 | 0.96 | 0.19 | 0.00 | |
Memory-Augmented Language Models
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
See all 314 papers