Memory Canonicalization: A Framework and Benchmark for Cross-Model Drift in Persistent LLM Memory

Amit Vadnere; Aishwarya Lonarkar

cs.AIOct 4, 2026

Memory Canonicalization: A Framework and Benchmark for Cross-Model Drift in Persistent LLM Memory

Authors: Amit Vadnere, Aishwarya Lonarkar

Organizations: Product Solutions Engineer

Abstract

Persistent memory for Large Language Models (LLMs) has matured rapidly: systems such as MemGPT/Letta, Mem0, and Zep now provide agents with tiered, temporally-aware, model-agnostic external storage, while the Model Context Protocol (MCP) standardizes access to memory servers. A less addressed problem is that an identical stored memory object, retrieved by two different LLMs under otherwise identical conditions, may not be interpreted the same way, factually or emotionally. This paper proposes memory canonicalization: a write-time pipeline that detects ambiguity, conditional structure, and emotional loading in a raw memory object and rewrites it into an explicit, structurally disambiguated canonical form, with emotional valence represented as a separate field rather than inferred from tone. We formalize the pipeline, define a companion Cross-Model Semantic Drift / Emotional Consistency Score benchmark (CMSC-E), and report results from a three-arm pilot using 176 synthetic memory objects and three downstream model families. We find an uncorrected improvement in cross-model emotional consistency for fully canonicalized memory relative to raw memory (+0.050, 95% bootstrap CI [0.013, 0.086], paired t-test p = 0.010), but this result does not survive Bonferroni, Holm, or Benjamini-Hochberg correction across the six comparisons tested. None of the factual-drift (CMSD) comparisons reach significance at any correction level. We report these results as exploratory rather than confirmatory and outline needed follow-up work, including larger samples, independent judge models, human-validated rendering, and preregistration.

Figures & tables

System	Persistence	Temporal	Emotion	Cross-Model	Canonicalization
MemGPT / Letta [4], [5]	Yes	No	No	No	No
Mem0 [6]	Yes	Partial	No	No	No
Zep / Graphiti [7]	Yes	Yes	No	No	No
MCP [13]	N/A	N/A	No	No	No
Native platform memory	Yes	No	No	No	No
MemCollab [17]	No	No	No	Partial	No

TABLE I: Comparison of Persistent Memory and Cross-Model Drift Systems

Panel of Figure 2 — Fig. 1: Canonicalization pipeline architecture. Steps 1–5b (Phase 1) are stateless and run per object; Step 6 (Phase 2) is store-aware and runs after Phase 1.

Metric	Comparison (A $\rightarrow$ B)	Mean A	Mean B	Diff	95% CI	$p_{t}$	$p_{w}$
CMSD	Arm 1 $\rightarrow$ 2 (disambiguation)	0.7301	0.7388	+0.0088	[ $-0.014$ , 0.033]	0.463	0.729
CMSD	Arm 2 $\rightarrow$ 3 (emotional extraction)	0.7388	0.7349	$-0.0039$	[ $-0.022$ , 0.013]	0.664	0.573
CMSD	Arm 1 $\rightarrow$ 3 (overall)	0.7301	0.7349	+0.0048	[ $-0.022$ , 0.032]	0.721	0.931
ECS	Arm 1 $\rightarrow$ 2 (disambiguation)	0.6818	0.7002	+0.0184	[ $-0.015$ , 0.052]	0.294	0.292
ECS	Arm 2 $\rightarrow$ 3 (emotional extraction)	0.7002	0.7316	+0.0314	[ $-0.006$ , 0.068]	0.103	0.079
ECS	Arm 1 $\rightarrow$ 3 (overall)	0.6818	0.7316	+0.0498	[0.013, 0.086]	0.010	0.006

TABLE II: Paired Arm Comparisons, CMSD (

n=172

) and ECS (

n=86

)

Explore similar work

Aug 8, 2026cs.AI

Mitigating Over-Personalization in LLMs via Structured Memory

Conversational assistants increasingly rely on persistent long-term memory to personalize responses across sessions. However, when stored user information is reintroduced into the model context, it can also influence responses in inappropriate or unrelated settings. We study two such failure modes in memory-augmented LLMs: cross-domain leakage, where memories from one life domain affect responses in another, and memory-induced sycophancy, where stored user beliefs make models more likely to agree with the user rather than respond truthfully. We apply a simple inference-time modification to how memories are presented to the model, without changing the model or the memory contents. Across seven models on PersistBench, we compare the commonly used all-in context format, where memories are injected as an unstructured list, with structured formats that partition memories by domain. This simple modification consistently reduces cross-domain leakage while preserving utility, with our strongest method reducing leakage by

8.8\%

on average relative to the baseline.

Hakeem Hannoon, Andrew Zhao, Mihir Narayan +2

University of Saskatchewan · University of California–Irvine · University of Wisconsin–Madison +2

Large Language Model Memory Long-Term Memory Peak Memory Sycophancy

Ask PDF

May 26, 2026cs.AI

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.

Ishir Garg, Neel Kolhe, Dawn Song +1

University of California, Berkeley

Large Language Model Memory Large Language Models Fail Large Language Model Benchmarks Latent Failure Patterns Diagnostic Benchmark

Ask PDF

Apr 22, 2026cs.LG

SCM: Sleep-Consolidated Memory with Algorithmic Forgetting for Large Language Models

We present SCM (Sleep-Consolidated Memory), a research preview of a memory architecture for large language models that draws on neuroscientific principles to address a fundamental limitation in current systems: the absence of persistent, structured, and biologically plausible memory. Existing approaches rely on truncating context windows, growing vector databases without bound, or tiered storage systems that lack consolidation and forgetting mechanisms. SCM implements five core components inspired by human memory: a limited-capacity working memory, multi-dimensional importance tagging, offline sleep-stage consolidation with distinct NREM and REM phases, intentional value-based forgetting, and a computational self-model enabling introspection. Across a standardized benchmark suite of eight tests, the prototype achieves perfect recall accuracy over ten-turn conversations while reducing memory noise by 90.9% through adaptive forgetting. Memory search latency remains below one millisecond even with hundreds of stored concepts. This work establishes the architectural foundations for memory systems that consolidate, prioritize, and forget, offering a testable platform for advancing LLM memory research.

Saish Sachin Shinde

Clyrai IP Studio

Large Language Model Memory Catastrophic Forgetting Sleep

Ask PDF