EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

Zeyu Liu; Jian Zhong; Rongduo Han; Ziyang Wu; Shunye Tang; Chenghao He; Yaxuan Yang; Yihang Qiu

cs.CLSep 3, 2026

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

Authors: Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu, Shunye Tang, Chenghao He, Yaxuan Yang, Yihang Qiu,

Organizations: Nankai University · vivo AI Lab · Tsinghua University · Zhejiang University

Abstract

Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.

Figures & tables

Figure 1: EvalMem at a glance. A query is examined by three parallel Examiners over Encoding, Retrieval, and Generation; their decisions are reconciled into a multi-label defect set instead of a single end-to-end score.

Figure 2: Overall view of EvalMem: the tested system runs natively, then Encoding, Retrieval, and Generation Examiners map task-aware states into a multi-label defect set.

Code	Full name	Layer	Condition
EM	Encoding Missing	Enc.	POS store Miss
EA	Encoding Ambiguous	Enc.	ambiguous value
EW	Encoding Wrong	Enc.	wrong value
DMP	Dirty Memory Pollution	Enc.	NEG dirty memory
RF	Retrieval Failure	Ret.	POS Miss with $S_{\text{enc}}\!=\!\textsc{Exist}$
LATE	Retrieval Late	Ret.	hit rank $>\tau_{\text{rank}}$

Table 1: The

11

defect codes used by EvalMem, grouped by layer. The full attribution for query

i

is the multi-label union

\mathcal{D}_{\text{total}}^{(i)}

System	Encoding $\downarrow$					Retrieval $\downarrow$					Generation $\downarrow$				Acc $\uparrow$
System	EM	EA	EW	DMP	Enc	RF	LATE	NOI	NIR	Ret	GF	GRF	GH	Gen	Acc $\uparrow$
Set 1: LoCoMo + gpt-4o-mini $N\!=\!1986$ mean Acc $=68.1$ , mean Ret $=22.1$
TiMem	3.3	1.8	1.2	0.8	7.1	9.1	2.7	3.5	1.7	17.0	2.1	2.6	1.3	6.0	75.3
O-Mem	2.1	1.3	1.0	0.5	4.9	10.5	3.3	3.7	2.1	19.6	2.4	2.5	1.2	6.1	74.9
EverMemOS	4.6	2.0	1.3	0.7	8.6	10.4	3.4	4.0	2.2	20.0	2.5	2.4	1.4	6.3	72.4
MemOS	4.2	1.6	1.4	0.9	8.1	11.0	3.6	4.4	2.0	21.0	2.6	2.7	1.2	6.5	70.1

Table 2: Accuracy and per-layer defect rates for seven memory systems on three configurations: LoCoMo + gpt-4o-mini (Set 1), LoCoMo + gpt-4.1 (Set 2, cross-backbone), and DynaMem-Bench + gpt-4o-mini (Set 3). Columns: encoding codes + Enc total, retrieval codes + Ret total, generation codes + Gen total, end-to-end Acc ; defect codes are assigned only to incorrectly answered outputs, while each defect column reports a sample-level union rate with the full-query denominator. Defect codes may overlap. Bold red = best per column within each Set; blue underline = second-best (ties marked at every tied cell). Qwen3 , LongMemEval -S, and the full

7\!\times\!3\!\times\!3

matrix are in Appendix C.1 .

Figure 3: Mechanism analyses on LoCoMo with gpt-4o-mini . (a) POS Hit@

K

saturates around

K\!=\!10

. (b) Final accuracy peaks at

K\!\in\!\{8,10\}

. (c) Agentic-RAG reliability analysis: the evidence-aware query is the largest contributor to reducing diagnostic MISS, while the agentic-RAG pipeline mechanisms provide additional gains. (d) MemWiki per-class and combined intervention: factual records are the strongest single auxiliary class.

System	Bsl Acc	+MW Acc	$\Delta$ Acc	$\Delta$ Ret
EverMemOS	82.6	83.4	$+0.8$	$-1.4$
O-Mem	81.4	82.8	$+1.4$	$-2.6$
MemBox	77.0	79.6	$+2.6$	$-3.4$
GAM	76.6	79.4	$+2.8$	$-3.6$
TiMem	76.0	76.6	$+0.6$	$-1.4$
MemOS	65.0	68.4	$+3.4$	$-4.4$

Table 3: MemWiki on LongMemEval -S (gpt-4o-mini). Mean Acc / Ret shifts mirror the LoCoMo result within

0.2

pp; per-system significance and Pearson

r(\Delta\text{Ret},\Delta\text{Acc})

in Appendix C.7 .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

System	gpt-4o-mini (default)				gpt-4.1				Qwen3-30B-Instruct
System	Acc $\uparrow$	Enc $\downarrow$	Ret $\downarrow$	Gen $\downarrow$	Acc $\uparrow$	Enc $\downarrow$	Ret $\downarrow$	Gen $\downarrow$	Acc $\uparrow$	Enc $\downarrow$	Ret $\downarrow$	Gen $\downarrow$
Set 1: LoCoMo $N\!=\!1986$
TiMem	75.3	7.1	17.0	6.0	77.8	5.5	15.4	4.1	74.2	7.8	17.6	6.5
O-Mem	74.9	4.9	19.6	6.1	78.1	3.7	17.5	4.2	73.8	5.5	20.2	6.7
EverMemOS	72.4	8.6	20.0	6.3	75.2	6.6	17.8	4.4	71.0	9.3	20.7	6.9
MemOS	70.1	8.1	21.0	6.5	73.1	6.1	18.7	4.7	69.2	8.7	21.8	7.1

Appendix

Table 6: Full

7\!\times\!3\!\times\!3

matrix of end-to-end accuracy (Acc) and the three layer-aggregate defect rates (Enc, Ret, Gen) across seven memory systems, three datasets (Sets 1–3), and three answer-generation backbones. Defect codes are assigned only to incorrectly answered outputs; all defect rates are sample-level union incidences with the full-query denominator; defect codes may overlap. Bold red marks the best per column within each Set (highest Acc, lowest defect); blue underline marks the second-best; ties receive the same marker at every tied position. For LongMemEval -S (Set 2) we report only Acc and the three layer totals: with

N_{\text{NEG}}\!=\!30

, the smallest 11-code cell would be unstable beyond statistical reliability (Wilson 95% CI

\geq\!25

pp); the layer-total view remains diagnostic and is the primary use of this dataset as a generalisation check on LoCoMo . Systems within each Set are ordered by Acc under the default backbone ( gpt-4o-mini ). The companion

11

-code resolution for LoCoMo and DynaMem-Bench appears in Figs. 4 – 5 .

Figure 4: Heatmap view of the full $11$ -code sub-code breakdown on LoCoMo across three answer-generation backbones (Set 1.1–1.3). Rows are memory systems, columns are defect codes, and each cell shows the sample-level union defect rate (%) over the full $N\!=\!1986$ POS+NEG query pool; codes are assigned only to incorrectly answered outputs. Color indicates defect magnitude (lighter is lower, darker is higher), while text annotations preserve exact values. Green boxes mark per-column best (lowest defect) within each Set; blue boxes mark second-best (ties are all marked).

Figure 5: Heatmap view of the full $11$ -code sub-code breakdown on DynaMem-Bench across three answer-generation backbones (Set 2.1–2.3). Values are sample-level union defect rates (%) over the full $N\!=\!962$ POS+NEG query pool; defect codes are assigned only to incorrectly answered outputs. The NIR column remains visibly darker than in LoCoMo , reflecting the larger share of retrieval defects caused by stale-evidence inference risk. Green and blue boxes denote per-column best and second-best systems within each Set.

System	Released/native embedding	Acc S/N	$\Delta$ Acc	Enc/Ret/Gen (S)	Enc/Ret/Gen (N)
TiMem	Qwen3-Embedding-0.6B (same as standardized)	$75.3/75.3$	$0.0$	$7.1/17.0/6.0$	$7.1/17.0/6.0$
O-Mem	all-MiniLM-L6-v2	$74.9/69.8$	$-5.1$	$4.9/19.6/6.1$	$6.2/23.4/6.7$
EverMemOS	Qwen3-Embedding-4B	$72.4/76.0$	$+3.6$	$8.6/20.0/6.3$	$7.5/17.5/5.9$
MemOS	nomic-embed-text:latest	$70.1/72.9$	$+2.8$	$8.1/21.0/6.5$	$7.2/18.8/6.1$
MemBox	text-embedding-3-small	$63.2/56.4$	$-6.8$	$5.9/25.7/6.9$	$7.5/30.8/7.6$
MemoryOS	all-MiniLM-L6-v2	$60.8/63.2$	$+2.4$	$9.3/26.0/6.6$	$8.5/23.7/6.3$

Appendix

Table 7: Native-embedding robustness on LoCoMo with gpt-4o-mini . S/N denotes the standardized/native embedding runs;

\Delta

is native minus standardized accuracy in percentage points. Only the embedding encoder used by the system’s embedding-based retrieval/index and the corresponding Encoding-Examiner diagnostic retrieval is changed; its vectors/index entries are rebuilt as needed, while the answer backbone, Top-

K

, EvalMem judges, memory construction, index type/structure, retrieval logic, and all other settings remain fixed. TiMem uses the standardized encoder natively, and Retrieval remains the most frequently attributed layer in every row. Layer rates are unions assigned only to incorrectly answered outputs, with the full-query denominator. This is an embedding-only ablation, not a complete per-system native-configuration study.

Threshold	Candidates	Controlled sub-code rate (%)	Calibration macro-F1	Selected
$\tau_{\mathrm{rank}}$	$3/5/7/9$	$6.3/3.8/2.6/1.7$	$0.78/0.84/0.76/0.58$	$5$
$\tau_{\mathrm{snr}}$	$0.10/0.15/0.20/0.25/0.30$	$1.8/2.9/4.4/6.0/8.1$	$0.61/0.75/0.82/0.79/0.74$	$0.20$
$\tau_{H}$	$0.45/0.50/0.55/0.60/0.65$	$3.2/2.5/2.0/1.6/1.3$	$0.79/0.82/0.84/0.78/0.69$	$0.55$

Appendix

Table 8: Threshold sensitivity and calibration under the Set 1 protocol. One threshold is varied at a time; the controlled sub-code rate is the unweighted mean over seven systems. LATE/NOI calibration uses 200 erroneous POS–HIT pairs and NIR calibration uses 200 NEG pairs. Two annotators label independently and a third adjudicates disagreements; the selected value maximizes calibration macro-F1.

System	Enc mean $\pm$ SD	Ret mean $\pm$ SD	Gen mean $\pm$ SD	Fleiss’ $\kappa$
TiMem	$7.16\!\pm\!0.17$	$16.84\!\pm\!0.22$	$6.09\!\pm\!0.11$	$0.93$
O-Mem	$4.97\!\pm\!0.14$	$19.72\!\pm\!0.20$	$6.05\!\pm\!0.10$	$0.91$
EverMemOS	$8.53\!\pm\!0.19$	$20.17\!\pm\!0.23$	$6.39\!\pm\!0.11$	$0.93$
MemOS	$8.18\!\pm\!0.18$	$20.91\!\pm\!0.21$	$6.57\!\pm\!0.13$	$0.94$
MemBox	$6.03\!\pm\!0.16$	$25.86\!\pm\!0.25$	$6.83\!\pm\!0.12$	$0.92$
MemoryOS	$9.37\!\pm\!0.21$	$26.12\!\pm\!0.24$	$6.77\!\pm\!0.14$	$0.89$

Appendix

Table 9: Five-run diagnostic stability under the Set 1 configuration ( LoCoMo with gpt-4o-mini ). Baseline systems’ outputs are held fixed while the full EvalMem diagnostic pipeline is rerun five times. Layer values are mean

\pm

SD in percentage points, using the error gate and full-query denominator in § 2.2 ; Fleiss’

\kappa

measures agreement across the five run-level diagnoses. This is evaluator-run stochasticity, not memory-system training or evaluation-seed variance.

System	EM	EA	EW	DMP	Enc total
TiMem	$3.45\!\pm\!0.21$	$1.67\!\pm\!0.22$	$1.31\!\pm\!0.20$	$0.73\!\pm\!0.15$	$7.16\!\pm\!0.17$
O-Mem	$2.11\!\pm\!0.05$	$1.38\!\pm\!0.19$	$1.01\!\pm\!0.04$	$0.47\!\pm\!0.05$	$4.97\!\pm\!0.14$
EverMemOS	$4.51\!\pm\!0.16$	$1.96\!\pm\!0.09$	$1.34\!\pm\!0.07$	$0.72\!\pm\!0.04$	$8.53\!\pm\!0.19$
MemOS	$4.14\!\pm\!0.14$	$1.69\!\pm\!0.23$	$1.54\!\pm\!0.18$	$0.81\!\pm\!0.16$	$8.18\!\pm\!0.18$
MemBox	$2.31\!\pm\!0.10$	$1.43\!\pm\!0.08$	$1.21\!\pm\!0.20$	$1.09\!\pm\!0.19$	$6.03\!\pm\!0.16$
MemoryOS	$4.60\!\pm\!0.15$	$2.33\!\pm\!0.10$	$1.65\!\pm\!0.16$	$0.79\!\pm\!0.05$	$9.37\!\pm\!0.21$

Appendix

Table 10: Five-run diagnostic stability for the Encoding layer under Set 1. Each cell is a mean

\!\pm\!

SD in percentage points over the same five evaluator reruns; the four code columns are the resolution of the Enc total. The error gate and full-query denominator are those in § 2.2 .

System	RF	LATE	NOI	NIR	Ret total
TiMem	$9.30\!\pm\!0.22$	$2.71\!\pm\!0.08$	$3.30\!\pm\!0.29$	$1.53\!\pm\!0.18$	$16.84\!\pm\!0.22$
O-Mem	$10.56\!\pm\!0.18$	$3.11\!\pm\!0.27$	$3.90\!\pm\!0.27$	$2.15\!\pm\!0.15$	$19.72\!\pm\!0.20$
EverMemOS	$10.43\!\pm\!0.13$	$3.21\!\pm\!0.31$	$4.13\!\pm\!0.31$	$2.40\!\pm\!0.27$	$20.17\!\pm\!0.23$
MemOS	$11.20\!\pm\!0.23$	$3.58\!\pm\!0.07$	$4.21\!\pm\!0.28$	$1.92\!\pm\!0.24$	$20.91\!\pm\!0.21$
MemBox	$14.89\!\pm\!0.32$	$4.49\!\pm\!0.09$	$4.33\!\pm\!0.09$	$2.15\!\pm\!0.10$	$25.86\!\pm\!0.25$
MemoryOS	$13.87\!\pm\!0.27$	$4.70\!\pm\!0.32$	$4.98\!\pm\!0.08$	$2.57\!\pm\!0.29$	$26.12\!\pm\!0.24$

Appendix

Table 11: Five-run diagnostic stability for the Retrieval layer under Set 1. Each cell is a mean

\!\pm\!

SD in percentage points over the same five evaluator reruns; the four code columns are the resolution of the Ret total. The error gate and full-query denominator are those in § 2.2 .

System	GF	GRF	GH	Gen total
TiMem	$2.13\!\pm\!0.08$	$2.70\!\pm\!0.14$	$1.26\!\pm\!0.12$	$6.09\!\pm\!0.11$
O-Mem	$2.31\!\pm\!0.10$	$2.59\!\pm\!0.10$	$1.16\!\pm\!0.11$	$6.05\!\pm\!0.10$
EverMemOS	$2.56\!\pm\!0.11$	$2.43\!\pm\!0.09$	$1.41\!\pm\!0.05$	$6.39\!\pm\!0.11$
MemOS	$2.51\!\pm\!0.11$	$2.74\!\pm\!0.12$	$1.32\!\pm\!0.17$	$6.57\!\pm\!0.13$
MemBox	$2.70\!\pm\!0.06$	$2.65\!\pm\!0.12$	$1.48\!\pm\!0.16$	$6.83\!\pm\!0.12$
MemoryOS	$2.22\!\pm\!0.19$	$2.90\!\pm\!0.13$	$1.65\!\pm\!0.19$	$6.77\!\pm\!0.14$

Appendix

Table 12: Five-run diagnostic stability for the Generation layer under Set 1. Each cell is a mean

\!\pm\!

SD in percentage points over the same five evaluator reruns; the three code columns are the resolution of the Gen total. The error gate and full-query denominator are those in § 2.2 .

System	Calls	Input (K)	Output (K)	Critical latency (s)
				p50	p95
TiMem	8.24	38.0	1.81	11.4	25.4
O-Mem	8.38	40.5	1.88	11.9	26.4
EverMemOS	8.42	44.0	1.97	12.4	27.7
MemOS	8.36	43.0	1.95	12.2	27.3
MemBox	8.64	53.0	2.18	13.8	30.8

Appendix

Table 13: Per-example EvalMem diagnostic overhead by memory system under the Set 1 protocol. Input/output values are in

K=10^{3}

tokens; calls and token counts are means, and critical latency is the per-example p50/p95 on the parallel-Examiner path followed by Attribution. The mean row is the unweighted mean over seven systems.

Component	Calls	Input (K)	Output (K)	Latency (s)
				p50	p95
Agentic-RAG (AR)	3.48	21.8	0.94	8.2	18.7
Encoding judge (EJ)	1.00	11.6	0.23	3.2	6.8
Retrieval	1.00	8.4	0.31	3.1	6.9
Generation	2.00	3.9	0.39	4.0	8.5
Attribution	1.00	1.44	0.16	1.5	3.4

Appendix

Table 14: Mean component-level EvalMem diagnostic overhead over seven systems under Set 1. Input/output values are in

K=10^{3}

tokens. Rounded component means sum to the per-system mean up to rounding. Agentic-RAG (AR) and the Encoding judge (EJ) form the Encoding path; the three Examiners are parallel, so component p50/p95 values are not additive for the critical path.

Figure 6: Adapted AgenticRAG diagnostic pipeline. For each $(Q,F_{\text{key}})$ , the agent plans a set of views (dense, BM25, time, entity, topic, plus an on-demand gap view), routes a granularity-aware top- $K$ per view, fuses with RRF, and reranks by evidence to $F_{\text{key}}$ . The loop terminates by pool-saturation, Jaccard stability, or $T_{\max}$ , so it bounds observation MISS without false confidence on truly absent facts.

Configuration	MISS / 1540	Rate
Controlled study (oracle vector store)
$C_{1}$ Vanilla RAG + $Q$	459	29.8%
$C_{2}$ AgenticRAG + $Q$	283	18.4%
$C_{3}$ Vanilla RAG + $Q$ + $F_{\text{key}}$	174	11.3%
$C_{4}$ AgenticRAG + $Q$ + $F_{\text{key}}$ ( $\star$ )	67	4.4%
Leave-one-out from $C_{4}$

Appendix

Table 15: Adapted AgenticRAG ablation on the LoCoMo POS subset (

N\!=\!1540

). (

\star

)

C_{4}

is the default in all main experiments. The adapted agentic-RAG routine comprises four pipeline mechanisms (multi-view dense retrieval, BM25 sparse view, LLM rerank, multi-round); evidence-aware query is an orthogonal input-level mechanism. Leave-one-out contributions: evidence-aware

+14.0

, multi-view

+2.5

, rerank

+2.0

, multi-round

+1.4

, BM25

+0.7

pp. McNemar

C_{1}

vs.

C_{4}

\chi^{2}\!=\!298.4

(

p\!<\!10^{-3}

);

C_{3}

vs.

C_{4}

\chi^{2}\!=\!41.2

(

p\!<\!10^{-3}

). The four-mechanism leave-one-out sum (

6.6

pp) is close to the net pipeline contribution (

C_{4}

vs.

C_{3}

6.9

pp), indicating near-additivity with

0.3

pp residual overlap.

Figure 7: MemWiki as a store-for-search auxiliary layer. The raw memory $\mathcal{M}$ is organised into a wiki-style index whose source pages and hub pages are flattened into four AuxRecord classes (factual, tag, query-shaped, aggregate). The system’s native retriever runs over $\mathcal{M}\!\cup\!\mathcal{A}$ ; any auxiliary hit pulls in its source records via $\mathrm{src}(\cdot)$ , and a single rerank pass orders the merged pool subject to a $30\%$ auxiliary-record quota.

System	Tag	Fact	Query	Sum.	Sum	All	Crowd loss
TiMem	+0.1	+0.4	+0.2	+0.1	+0.8	+0.3	$-62.5\%$
O-Mem	+0.4	+0.9	+0.6	+0.5	+2.4	+1.5	$-37.5\%$
EverMemOS	+0.2	+0.5	+0.4	+0.3	+1.4	+1.0	$-28.6\%$
MemOS	+0.6	+1.5	+1.1	+0.8	+4.0	+2.6	$-35.0\%$
MemBox	+1.1	+2.6	+1.7	+1.4	+6.8	+4.2	$-38.2\%$
MemoryOS	+0.8	+1.8	+1.3	+1.0	+4.9	+3.0	$-38.8\%$

Appendix

Table 16: MemWiki AuxRecord single-class ablation (

\Delta

Final Acc, pp). Tag =tag records, Fact =factual records, Query =query-shaped records, Sum. =aggregate summaries. Sum =sum of singletons; All =all four enabled; Crowd loss =

1\!-\!\text{All}/\text{Sum}

, the share of gain dropped to the

30\%

auxiliary quota.

System	$b$	$c$	$c\!-\!b$	$\chi^{2}$	$p$ -value	sig.
TiMem	19	25	6	0.82	0.366	—
O-Mem	20	50	30	12.86	$<\!10^{-3}$	$\star\star\star$
EverMemOS	20	40	20	6.67	0.010	$\star$
MemOS	28	80	52	25.04	$<\!10^{-3}$	$\star\star\star$
MemBox	47	130	83	38.92	$<\!10^{-3}$	$\star\star\star$
MemoryOS	38	98	60	26.47	$<\!10^{-3}$	$\star\star\star$

Appendix

Table 17: Per-system McNemar tests for MemWiki on LoCoMo (

N\!=\!1986

paired observations per system).

b

= baseline correct

\wedge

MemWiki wrong ( spoils );

c

= baseline wrong

\wedge

MemWiki correct ( recovers );

\chi^{2}\!=\!(b\!-\!c)^{2}/(b\!+\!c)

with

df\!=\!1

. Symbols:

\star

p<0.05

;

\star\star\star

p<10^{-3}

. Six of seven systems show significant improvement; TiMem’s marginal

+0.3

pp gain is not significant, consistent with its

-62.5\%

crowd loss (Table 16 ).

System	Acc $\uparrow$		Ret $\downarrow$
System	Bsl	+ MemWiki	$\Delta$	$\Delta$
TiMem	75.3	75.6	$+0.3$	$-0.9$
O-Mem	74.9	76.4	$+1.5$	$-2.3$
EverMemOS	72.4	73.4	$+1.0$	$-1.6$
MemOS	70.1	72.7	$+2.6$	$-3.0$
MemBox	63.2	67.4	$+4.2$	$-5.0$

Appendix

Table 18: MemWiki retrieval-side intervention on LoCoMo with gpt-4o-mini . MemWiki is built per-system from each baseline’s own memory export, then re-injected through L1/L2/L3 (

\leq\!30\%

auxiliary quota). Final Acc (%) and retrieval defect rate Ret (%) before/after intervention. Bold red : largest gain; blue underline : second-largest. Aggregate McNemar over

7

systems

\times 1986

paired observations:

\chi^{2}\!=\!145.8

(

p\!<\!10^{-3}

); per-system breakdown in Appendix C.7 , Table 17 .

System	bsl	Final Acc (%) under MemWiki quota $\alpha$
System	bsl	$.10$	$.20$	$.30^{\star}$	$.40$	$.50$	$.70$	$1.0$
TiMem	75.3	75.5	75.7	75.6	75.3	74.8	73.4	69.8
O-Mem	74.9	75.4	76.2	76.4	76.1	75.5	73.9	70.4
EverMemOS	72.4	72.9	73.3	73.4	73.5	72.8	70.6	67.5
MemOS	70.1	71.0	72.1	72.7	72.6	71.9	70.2	66.5
MemBox	63.2	64.7	66.3	67.4	67.5	66.8	64.6	60.5

Appendix

Table 19: MemWiki

\alpha

-quota ablation on LoCoMo (gpt-4o-mini).

\alpha

caps the fraction of MemWiki auxiliary records in the top-

K\!=\!10

context, so

\alpha K

is the maximum number of MemWiki slots (

\alpha\!=\!.30

allows

\leq\!3

\alpha\!=\!1.0

removes the cap). (

\star

)

\alpha\!=\!0.30

is the default used in all main experiments. Bold red : per-system peak. Per-system optima span

\alpha\!\in\!\{0.20,0.30,0.40\}

, yet every system at

\alpha\!=\!0.30

is within

0.2

pp of its own peak. The plateau region

\alpha\!\in\![0.30,0.40]

gives identical mean Final Acc (

70.6\%

); removing the quota (

\alpha\!=\!1.0

) drops mean Final Acc to

64.1\%

4.0

pp below the no-MemWiki baseline.

System	Metric	$K\!=\!1$	$K\!=\!2$	$K\!=\!3$	$K\!=\!5$	$K\!=\!8$	$K\!=\!10$	$K\!=\!12$	$K\!=\!15$	$K\!=\!20$
TiMem	Hit@ $K$	38.4	47.5	55.7	64.7	71.0	73.5	75.2	77.3	79.5
TiMem	Final Acc	64.2	68.5	70.5	73.6	75.0	75.3	75.5	75.3	74.5
O-Mem	Hit@ $K$	36.4	45.5	54.3	63.2	68.7	71.1	73.0	75.2	77.7
O-Mem	Final Acc	63.8	68.0	70.2	73.0	74.7	74.9	74.8	74.0	72.6
EverMemOS	Hit@ $K$	34.9	43.8	52.8	61.8	67.2	69.4	71.4	73.5	76.0
EverMemOS	Final Acc	61.6	65.8	67.9	70.5	72.1	72.4	72.3	71.6	70.0

Appendix

Table 20: Combined top-

K

sensitivity on LoCoMo with gpt-4o-mini . Hit@

K

(%, POS subset) is monotone increasing in

K

, whereas Final Acc (%) peaks at

K\!\in\!\{8,10\}

for every system. Bold red marks each row’s Final Acc peak; the two yellow columns mark the

K\!\in\!\{8,10\}

sweet spot shared by Hit@

K

’s elbow and Acc’s maximum. Beyond this band, Acc declines although Hit@

K

keeps rising; the gap is explained by the NOI rate in Table 21 . System rankings under both metrics are stable across all

K

(Spearman

\rho\!\geq\!0.96

between any two columns).

$K$	RF	LATE	NOI	Ret. total
$1$	16.1	0.0	0.2	16.3
$2$	14.4	0.3	0.5	15.2
$3$	12.5	0.7	1.0	14.2
$5$	9.8	1.4	2.1	13.3
$8$	6.6	2.9	3.7	13.2
$10$	5.0	3.4	5.5	13.9

Appendix

Table 21: Retrieval-layer code rates (%) on LoCoMo POS as a function of top-

K

(mean over

7

systems). RF ( retrieval failure : key fact never recalled) decays monotonically as more candidates are fetched. LATE (key fact recalled below rank-

5

) and NOI (low signal-to-noise ratio) grow with

K

. The total retrieval defect rate is U-shaped, with a flat minimum at

K\!\in\!\{8,10\}

— the same range that maximises Final Acc in Table 20 .

Session	Rule
$1$ – $4$	Initial state $\sigma_{0}$ (introduce baseline facts)
$5$	Hold $\sigma_{4}$ (let old facts settle)
$6$	Main-core change (e.g. city: SH $\to$ BJ )
$7$	Auxiliary cascade (e.g. commute )
$8$	$\rho$ -driven single-point update
$9$	Rollback for $30\%$ of personas

Appendix

Table 22: State trajectory rules used to evolve

\sigma_{t}

across sessions in DynaMem-Bench; cascade and rollback target dynamic-state failures.

Figure 8: DynaMem-Bench construction. An offline phase fixes everything the cross-system comparison requires: $10$ personas, a $10$ -session state schedule with cascade and rollback events, must-expose sets $E_{t}$ , seed utterances, evaluation questions, and oracle contexts. An on-policy phase is repeated per system: a user simulator drives the dialogue conditioned on $\sigma_{t}$ and residual exposure, with a force-expose fallback at turn $19$ . Probe questions go through a read-only adapter so they cannot pollute $\mathcal{M}$ .

System	LoCoMo10	DynaMem-Bench	$\Delta$ (pp)
TiMem	75.3	71.2	$-4.1$
O-Mem	74.9	68.9	$-6.0$
MemOS	70.1	62.5	$-7.6$
EverMemOS	72.4	61.0	$-11.4$
MemoryOS	60.8	56.4	$-4.4$
MemBox	63.2	55.9	$-7.3$

Appendix

Table 23: Static vs. dynamic comparison. The mean accuracy drop (

-7.3

pp) is absorbed by retrieval (

+3.8

pp) and encoding (

+2.0

pp); generation defects decrease (

-1.5

pp), reflecting that DynaMem-Bench has fewer multi-hop temporal questions than LoCoMo . Pure retrieval-driven systems (GAM

-10.6

, EverMemOS

-11.4

) degrade most.

Difficulty subset	N (asks)	Correct	Acc
state-tracking (multi-session re-asks)	2800	1500	53.6
immediate cascade (3 personas)	609	397	65.2
delayed cascade (7 personas)	1421	831	58.5
rollback (3 personas, sess. 9)	602	285	47.3

Appendix

Table 24: Per-subset accuracy on DynaMem-Bench (7 systems,

\times\,7

aggregation). Rollback is hardest: half the answers go to the wrong direction of the state flip. State-tracking (53.6%) shows that consistent multi-session truth-tracking is unsolved.

System	Pre-flip	Post-flip	$\Delta$
TiMem	73.4	62.1	$-11.3$
O-Mem	71.0	60.7	$-10.3$
MemOS	64.8	53.5	$-11.3$
EverMemOS	63.1	45.5	$\mathbf{-17.6}$
MemoryOS	58.7	52.6	$-6.1$
MemBox	57.9	47.4	$-10.5$

Appendix

Table 25: Failure-recovery analysis on DynaMem-Bench. For each persona we identify session

\tau

at which a fact is rewritten (cascade or rollback) and split that fact’s re-asks into the pre-flip window (asked before

\tau

, ground truth = original) and the post-flip window (asked after

\tau

, ground truth = revised). Accuracy in the post-flip window measures the system’s ability to overwrite stale evidence. Retrieval-driven systems (EverMemOS, GAM) suffer the largest drops because their stores append rather than supersede records.

System	Acc		Enc		Ret		Gen
System	GPT-4o	Qwen3	GPT-4o	Qwen3	GPT-4o	Qwen3	GPT-4o	Qwen3
TiMem	71.2	70.3	7.6	7.9	19.0	19.4	4.3	4.4
O-Mem	68.9	69.6	7.5	7.3	21.5	21.1	4.5	4.4
MemOS	62.5	61.4	9.0	9.4	25.4	25.9	4.9	5.1
EverMemOS	61.0	60.6	9.6	9.7	26.1	26.4	4.7	4.8
MemoryOS	56.4	55.8	11.0	11.2	28.4	28.8	5.1	5.2

Appendix

Table 26: DynaMem-Bench under two user simulators (GPT-4o vs. Qwen3-30B-Instruct ), with the memory backbone fixed at gpt-4o-mini and the must-expose schedule and state evolution preserved. Mean accuracy shifts by

0.1

pp; per-system

|\Delta\text{Acc}|\!\leq\!1.1

pp (mean

0.73

pp). The ranking is preserved up to a single transposition between MemoryOS and MemBox, whose baseline accuracies differ by only

0.5

pp; Spearman

\rho\!=\!0.964

, Pearson

r\!=\!0.995

Judge	N	LLM–human agr.	Cohen’s $\kappa$
Answer judge	120	$96.7\%$	$0.91$

Appendix

Table 27: Answer-level judge validation on

120

randomly sampled system–question pairs. Agreement and Cohen’s

\kappa

compare the main answer judge with adjudicated human labels; no human–human ceiling is assumed.

Judge target	#Cases	Human–human	LLM–human	Defect F1
		agr./ $\kappa$	agr./ $\kappa$
Encoding Examiner	150	$96.0/0.91$	$94.7/0.89$	$0.94$
Retrieval Examiner	150	$96.7/0.92$	$95.3/0.90$	$0.95$
Generation Examiner	150	$94.7/0.88$	$93.3/0.86$	$0.92$
All Examiners	450	$95.8/0.90$	$94.4/0.88$	$0.94$

Appendix

Table 28: Examiner-level judge validation on

150

randomly sampled questions, yielding

450

audited Examiner decisions. Human–human reports agreement between the two independent annotators; LLM–human compares the main Examiner output with adjudicated human labels; Defect F1 is computed over the induced defect-code set.

Verification target	Random sample	Human–human	Gemini–human
		agr./ $\kappa$	agr./ $\kappa$
Schema–state consistency	80 bundles	$98.8/0.95$	$97.5/0.92$
Must-expose coverage	200 exposure events	$99.0/0.96$	$98.5/0.94$
POS oracle support	200 POS asks	$97.5/0.91$	$96.5/0.88$
NEG unanswerability	120 NEG asks	$96.7/0.89$	$95.8/0.86$
Trap / temporal consistency	120 tracking asks	$97.5/0.91$	$96.7/0.88$

Appendix

Table 29: Reliability of Gemini-2.5-Pro as the DynaMem-Bench construction verifier against adjudicated human labels. The audit sample is drawn by uniform random sampling and includes both pass and fail verdicts overall fail rate

\approx 18\%

LLM Memory

Large language model memory studies how models store, retrieve, update, and forget information across prompts, tasks, and interactions.

See all 95 papers

MemFail: Stress-Testing Failure Modes of LLM Memory Systems
May 26, 2026Ishir Garg, Neel Kolhe, Dawn Song +1LLM Agent Memory LLM Memory
University of California, Berkeley
MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
Jul 14, 2026Xixuan Hao, Zeyu Zhang, Zehao Lin +6Conversational Memory Persistent Memory for Language Models
MemTensor (Shanghai) Technology Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou) · Renmin University of China
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University

May 26, 2026J/K move · Enter open · S save

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

Ishir Garg, Neel Kolhe, Dawn Song, Xuandong Zhao

University of California, Berkeley

LLM Agent Memory LLM Memory AI Agent Benchmarks Memory Agent Benchmarks Agent Memory Management

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.

Abstract

LLM Memory

Explore similar work

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory