Can Agent Memory Systems Track Evolving State?

Xinyi Fan; Miri Liu; Ruozhen Yang; Siru Ouyang; Jiawei Han

cs.AIAug 20, 2026

Can Agent Memory Systems Track Evolving State?

Authors: Xinyi Fan, Miri Liu, Ruozhen Yang, Siru Ouyang, Jiawei Han

Organizations: University of Illinois Urbana-Champaign

Abstract

As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmarks focus largely on recall-shaped tasks, we argue an effective memory system must track the evolving state of the world; as facts, constraints, and decisions are revised over a long interaction, answers must reflect the current state and not a superseded one. We define this capability as state tracking and instantiate it in StateMemBench, a benchmark of 234 multi-session scenarios spanning two conversation-length regimes. Its closed-pool grading scores whether an answer reflects the current state, the superseded state, or fails otherwise, separating state-tracking failures from other errors by construction. Our analysis shows that this task is challenging for existing memory systems, retrieval-augmented baselines, and long-context baselines. We then present StateMem, a state-first memory method that explicitly tracks supersession and relational dependencies, and show it improves current-state accuracy over the strongest same-backbone baseline by 1.8x (0.205 -> 0.363) on DeepSeek-V4-Flash and over the strongest memory system by 1.6x (0.149 -> 0.233) on Qwen-3.5-9B, while remaining competitive with the long-context baselines. Finally, we show the same state approach can be applied as a lightweight single-call wrapper over existing memory systems, lifting current-state accuracy by +32 to +67 points on StateMemBench across six memory and retrieval backends. A length- and cost-matched control attributes +15 to +32 of those points to state structure rather than added context.

Figures & tables

Figure 1: StateMemBench targets state tracking: maintaining currently operative values of facts, rules, and derived quantities under cross-session revision, separable from recall.

MemoryArena
Benchmark	$n$	drift	retr.	comp.	schema	reason	FA
Shop.	200	63.5	22.5	2.5	0.0	9.5	–
Search	200	38.5	25.5	11.0	17.5	4.5	–
Travel	200	19.0	11.0	46.5	21.0	0.0	–
LongMemEval oracle
LME †	36	44.4	0.0	0.0	25.0	2.8	27.8

Table 1: Cross-benchmark failure-mode distribution. Each cell is the share of that benchmark’s confirmed failures assigned to each bucket; the drift column is highlighted. † On LongMemEval oracle every gold fact is in the prompt by construction, so no failure there can be a retrieval failure. The small-

n

\tau^{2}

-bench-Z result (

n{=}16

) is reported in § 3.3 . Labeling, adjudication, and per-row details: Appendix A .

Benchmark	Multi-sess. dialogue	Scale (native unit)	Updates central	Super- session	Verifiable gold	Drift scored	Anti-update controls	Paired horizons
LoCoMo ( 2024 )	✓	$\sim$ 300 turns	✗	✗	✗	✗	✗	✗
LongMemEval ( 2025 )	✓	$\sim$ 40–80 sess. ( $S$ )	(✓)	(✓)	✗	✗	✗	✗
MemoryAgentBench ( 2025 )	(✓)	varies	(✓)	(✓)	(✓)	✗	✗	✗
MemoryArena ( 2026 )	(✓)	varies	✗	✗	✓	✗	✗	✗
STATE-Bench ( 2026 )	✗	450 tasks	✗	✗	(✓)	✗	(✓)	✗
StateMemBench (ours)	✓	$\sim$ 600 turns	✓	✓	✓	✓	✓	✓

Table 2: Comparison with agent-memory and long-horizon benchmarks with state-tracking evaluation. (✓) = partial satisfied. Scale is reported in each benchmark’s native unit. Column definitions and per-benchmark justifications are in Appendix C .

Figure 2: StateMem has three stages: an ingestion stage in which a conversation turn is parsed into state units, an update stage that updates the value of existing units, and a test time stage where the agent draws upon the valid state to answer the question. Here, we show how StateMem would handle a finance-based scenario.

	Condition	Overall	Short	Long
Long-context	Qwen-3.5-9B	0.149	0.137	0.167
	Qwen-3.6-35B-A3B	0.097	0.063	0.146
	GPT-5.4-Nano	0.277	0.311	0.227
	DeepSeek-V4-Flash	0.149	0.132	0.174
Qwen-3.5-9B	No memory	0.006	0.000	0.015
	BM25	0.125	0.095	0.167

Table 3: State-agreement accuracy (gold rate) on StateMemBench :

190

in short scenarios (

\sim

165 turns, one probe each) and

132

44

long fused scenarios (

\sim

600 turns, three probes each), graded by a fixed deepseek-v4-pro judge. Memory systems (Mem0 ( Chhikara et al. 2025 ) , A-Mem ( Xu et al. 2025 ) , LightMem ( Fang et al. 2026 ) , MemoryOS ( Kang et al. 2025 ) , StateMem), retrieval baselines (BM25, text-embedding-3-small ( OpenAI 2024 ) ), and graph-RAG systems (nano-graphrag, HippoRAG, LightRAG, GraphRAG) run on each substrate (Qwen-3.5-9B ( Qwen Team 2026 ) , thinking off; deepseek-v4-flash ( Xu et al. 2026 ) , thinking off); long-context baselines are raw models with full history in context. StateMem ablations (indented) excluded from best-marking. Per column within each panel, best bold , second best underlined . *** StateMem’s overall gold rate beats the strongest memory baseline on both backbones (paired McNemar test; both p < 0.001)

	LongMemEval		LoCoMo
Condition	Qwen-9B	DS-V4	Qwen-9B	DS-V4
Long context	0.550	0.666	0.612	0.587
No memory	0.062	0.056	0.010	0.005
Mem0	0.566	0.594	0.481	0.462
A-Mem	0.520	0.510	0.266	0.279
LightMem	0.064	0.056	0.027	0.020

Table 4: Accuracy on LongMemEval (full set,

n{=}500

k{=}20

) and LoCoMo (

n{=}1{,}985

), with each method run on both substrates (Qwen-3.5-9B; deepseek-v4-flash; thinking off) and all answers graded by a fixed deepseek-v4-pro judge. Per column, best in bold , second best underlined .

	Qwen-3.5-9B		DeepSeek-V4-Fl.
Method	I-P	D-R ( $\downarrow$ )	I-P	D-R ( $\downarrow$ )
full context	263	65.5	258	64.0
Mem0	254	62.7	268	59.9
BM25	243	60.9	244	59.9
LightMem	109	32.0	52	14.9
StateMem (ours)	290	64.3	286	49.1

Table 5: Closed-pool (

n{=}322

) drift-rate and in-pool engagement, methods that produce non-trivial in-pool counts on at least one backbone. In-pool (I-P) = method produced one of the closed-pool options. Drift-rate (D-R) = drift /

n

(errors are 0 for every method). StateMem’s I-P is bold .

		Qwen-3.5-9B			deepseek-v4-flash
	Backend	Alone	+Ctrl	+SMW	Alone	+Ctrl	+SMW
StateMemBench	Mem0	25.0	28.3	56.7	28.3	56.7	71.7
	A-Mem	23.3	35.0	56.7	33.3	51.7	75.0
	LightMem	3.3	30.0	61.7	1.7	51.7	68.3
	MemOS	5.0	25.0	56.7	1.7	50.0	68.3
	BM25	20.0	31.7	56.7	21.7	48.3	70.0

Table 6: StateMemWrapper composition sweep ; each backend ingests with its unmodified pipeline and is evaluated on the same frozen paired

n{=}60

set per benchmark (StateMemBench: 30 short + 30 long scenarios,

k{=}10

; LongMemEval:

k{=}20

) under three answer conditions: the backend alone , a length- and cost-matched generic control ( +Ctrl ), and our state-tracing wrapper ( +SMW ). All answers are scored by a fixed deepseek-v4-pro judge. Best condition per row in bold . Dense on LongMemEval/Qwen has

n{=}59

(one case failed at ingest).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$n$	agr (%)	$\kappa$	PABAK	AC1	$\kappa_{\text{mode}}$
MA-shopping	179	91.6	0.17	0.83	0.91	0.00
MA-search	110	72.7	0.31	0.45	0.55	0.09
LME oracle	36	77.8	0.67	0.56	0.56	0.01
LME oracle (cross)	36	69.4	0.37	0.39	—	—

Appendix

Table 7: Inter-judge agreement on the binary drift decision: two same-model DeepSeek-V4 decodes (top), and DeepSeek-V4

\times

GPT-4o (bottom row cross family). MA-travel is omitted (

2

doubly-labeled cases). MA-shopping is the one cell where

\kappa

misrepresents reliability: both passes call

{>}90\%

of failures drift pre-filter , which crushes the chance correction despite

91.6\%

raw agreement.

Arm	correct	rate
thinking off	42/50	84.0%
thinking on	38/50	76.0%
$\Delta$		$-8.0$ pp

Appendix

Table 8: Paired reasoning A/B. Reasoning does not close the gap: the

-8.0

pp point estimate rests on

3

rescues against

7

new failures, which does not establish a direction (exact McNemar

p{=}0.34

). The off-pass

\to

on-fail cell includes the case dissected in Appendix B .

Source	DOI
Biomedical (bioRxiv)	10.1101/482679
	10.1101/512657
Computer Science (arXiv)	10.48550/arXiv.2407.13248
	10.48550/arXiv.2604.10985
	10.48550/arXiv.2604.11543
	10.48550/arXiv.2604.21253

Appendix

Table 9: Source DOIs used to ground synthetic scenario generation, grouped by domain.

		By horizon			By category					By domain
	Condition	Short	Long	Overall	Status	Sequence	Salience	Anti-trap	Compound	Finance	Shopping	Research
Long-context	Qwen-3.5-9B	0.137	0.167	0.149	0.078	0.000	0.098	0.692	0.159	0.118	0.197	0.118
	Qwen-3.6-35B-A3B	0.063	0.146	0.097	0.026	0.000	0.059	0.632	0.023	0.110	0.165	0.000
	GPT-5.4-Nano	0.311	0.227	0.277	0.172	0.153	0.569	0.615	0.114	0.344	0.402	0.059
	DeepSeek-V4-Flash	0.132	0.174	0.149	0.103	0.056	0.059	0.641	0.091	0.161	0.244	0.020
Qwen-3.5-9B	No memory	0.000	0.015	0.006	0.009	0.000	0.000	0.026	0.000	0.000	0.000	0.020

Appendix

Table 10: State-agreement accuracy on StateMemBench with deepseek-v4-pro judge. Left block: accuracy per horizon and

n

-weighted overall. Middle block: accuracy by probe category over all 322 probes (status

n{=}116

, sequence

n{=}72

, salience

n{=}51

, anti-trap

n{=}39

, compound

n{=}44

); Right block: accuracy by domain (finance

n{=}93

, shopping

n{=}127

, research

n{=}102

). Dense retrieval embeds with OpenAI text-embedding-3-small on both substrates; answers come from the panel’s substrate model. Per column within each panel, best in bold , second best underlined ; StateMem ablations excluded from marking.

			LongMemEval question type
	Condition	Overall	SS-user (70)	SS-asst. (56)	SS-pref. (30)	Multi-sess. (133)	Know.-upd. (78)	Temporal (133)
Qwen-3.5-9B	Long context	0.550	0.900	0.946	0.333	0.436	0.744	0.248
	No memory	0.062	0.086	0.054	0.067	0.053	0.038	0.075
	Mem0	0.566	0.871	0.607	0.533	0.602	0.603	0.338
	A-Mem	0.520	0.886	0.929	0.467	0.361	0.679	0.233
	LightMem	0.064	0.086	0.073	0.000	0.053	0.038	0.092

Appendix

Table 11: Accuracy by question type on LongMemEval (top; full set,

n{=}500

k{=}20

) and LoCoMo (bottom;

n{=}1{,}985

; all retrieval

k{=}20

), both substrates (thinking off), graded by a fixed deepseek-v4-pro judge. Question-type

n

in the header. Knowledge-update and temporal-reasoning (LME) and temporal-reasoning (LoCoMo) are the drift-shaped subsets; LoCoMo adversarial rewards abstention. deepseek LME memory-system cells are from the final unified

n{=}500

judge pass (all arms scored together). Per column within each panel, best in bold , second best underlined .

			By category					By domain
	Backend	Cond.	Status	Seq.	Sal.	Anti	Comp.	Fin.	Shop.	Res.
Qwen-3.5-9B	Mem0	alone	0	8	0	92	10	27	17	20
		+Ctrl	17	17	23	92	0	23	33	40
		+SMW	50	58	77	92	40	64	56	70
	A-Mem	alone	0	0	0	100	10	18	28	20
		+Ctrl	8	25	23	100	10	27	28	50

Appendix

Table 12: StateMemWrapper sweep on StateMemBench , by probe category and domain (gold %, frozen

n{=}60

mixed set; category

n

: status 12, sequence 12, salience 13, anti-trap 13, compound 10; domain

n

: finance 22, shopping 18, research 20; deepseek-v4-pro judge). Anti-trap probes are controls (correct behaviour is not updating), hence high accuracy even for weak arms — and +SMW does not inflate there, i.e. it updates selectively rather than aggressively. Per-cell

n

10

–

13

; splits are exploratory. +SMW rows shaded.

	Qwen-3.5-9B				deepseek-v4-flash
	+Ctrl		+SMW		+Ctrl		+SMW
Backend	Fix $\uparrow$	Brk $\downarrow$	Fix $\uparrow$	Brk $\downarrow$	Fix $\uparrow$	Brk $\downarrow$	Fix $\uparrow$	Brk $\downarrow$
Mem0	10	11	18	12	15	7	15	6
A-Mem	7	13	17	8	18	8	16	4
LightMem	4	0	14	0	17	0	20	1
MemOS	4	3	10	0	17	2	20	2

Appendix

Table 13: StateMemWrapper sweep on LongMemEval: paired outcome transitions vs. the backend alone (frozen

n{=}60

; Dense/Qwen has

n{=}59

, one case failed at ingest. The set is drawn from the multi-session question type by construction, so no category split applies). For each backend, each condition’s answers are compared case-by-case against the alone answers: Fix = alone wrong

\to

condition right; Brk = alone right

\to

condition wrong; Fix

-

Brk equals the net case change in Table 6 . +SMW is net-positive for every backend on both substrates; the control is net-negative for three Qwen backends (Mem0, A-Mem, Dense).

Method	Correct	Drift	Other-opt	Off-pool	Drift-rate
Long-context (V4-Flash)	48	206	4	64	64.0%
Long-context (GPT5.4n)	89	159	1	73	49.4%
StateMem (ours)	117	158	11	36	49.1%
extraction-only	56	226	6	34	70.2%
+ supersession	96	184	9	33	57.1%
$-$ dependency-propagation	120	157	9	36	48.8%

Appendix

Table 14: Closed-pool failure modes on DeepSeek-V4-Flash,

n{=}322

(combined Set A+B; errors are 0 for every method). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option ( matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift /

n

Method	Correct	Drift	Other-opt	Off-pool	Drift-rate
Long-context-9B	48	211	4	59	65.5%
Long-context-35B	31	215	6	68	67.2%
StateMem (ours)	75	207	8	32	64.3%
extraction-only	50	239	9	24	74.2%
+ supersession	55	230	10	27	71.4%
$-$ dependency-propagation	71	212	13	24	66.2%

Appendix

Table 15: Closed-pool failure modes on Qwen-3.5-9B,

n{=}322

(combined Set A+B; two cells

n{=}320

after 2 grading drops, errors otherwise 0). Other-opt means an answer that was neither the drift target nor the gold truth but was in the closed-pool plausible options (constructed at generation time). Off-pool means the method did not answer with a recognizable option ( matched_option null). Drift-rate is the percentage of answers selecting the drift target over all answers (drift /

n

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 191 papers

MEME: Multi-entity & Evolving Memory Evaluation
May 12, 2026Seokwon Jung, Alexander Rubinstein, Arnas Uselis +2Agent Memory Persistent Memory for Language Models
1KAIST AI · 2Tübingen AI Center, University of Tübingen · 3NAVER AI Lab
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University
MemDerive: A Diagnostic Benchmark for Agent Memory over Evolving Data
Oct 18, 2026Sen Gao, Yongan Li, Han Chen +5Agent Memory Memory Agent Benchmarks
National University of Singapore, Singapore, Singapore · Shanghai Jiao Tong University, Shanghai, China · Zero Gravity Labs, Singapore, Singapore

May 12, 2026J/K move · Enter open · S save

MEME: Multi-entity & Evolving Memory Evaluation

Seokwon Jung, Alexander Rubinstein, Arnas Uselis, Sangdoo Yun+1

1KAIST AI · 2Tübingen AI Center, University of Tübingen · 3NAVER AI Lab

Agent Memory Persistent Memory for Language Models LLM Agent Evaluation Memory Agent Benchmarks Memory Management for LLM Agents

Open paper AskPDF arXiv

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 as its internal LLM partially closes the gap, but at ~70x the baseline cost, indicating closure currently depends on configurations that are not practical at scale. Code and data are available on the project page: https://seokwonjung-jay.github.io/meme-eval/.

Abstract

Agent Memory

Explore similar work

MEME: Multi-entity & Evolving Memory Evaluation

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

MemDerive: A Diagnostic Benchmark for Agent Memory over Evolving Data