D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory

Xule Liu; Yijun Liu; Chao Li; Shao Kun

cs.AIAug 18, 2026

D $^2$ ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory

Authors: Xule Liu, Yijun Liu, Chao Li, Shao Kun

Organizations: Xiaomi Inc.

Abstract

Memory is a key capability of LLM agents. Persistent memory extends this across sessions---enabling recall, revision, and personalization. Yet its multi-stage pipeline (ingestion, retrieval, filtering, generation) makes failures difficult to localize: end-to-end evaluation reveals that an error occurred, but not which stage caused it. Existing evaluations often report aggregate performance without paired statistical comparisons, slice-level non-regression checks, or stage-level diagnostic traces. We propose D $^2$ ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol whose outer diagnostic gate promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability. We further introduce DCR, a graded observability metric that measures whether failures remain localizable, and D $^2$ ACCI-Eval, a reusable artifact for gate replay. We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Five paired ablations show that supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains (+1.9 to +3.7pp, all p $\le$ .003). In contrast, BM25/RRF is retained as a monitored feature flag---a distinction invisible to aggregate-only evaluation. A diagnostic audit shows enriched traces substantially improve root-cause agreement over result-only relabeling. Diagnostic artifacts reach 98--100% DCR@3 versus 0% for results-only logs. These results establish that robust memory-system iteration demands traceable, statistically grounded, and regression-aware evidence---exactly the gap D $^2$ ACCI fills.

Figures & tables

Figure 1: (a) Current end-to-end evaluation reports only aggregate accuracy; internal pipeline states are hidden and failures cannot be attributed to a specific stage. (b) D 2 ACCI emits per-stage traces, enabling paired gates to localize failures and drive targeted repairs.

Figure 2: MemStack under the D 2 ACCI dual-loop protocol. Top : the inner memory loop processes queries through feature-flagged stages (FF; independently togglable by the outer gate): ingestion, multi-layer storage, retrieval, constraint filtering, context assembly, and generation; each emits a typed diagnostic trace (IDs, layers, ranks, filters, budget, judge). Ablation targets (✗) mark paired ablations in Table 3 . Bottom : the outer diagnostic loop feeds traces into D 2 ACCI -Eval (paired stats, slice deltas, DCR , discordance, audit), producing a gate decision that updates inner-loop feature flags for iteration $t{+}1$ . The iteration timeline shows automatic gate replay including the $t_{3}$ rejection.

Benchmark	Eval set	Ours	MemBrain	$\Delta$	Representative slices (category / ( count , accuracy ))
LoCoMo	1540 non-adv QA	93.59%	93.25%	+0.34pp	Single-hop	Multi-hop	Temporal	Open
LoCoMo	1540 non-adv QA	93.59%	93.25%	+0.34pp	( 841 , 96.2 )	( 282 , 89.5 )	( 321 , 92.4 )	( 96 , 86.8 )
LongMemEval	500/500 questions	90.93%	85.60%	+5.33pp	Single-user	Knowledge-update	Multi-session	Temporal
LongMemEval	500/500 questions	90.93%	85.60%	+5.33pp	( 70 , 99.5 )	( 78 , 93.6 )	( 133 , 85.0 )	( 133 , 89.0 )
PersonaMem-V2	200 personas, 5000 MCQ	57.20%	55.72%	+1.48pp	Sensitive	Forget	Neutral	Therapy
PersonaMem-V2	200 personas, 5000 MCQ	57.20%	55.72%	+1.48pp	( 511 , 86.3 )	( 1048 , 66.9 )	( 858 , 51.2 )	( 627 , 44.5 )

Table 1: Main results and representative slice breakdowns. External rows are reference points, not fully normalized same-stack baselines. Slice-level reporting is required because memory interventions are frequently non-monotonic.

Condition vs. Human A	Fine $\kappa$	Coarse $\kappa$
Enriched GPT-4o trace relabel	0.571	0.619
Result-only GPT-4o relabel	0.258	0.272
Human B	0.674	0.711

Table 2: Audit-label contrast on the 60-case packet. Human B is an independent annotator; all rows are pre-adjudication.

Benchmark	Comparison	$n$	Ablated	$\Delta$	Paired evidence			Gate decision
Benchmark	Comparison	$n$	Ablated	$\Delta$	$p$	95% CI	Full wins / Ablated wins	Metric-only	D 2 ACCI gate
LoCoMo	No BM25/RRF	1,540	94.00%	$-0.41$ pp	.4426	[ $-1.36$ , 0.56]	27/34	Demote	Feature-flag
LoCoMo	No Supplement	1,540	90.89%	$+2.71$ pp	.0009	[1.26, 4.20]	90/50	Promote	Accept; MH monitor
LME	No BM25/RRF	500	89.93%	$+1.00$ pp	.4583	[ $-1.13$ , 3.13]	17/12	Promote	Feature-flag
LME	No Session	500	87.27%	$+3.67$ pp	.0026	[1.33, 6.00]	28/9	Promote	Accept; KU monitor
PMem	No Forget Guard	5,000	55.28%	$+1.92$ pp	.0030	[0.64, 3.18]	560/464	Promote	Accept; slice monitor

Table 3: Completed paired diagnostic ablations with metric-only vs. D 2 ACCI gate decisions. CI is bootstrap 95% for Full

-

Ablated; Full wins / Ablated wins count correct examples. Metric-only: accept if

\Delta{>}0

, demote if

\Delta{<}0

; D 2 ACCI adds statistical and slice-level checks.

Figure 3: LoCoMo iteration trajectory with automatic gate-replay decisions. Each transition is produced by the replay script (

<

1 s per candidate); all five decisions match the human-in-the-loop outcome. The dashed red segment marks

t_{3}

: traces localized over-merging, triggering automatic rejection.

AP ∗	Trigger	Observed Behavior	Resolution
OM	Threshold $(0.75)$	Facts collapse into a single memory representation	Reverted
DA	Option-aware filter	Retrieval surfaces distractor evidence	Rejected
NME	Gated expansion	Recovers some targets, introduces off-target noise ( $57.34\%\to 57.24\%$ )	Slice-gated
∗ AP: Anti-pattern

Table 4: Representative archived anti-patterns (OM: Over-merge, DA: Distractor amplification, NME: Non-monotonic expansion) exposed by D 2 ACCI traces.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 447 papers

Sep 3, 2026

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.

Zeyu Liu, Jian Zhong, Rongduo Han +12

Nankai University · vivo AI Lab · Tsinghua University +1

LLM Memory LLM Agent Memory Long-Term Conversational Memory Memory Agent Benchmarks

Ask PDF

Jun 15, 2026

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change. We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings. Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.

Xianxuan Long, Zhikai Chen, Shenglai Zeng +3

Michigan State University · Case Western Reserve University

Temporal Reasoning in Language Models LLM Agent Memory Persistent Memory for Language Models AI Agent Benchmarks Memory Agent Benchmarks Language Modeling+2

Ask PDF

Oct 5, 2026

PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents

LLM agents rely on long-term memory to retain and reuse information when performing tasks over long horizons. Existing methods provide limited support for handling memories that become outdated as new observations or domain evidence arrive. Such outdated memories may remain semantically relevant, continue to affect dependent records, and retain value as historical evidence. This calls for two capabilities: dependency tracking to identify downstream effects and historical preservation to retain useful past records. We propose Provenance-Aware Cascading Memory Invalidation (PACMI), a framework that represents memories and new evidence in a provenance graph with typed dependency edges. PACMI assigns records to a four-state validity lattice, propagates validity changes to dependent memories, and uses the resulting states for retrieval and stale-premise detection. We also introduce a diagnostic benchmark with 100 cases and 300 queries across five domains. The evaluation separates node, context-, and answer-level performance. PACMI achieves the highest final-answer accuracy on this benchmark, and its paired difference from the strongest baseline is significant under an exact McNemar test. The premise checker achieves perfect precision, recall, and F 1 on the controlled query distribution. Cascading propagation primarily improves memorystate correctness: removing it increases final-answer errors from 3 to 11, but the paired difference does not reach the 0.05 significance threshold. Code and data will be made publicly available.

Yiqi Wang, Jiaqi Liu, Jiaqi Zhang +4

University of Southern Queensland · Southern University of Science and Technology · Jiangsu University +2

Data Provenance Memory Agent Benchmarks Graph-Based Agent Memory Agent Memory Management

Ask PDF

Abstract

Agent Memory Management

Explore similar work

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents