D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory

Xule Liu; Yijun Liu; Chao Li; Shao Kun

cs.AIAug 18, 2026

D $^2$ ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory

Authors: Xule Liu, Yijun Liu, Chao Li, Shao Kun

Organizations: Xiaomi Inc.

Abstract

Memory is a key capability of LLM agents. Persistent memory extends this across sessions---enabling recall, revision, and personalization. Yet its multi-stage pipeline (ingestion, retrieval, filtering, generation) makes failures difficult to localize: end-to-end evaluation reveals that an error occurred, but not which stage caused it. Existing evaluations often report aggregate performance without paired statistical comparisons, slice-level non-regression checks, or stage-level diagnostic traces. We propose D $^2$ ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol whose outer diagnostic gate promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability. We further introduce DCR, a graded observability metric that measures whether failures remain localizable, and D $^2$ ACCI-Eval, a reusable artifact for gate replay. We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Five paired ablations show that supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains (+1.9 to +3.7pp, all p $\le$ .003). In contrast, BM25/RRF is retained as a monitored feature flag---a distinction invisible to aggregate-only evaluation. A diagnostic audit shows enriched traces substantially improve root-cause agreement over result-only relabeling. Diagnostic artifacts reach 98--100% DCR@3 versus 0% for results-only logs. These results establish that robust memory-system iteration demands traceable, statistically grounded, and regression-aware evidence---exactly the gap D $^2$ ACCI fills.

Figures & tables

Figure 1: (a) Current end-to-end evaluation reports only aggregate accuracy; internal pipeline states are hidden and failures cannot be attributed to a specific stage. (b) D 2 ACCI emits per-stage traces, enabling paired gates to localize failures and drive targeted repairs.

Figure 2: MemStack under the D 2 ACCI dual-loop protocol. Top : the inner memory loop processes queries through feature-flagged stages (FF; independently togglable by the outer gate): ingestion, multi-layer storage, retrieval, constraint filtering, context assembly, and generation; each emits a typed diagnostic trace (IDs, layers, ranks, filters, budget, judge). Ablation targets (✗) mark paired ablations in Table 3 . Bottom : the outer diagnostic loop feeds traces into D 2 ACCI -Eval (paired stats, slice deltas, DCR , discordance, audit), producing a gate decision that updates inner-loop feature flags for iteration $t{+}1$ . The iteration timeline shows automatic gate replay including the $t_{3}$ rejection.

Benchmark	Eval set	Ours	MemBrain	$\Delta$	Representative slices (category / ( count , accuracy ))
LoCoMo	1540 non-adv QA	93.59%	93.25%	+0.34pp	Single-hop	Multi-hop	Temporal	Open
LoCoMo	1540 non-adv QA	93.59%	93.25%	+0.34pp	( 841 , 96.2 )	( 282 , 89.5 )	( 321 , 92.4 )	( 96 , 86.8 )
LongMemEval	500/500 questions	90.93%	85.60%	+5.33pp	Single-user	Knowledge-update	Multi-session	Temporal
LongMemEval	500/500 questions	90.93%	85.60%	+5.33pp	( 70 , 99.5 )	( 78 , 93.6 )	( 133 , 85.0 )	( 133 , 89.0 )
PersonaMem-V2	200 personas, 5000 MCQ	57.20%	55.72%	+1.48pp	Sensitive	Forget	Neutral	Therapy
PersonaMem-V2	200 personas, 5000 MCQ	57.20%	55.72%	+1.48pp	( 511 , 86.3 )	( 1048 , 66.9 )	( 858 , 51.2 )	( 627 , 44.5 )

Table 1: Main results and representative slice breakdowns. External rows are reference points, not fully normalized same-stack baselines. Slice-level reporting is required because memory interventions are frequently non-monotonic.

Condition vs. Human A	Fine $\kappa$	Coarse $\kappa$
Enriched GPT-4o trace relabel	0.571	0.619
Result-only GPT-4o relabel	0.258	0.272
Human B	0.674	0.711

Table 2: Audit-label contrast on the 60-case packet. Human B is an independent annotator; all rows are pre-adjudication.

Benchmark	Comparison	$n$	Ablated	$\Delta$	Paired evidence			Gate decision
Benchmark	Comparison	$n$	Ablated	$\Delta$	$p$	95% CI	Full wins / Ablated wins	Metric-only	D 2 ACCI gate
LoCoMo	No BM25/RRF	1,540	94.00%	$-0.41$ pp	.4426	[ $-1.36$ , 0.56]	27/34	Demote	Feature-flag
LoCoMo	No Supplement	1,540	90.89%	$+2.71$ pp	.0009	[1.26, 4.20]	90/50	Promote	Accept; MH monitor
LME	No BM25/RRF	500	89.93%	$+1.00$ pp	.4583	[ $-1.13$ , 3.13]	17/12	Promote	Feature-flag
LME	No Session	500	87.27%	$+3.67$ pp	.0026	[1.33, 6.00]	28/9	Promote	Accept; KU monitor
PMem	No Forget Guard	5,000	55.28%	$+1.92$ pp	.0030	[0.64, 3.18]	560/464	Promote	Accept; slice monitor

Table 3: Completed paired diagnostic ablations with metric-only vs. D 2 ACCI gate decisions. CI is bootstrap 95% for Full

-

Ablated; Full wins / Ablated wins count correct examples. Metric-only: accept if

\Delta{>}0

, demote if

\Delta{<}0

; D 2 ACCI adds statistical and slice-level checks.

Figure 3: LoCoMo iteration trajectory with automatic gate-replay decisions. Each transition is produced by the replay script (

<

1 s per candidate); all five decisions match the human-in-the-loop outcome. The dashed red segment marks

t_{3}

: traces localized over-merging, triggering automatic rejection.

AP ∗	Trigger	Observed Behavior	Resolution
OM	Threshold $(0.75)$	Facts collapse into a single memory representation	Reverted
DA	Option-aware filter	Retrieval surfaces distractor evidence	Rejected
NME	Gated expansion	Recovers some targets, introduces off-target noise ( $57.34\%\to 57.24\%$ )	Slice-gated
∗ AP: Anti-pattern

Table 4: Representative archived anti-patterns (OM: Over-merge, DA: Distractor amplification, NME: Non-monotonic expansion) exposed by D 2 ACCI traces.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 447 papers

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems
Sep 3, 2026Zeyu Liu, Jian Zhong, Rongduo Han +12LLM Memory LLM Agent Memory
Nankai University · vivo AI Lab · Tsinghua University +1
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University
PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents
Oct 5, 2026Yiqi Wang, Jiaqi Liu, Jiaqi Zhang +4Data Provenance Memory Agent Benchmarks
University of Southern Queensland · Southern University of Science and Technology · Jiangsu University +2

Sep 3, 2026J/K move · Enter open · S save

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu+11

Nankai University · vivo AI Lab · Tsinghua University+1

LLM Memory LLM Agent Memory Long-Term Conversational Memory Memory Agent Benchmarks

Open paper AskPDF arXiv

Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.

Abstract

Agent Memory Management

Explore similar work

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents