cs.CLAug 20, 2026

Remember, Verify, or Ask? Evaluating Memory Decisions at Reuse Time

Authors: Baichuan Li, Junyi Yao, Zihao Zheng

Organizations: Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, USA · Department of Computer Science & Engineering, Washington University in St. Louis, St. Louis, USA

Abstract

Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification boundary: whether interaction-derived information should be persisted, used only in the current context, re-verified, or clarified with the user. MCB contains 140 primary scenarios, split into 70 development and 70 held-out items, plus a separate 70-item contrast set. It evaluates both action labels and structured tool-call selection. Two non-authors independently label the 70 held-out primary and 70 contrast items (97.1% agreement, Cohen's kappa = 0.962); a blind third resolves four disagreements, replacing eight author labels by non-author majority. Across Claude and Qwen, models verify changing facts more reliably than they ask users to resolve ambiguity. Bare Qwen asks on 0/12 clarification items while verifying 12/18 freshness items. Few-shot prompting raises accuracy from 0.557 to 0.771 (paired delta = +0.214, Holm-adjusted exact McNemar p_H = 0.002), yet clarification recall remains 0.333. The policy prompt reduces erroneous persistence from 0.243 to 0.100 (p_H = 0.038), although its accuracy gain is not significant. Label-tool agreement is 57% for each Claude model and 23% for Qwen; Qwen accuracy falls from 0.557 to 0.343 (p_H = 0.047). Memory evaluation must test both stated decisions and tool-call choices.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

    May 7, 2026Hanxiang Chao, Yihan Bai, Rui Sheng +2Agent MemoryLLM Agent Evaluation

  2. PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents

    Oct 5, 2026Yiqi Wang, Jiaqi Liu, Jiaqi Zhang +4Data ProvenanceMemory Agent Benchmarks

  3. When Personal Memory Has No Single Answer: Evaluating LLM Agents under Irreducible Conflict

    Aug 14, 2026Lu Yang, Shusheng Xu, Zhuoran Li +2LLM Agent MemoryKnowledge Conflicts in Language Models