ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment

Shuning Zhang; Tenghao Ji; Jingruo Chen; Eryue Xu; Bohao Chu; Xin Yi

cs.AIOct 24, 2026

ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment

Authors: Shuning Zhang, Tenghao Ji, Jingruo Chen, Eryue Xu, Bohao Chu, Xin Yi

Organizations: Tsinghua University · Independent Researcher · Cornell University · University of Illinois Urbana-Champaign ·

Abstract

Long-term memory is critical for autonomous Large Language Model (LLM) agents executing multi-session, long-horizon tasks. However, existing memories often inappropriately store, retain, or retrieve over-generalized, misattributed, or privacy-sensitive information, a problem we term memory misalignment. To study this problem, we construct MemAlign, a benchmark comprising 1,079 curated memory cases grounded in synthetic, validated multi-session conversational contexts covering diverse misalignment taxonomies. We then propose ConMem, an algorithm that reframes retention decisions as prospective counterfactual causal inference. ConMem synthesizes counterfactual interaction scenarios to prospectively audit downstream intervention effects across three aspects: relevant task execution, irrelevant invariance checks, and risk-sensitive boundary shifts. Experiments show that ConMem outperforms other methods, achieving 75.16% accuracy, 64.15% balanced accuracy, and a Macro-F1 of 65.31%, effectively mitigating over-retention biases without sacrificing personalization utility.

Figures & tables

Figure 1: An illustration of memory misalignment.

Figure 2: Distribution of (a) storage decisions and (b) reason types in MemAlign (

N=1{,}079

). Labels indicate instance counts and percentages.

Figure 3: Overview of ConMem. Given a historical context

C

and a candidate memory

M

, ConMem (1) constructs and validates three future probes (relevant, irrelevant, and risk-sensitive); (2) generates a control response

R_{i}^{0}

without

M

and a treatment response

R_{i}^{1}

with

M

for each probe

q_{i}

; (3) compares the two responses with a blind quality judge and a memory-attribution auditor; and (4) aggregates the signed probe contributions

c_{i}

into

\mathrm{CMTE}(M)

to decide Store or Do-Not-Store , invoking a tie-break judge when the score is zero.

C

is withheld from response generation and used only for evaluation.

Method	Acc.	BAcc.	Macro-F1	Store R.	DNS R.
MemoryBank	67.19	67.95	64.30	66.24	69.67
A-MEM	67.38	50.25	48.16	88.83	11.67
A-MAC	69.33	51.91	49.91	91.12	12.71
SAGE	40.96	46.41	40.55	34.15	58.67
ConMem	75.16	64.15	65.31	88.96	39.33

Table 1: Performance comparison across techniques.

Reason type	$N$	Recall (%)
context_misapplication	31	12.90
incomplete_memory_use	18	16.67
granularity_mismatch	33	21.21
identity_mis_inference	11	27.27
privacy_overuse	11	27.27
attribution_confusion	15	33.33

Table 2: ConMem performance across rejection categories.

Variant	Acc.	BAcc.	Macro-F1
Full ConMem	75.16	64.15	65.31
w/o Quality Judge	72.57	59.28	59.87
w/o Attribution Auditor	55.05	55.55	52.41
Zero $\rightarrow$ STORE	71.73	56.85	56.83
Zero $\rightarrow$ DO_NOT_STORE	66.82	60.93	60.38

Table 3: Ablation results for ConMem’s validation and decision components (%).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Calls	Mean	Median	P95
Probe generation	1,129	26.76	25.19	41.63
Probe audit	1,139	40.94	31.93	104.52
Probe repair	15	17.00	15.96	23.49
Control response	3,237	21.14	15.63	54.46
Treatment response	3,237	21.72	16.88	53.95
Quality judge	3,237	20.31	18.72	36.87

Appendix

Table 6: Latency statistics of ConMem on the primary benchmark. All latency values are in seconds. “Per-case” denotes the sum of recorded API latencies required to evaluate one candidate memory.

Metric	GPT-5.6	DeepSeek-V4-Flash
Accuracy	77.85	76.83
Balanced Acc.	70.93	67.76
Macro-F1	71.54	68.89
Store Precision	83.42	81.30
Store Recall	86.52	88.19
Store F1	84.94	84.61

Appendix

Table 7: ConMem admission performance across LLM backbones. All values are percentages except counts.

Metric	GPT-5.6	DeepSeek-V4-Flash
Accuracy	77.85–79.43	76.65–77.94
Balanced Acc.	70.93–71.92	67.63–68.33
Macro-F1	71.54–72.98	68.72–69.77
Store Recall	86.52–88.83	87.93–89.99
DNS Recall	54.67–55.67	46.67–47.33
Zero-score Rate	42.45–45.41	17.79–20.67

Appendix

Table 8: Performance ranges across 27 CMTE scoring settings. All values are percentages.

Memory Management for LLM Agents

See all 92 papers

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Agent Memory Persistent Memory for Language Models
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent
Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents
May 17, 2026Saksham Sahai SrivastavaConversational Memory Long-Term Conversational Memory
School of Computing, University of Georgia, Athens, Georgia, USA.
MemRefine: LLM-Guided Compression for Long-Term Agent Memory
Jun 11, 2026Minjae Kim, Jinheon Baek, Soyeong Jeong +1Persistent Memory for Language Models LLM Context Compression
Korea University · KAIST

Jun 19, 2026J/K move · Enter open · S save

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management Personal AI Agents Memory Management for LLM Agents

Open paper AskPDF arXiv

Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user's current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user's memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available: https://github.com/galaxyChen/AdaMem

Abstract

Memory Management for LLM Agents

Explore similar work

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents

MemRefine: LLM-Guided Compression for Long-Term Agent Memory