MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

Ryuichi Sumida; Koji Inoue; Tatsuya Kawahara

cs.CLAug 25, 2026

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

Authors: Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara

Organizations: Graduate School of Informatics, Kyoto University

Abstract

Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.

Figures & tables

Benchmark	Authoring	Users	Sess./u.	Turns/u.	Date span	Mem. % †	Nat. cued	Sat.
Synthetic / LLM-simulated dialogues
MemoryBank Zhong et al. (2024)	authors	15	10	37.7	10 days	17.7%	✗	✗
LoCoMo Maharana et al. (2024)	authors	20	27.2	294.1	$\sim$ 2 mo	24.0%	✗	✗
LongMemEval Wu et al. (2025)	experts	500	47.7	244.8	—	0.4%	✗	✗
Real human conversation
LUFY Sumida et al. (2025)	users	17	4	123.1	$\sim$ 9 days	23.8%	✗	✗

Table 1: Long-term memory benchmarks. MemUse is the only one combining multi-month real human-AI deployment, evaluation items detected from the conversation rather than authored as QA, and per-session satisfaction ratings. Column abbreviations: Sess./u. = sessions per user; Turns/u. = user turns per user; Mem. % † = % of user turns paired with an evaluation item; Nat. cued = evaluation items naturally cued by users (vs. externally authored); Sat. = per-session user satisfaction rating available. LoCoMo and RealTalk pair the same script across two speakers, so their 20 users represent 10 distinct dialogues.

Cond.	Tok. Gr.	vs Sum.	Mean Lat.	Mean TTFT
Summary	1.0 $\times$	1.0 $\times$	1.93	1.04
LC-10	2.0 $\times$	2.9 $\times$	2.00	1.12
LC-50	3.5 $\times$	11.9 $\times$	2.52	1.41
LC-100	5.8 $\times$	22.0 $\times$	3.27	1.79
RAG-10	1.8 $\times$	2.3 $\times$	2.03	1.05
RAG-50	1.0 $\times$	2.5 $\times$	2.03	1.09

Table 2: Per-condition cost and latency on the 29-user / 1,270-session subset. Tok. Gr. : within-condition input-token growth from month 1 to month 4. vs Sum. : mean session input tokens relative to the Summary baseline. Mean Lat. : per-session mean total response latency (s). Mean TTFT : per-session mean time-to-first-token (s).

Source sessions (prior context): In sessions 2–3, the system (“Luke”) recommended Bruno Mars (upbeat) and Ryuichi Sakamoto, Hikaru Utada, Yumi Matsutoya, and Haruka Nakamura (Japanese relaxing music).
Trigger (user turn, current session): “I’m thinking I’ll try to relax by listening to the music Luke recommended the other day.”
(i) Natural Integration (primary metric: does the system’s natural response demonstrate memory of the referenced topic?)
LC-100% response	“That sounds like a perfect way to unwind. I hope the music brings you some calm and comfort while your little one rests. Enjoy this peaceful moment—you deserve it!” ✗ generic response
Ideal response (authored)	“Glad you’re going to try the music I recommended! Ryuichi Sakamoto’s Merry Christmas Mr. Lawrence is my favorite—the solo piano eases you down.” ✓ integrates context
(ii) Direct QA (same model, same reconstructed context, each fact-seeking question asked literally; excerpt of 4)
Q1	Q: “What upbeat artist did the system recommend?” A: “Bruno Mars.” ✓

Table 3: One real MemUse instance scored under all three metrics (GPT-4.1-mini, LC-100% condition, same reconstructed context throughout). The same model that fails (i) Natural Integration and (iii) Reference answers the underlying questions correctly under (ii) Direct QA—the retrieval–integration gap of § 6.2 .

Figure 1: The retrieval–integration dissociation replicates across stronger models. Direct QA : per-question accuracy when each MemUse question is asked literally with the reconstructed context. Natural Integration : reader-level binary judgment of whether the conversational response demonstrates memory of the topic. Numerical values in Table 18 .

Mode & outcome	Excerpt (real deployment)
Reactive integration ✓ succeeds	User: “I finally went to Vienna with my coworker. We followed the plan we made together. …Just like AI suggested, it wasn’t crowded because it was a weekday. We were impressed by how accurate the plan was.” Luke: “…awesome your plan worked out so well and you could enjoy the views and that tasty schnitzel without the crowds. Figlmüller is a great pick—juicy veal with that crisp golden breading sounds mouthwatering.”
Reactive integration ✗ fails	User: “Do you remember what we were talking about earlier about Brittany?” Luke: “We haven’t talked about Brittany yet in our chats—but it sounds like a fascinating topic! What about Brittany caught your interest?”
Proactive recall ✓ welcomed	Luke: “…I hope your trip to Sapporo, Hakodate, and back to Sendai goes safely. It sounds like a busy day with driving and linguistic tasks—remember to take little breaks to refresh your mind!” User: “Oh, you remember our previous conversation! I am so glad that you remember my previous business trip.”
Proactive recall ✗ walked past	Luke: “Lovely to hear from you. How did your little superhero’s day go?” User: “Today I had a serious fight with my wife, and the whole situation has been sitting heavily in my chest ever since…”

Table 4: Real deployment excerpts illustrating the two modes. Reactive integration (top): the user invites memory; the system either weaves prior context into a natural response ( ✓ ) or hallucinates absence of the referenced topic ( ✗ ). Proactive recall (bottom): the system references a prior topic unprompted; the user either welcomes the callback ( ✓ ) or simply continues with their own agenda ( ✗ ).

Figure 2: Prompt-level interventions raise the level of Natural Integration but do not induce capacity-sensitivity (GPT-4.1-mini): every variant stays roughly flat across the 7 memory conditions. GPT-5.5 panel, per-condition numbers, and verbatim prompts in Appendix E.3 .

Figure 3: Asymmetric satisfaction landscape: only failure modes carry negative signal; reactive integration straddles zero. 95% CIs on

z

-rating; Appendix F.6 .

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The AI Diary chat interface (mobile view). The system references prior conversations in real time.

Block	Verbatim text
Persona (shared)	You are Luke, a friendly, kind, and understanding diary AI chatbot. Your goal is to make conversations feel like chatting with a friend. You are also a reflective companion who helps the user think about their day. - Ask questions only a third of the time. - Adapt your tone based on the user’s mood. - Aim for one or two sentences, like a real conversation. - You are also here to act as a personal diary.
Framing (shared)	This is the {N}th conversation. Summary of previous conversations: {summary}
Summary-only	System prompt : Persona $\oplus$ Framing $\oplus$ Generate the response using the conversation history. Messages : current-session turns only.
LC- $k\%$	System prompt : Persona $\oplus$ Framing $\oplus$ Generate the response using the following conversation history if necessary. Messages : {lc_history} (top $k\%$ user turns by importance, with their assistant pair-turns) $\oplus$ current-session turns.
RAG- $k\%$	System prompt : Persona $\oplus$ Framing $\oplus$ Generate the response using the following related memories if necessary: {mem_text} Messages : current-session turns only.

Appendix

Table 5: Verbatim generation prompts for the deployed conditions. {N} is the session index; {summary} is the previous-session summary (generation prompt in Appendix A.4 ); {lc_history} is the importance-filtered LC turn list; {mem_text} for RAG is the top-10 cosine-retrieved memories formatted as “ Memory from {date}: {text} ”. Generation: GPT-4.1-mini, temperature 0, max_completion_tokens

=\,500

. Offline reconstruction details in Appendix A.5 .

Stage	Users	Sessions
Enrolled	40	1,872
Valid conditions (0–6)	40	1,770
Rating $>0$	40	1,713
$-$ Adversarial users	38	1,635
$-$ Low-variance raters	29	1,270
Subsets

Appendix

Table 6: Data flow from enrollment to analysis. MemUse detection was run on all 1,872 enrolled sessions, regardless of rating status. Detection yielded 73 reactive instances, one of which (from annotator_40, flagged for pasting AI-generated content; see § 5 ) is excluded from the released benchmark, giving 72 benchmark instances. Excluding instances from the remaining 10 rating-filtered users yields 55 instances, distributed across 48 unique memory-moment sessions with rating-valid ratings (the within-instance analysis slice of § 6.1 ). Table 7 maps each reported analysis to its unit of aggregation.

Analysis	Section	Unit of analysis (one row per…)	$N$	What is averaged / computed per unit
Population / deployment level (all filtered sessions)
Existing-benchmark and matched-MemUse null vs. satisfaction (Table 13 )	§ 4	Session (filtered, rating $>$ 0)	1,270 sessions (29 users)	Session $z$ -rating paired with the condition-level mean of QA Acc, Direct QA, and matched-MemUse Natural Integration for that session’s condition.
Engagement predictors LMM (Appendix F.2 )	App. F.2	Session (filtered)	1,270 sessions	rating_z $\sim$ length + specificity + cross-session continuity + cond + (1 $\mid$ user); covariates are session-level.
Benchmark scoring (every instance re-run in every condition)
MemUse Natural Integration (Table 13 , column NI )	§ 5	Instance $\times$ condition response	$72\times 7=504$ responses	Binary Natural Integration judgment per response; condition means reported.
MemUse Direct QA (column Direct QA in Table 13 )	§ 5 , § 6.2	Instance $\times$ condition response	504 responses covering $316\times 7=2{,}212$ question judgments	Per-response question accuracy (fraction correct over that instance’s 3–5 questions); condition means reported.

Appendix

Table 7: What is averaged where. Each row names an analysis in the paper and its unit of aggregation, sample size, and aggregation rule. The benchmark comprises 72 instances and 316 questions; every instance is re-run in every condition, producing 504 Natural Integration judgments and 316 question judgments (columns NI , Direct QA in Table 13 ). The memory-moment subset restricts to each instance’s deployed condition only, yielding 55 filtered instances

\to

48 sessions

\to

207 questions. Session-level analyses collapse within-session instances by averaging question accuracy and OR-ing Natural Integration before correlating with the session’s rating.

Cond.	Nov	Dec	Jan	Feb
Summary	61	47	38	50
LC-10	48	45	45	48
LC-50	35	52	42	51
LC-100	30	52	45	64
RAG-10	38	43	56	38
RAG-50	30	54	50	41

Appendix

Table 8: Per-month condition counts (29 filtered users). Balance is maintained across time.

Event Type	Det.	Ver.	Prec.
User memory probe	12	11	91.7%
User re-provision	63	62	98.4%
System proactive recall	70	66	94.3%
User memory reaction	9	8	88.9%
Total	154	147	95.5%

Appendix

Table 9: Detection results. All events individually verified against raw conversation transcripts.

Mode	Judge inputs	Judge prompt (verbatim; system prompt: Answer ONLY ‘yes’ or ‘no’. )	A “yes” means
Natural Integration (Mode 1)	trigger quote, ground-truth facts, system’s natural response	User said: {trigger_quote} Expected: {ground_truth_facts} Assistant responded: {response} Does the response demonstrate memory of the referenced topic?	Judge makes a single overall yes/no decision that the response shows memory of the referenced topic. No rubric or worked examples are provided.
Direct QA (Mode 2)	system’s response, fact-seeking question, expected answer	Assistant’s response: {response} Question: {question} Expected answer: {expected_answer} Does the assistant’s response demonstrate knowledge of this fact?	The specific target fact (paraphrase accepted) is present in the response.

Appendix

Table 10: How each mode is judged. All prompts are used verbatim across all judge models and for human annotators. Natural Integration is the primary benchmark metric; Direct QA is a fact-anchored diagnostic (§ 6.2 ). Natural Integration scoring deliberately does not specify what “memory of the referenced topic” requires—by design, it measures whether integration appears to a reader rather than whether any particular token matches.

GPT-4.1-mini generation
Metric	Strict	Lenient
Natural Integration	25.4%	76.1%
Direct QA	7.6%	24.0%
GPT-5.4 generation
Natural Integration	49.5%	84.3%
Direct QA	11.8%	28.3%

Appendix

Table 11: Strict (GPT-5.4-nano) vs. lenient (GPT-5.4) judge on the same generated responses.

Pairwise Cohen’s $\kappa$
Rater	NI	Direct QA
LLM judge (GPT-5.4-nano)	51.8%	11.9%
Human Annotator 1	55.4%	13.6%
Human Annotator 2	51.8%	8.9%
LLM–Human 1	0.43	0.62
LLM–Human 2	0.21	0.65

Appendix

Table 12: Agreement between the production LLM judge (GPT-5.4-nano, used throughout the paper) and two human annotators. Natural Integration :

N=56

recalibration pilot; two newly recruited annotators. Direct QA :

N=316

original full validation; two original annotators. Both constructs reach substantial human–human agreement (

\kappa=0.57

, 95% bootstrap CI

[0.34,0.78]

N{=}56

for Natural Integration;

\kappa=0.70

for Direct QA). LLM–human

\kappa

on Natural Integration is asymmetric (LLM is closer to Annotator 1 than to Annotator 2), reflecting the residual interpretive latitude of a reader-level judgment. As a judge-stability check, re-running the Natural Integration pilot with GPT-5.5 ( gpt-5.5-2026-04-23 ) yields nearly identical labels (LLM–LLM

\kappa=0.89

, 94.6% agreement on the 56 items).

	Existing	Direct			Sat.
	QA Acc	Direct QA	Ref.	NI	(Z $\pm$ SE)
Cond.	(%)	(%)	(%)	(%)
Summary	19.7	44.9	7.9	23.6	$+$ .02 $\pm$ .07
LC-10	40.4	53.0	5.6	25.0	$+$ .05 $\pm$ .07
LC-50	65.9	72.4	7.2	25.0	$+$ .02 $\pm$ .07
LC-100	70.1	78.8	7.9	22.2	$-$ .04 $\pm$ .07

Appendix

Table 13: Numerical values for Figure . Existing QA Acc : average accuracy on existing benchmarks (LoCoMo, LUFY, RealTalk; per-turn QA; Appendix C.2 ). MemUse Direct QA : GPT-4.1-mini accuracy when each MemUse question is asked literally with the same reconstructed context—the apples-to-apples analogue of existing benchmarks on real memory moments. Ref. : MemUse Reference—whether each fact appears in the conversational response (question level). NI : MemUse Natural Integration—a reader-level binary judgment of whether the response demonstrates memory of the topic (instance level; the benchmark’s primary metric). Sat. : z-scored satisfaction

\pm

SE.

N\!=\!170

–

207

per condition. Per-session correlations: Existing QA Acc vs. Sat

\rho\!=\!.004

(

p\!=\!.86

); Natural Integration vs. Sat

\rho\!=\!{+}.008

(

p\!=\!.77

); within-user permutation

p\!=\!.91

; all

d\!<\!0.06

Condition	LUFY	LoCoMo	RealTalk	Avg.
Summary	34.0	15.3	9.8	19.7
LC-10	53.8	41.1	26.4	40.4
LC-50	83.1	63.4	51.2	65.9
LC-100	85.1	69.7	55.4	70.1
RAG-10	55.2	37.1	29.5	40.6
RAG-50	80.5	57.3	45.3	61.0

Appendix

Table 14: QA accuracy (%) on three existing benchmarks. All three show the same pattern: accuracy increases with memory capacity but none predicts satisfaction. The “QA Acc” column in Table 13 reports the average.

Sample	$N_{u}$	$N_{s}$	KW $p$	ME $\beta$	ME $p$
All 40 users	40	1,713	.93	$-$ .001	.71
No adversarial	38	1,635	.93	$-$ .001	.62
Original filter	29	1,270	.91	$-$ .001	.47

Appendix

Table 15: Null result across user subsets. KW: Kruskal-Wallis. ME: mixed-effects model (

z

-scored rating

\sim

QA accuracy + (1|user)).

Figure 6: Sensitivity analysis: condition effects on

z

-scored satisfaction across three user subsets. The null result is robust to all filtering decisions.

Figure 7: Effect sizes (Cohen’s

d

) for each condition vs. Summary-only with 95% CIs. All effects are negligible and span zero.

Rule	$\#\textsc{NI}{=}1$	$\rho$	$p_{\rho}$	$\beta_{\textsc{NI}}$	$p_{\beta}$
Any-success (default)	11/48	$+.289$	$.046$	$+.556$	$.082$
Majority-vote	10/48	$+.249$	$.088$	$+.543$	$.098$
All-success	10/48	$+.249$	$.088$	$+.543$	$.098$
Mean-NI (0–1)	—	$+.283$	$.051$	$+.563$	$.088$

Appendix

Table 16: Sensitivity of the successful-integration main effect to the session-level Natural Integration aggregation rule.

\rho

: Spearman Natural Integration vs. rating_z;

\beta_{\textsc{NI}}

: LMM coefficient from rating_z

\sim

Natural Integration + (1

\mid

user) fit with ML.

N{=}48

memory-moment sessions, 16 users. The point estimate is robust (

\beta_{\textsc{NI}}\in[+.543,+.563]

, a range of

0.02

); the Spearman

p

-value crosses the conventional

0.05

threshold depending on the rule, driven by the single multi-instance session described above (annotator_34 s32). The LMM

p

hovers in

[.082,.098]

across all rules and does not cross

0.05

. We report any-success as the default because it treats any in-session demonstration of memory integration as a positive outcome, consistent with the reader-level framing of the Natural Integration metric.

Figure 8: Cross-session continuity over time under three operationalizations. Left: Original (full history) shows strong increase (

\rho=.68

). Center: Fixed-window (last 10 sessions) reduces but does not eliminate the trend (

\rho=.27

). Right: Normalized by history size shows slight decrease (

\rho=-.14

), confirming partial mechanical confound. All three variants independently predict satisfaction (

p<.01

Figure 9: Per-user memory condition effect (memory

-

summary

z

-scored satisfaction). Only 11 of 29 users show a positive preference for memory conditions. Random slope variance = 0.042.

Failure Mode	N	%
Generic (ignores context)	28	38.9
Partial recall (gist only)	19	26.4
Hallucinated recall	17	23.6
Correct recall	6	8.3
Honest admission	2	2.8

Appendix

Table 17: LC-100% failure modes (GPT-4.1-mini,

N=72

). Despite having all prior turns available, 39% of responses are generic and 24% hallucinate recall.

Model	Cond.	Direct	Ref.	NI
		(%)	(%)	(%)
GPT-4.1-mini	Summary	44.9	7.9	23.6
	LC-10	53.0	5.6	25.0
	LC-50	72.4	7.2	25.0
	LC-100	78.8	7.9	22.2
	RAG-10	49.2	7.0	27.8

Appendix

Table 18: Cross-model MemUse accuracy across all 7 memory conditions,

N=73

instances each. Direct : MemUse Direct QA (per-question accuracy). Ref. : MemUse Reference (per question, against the natural response). NI : MemUse Natural Integration (holistic on the natural response, primary metric). GPT-4.1-mini values are from Table 13 ; GPT-5.5 ( gpt-5.5-2026-04-23 ) and Gemini 3.1 Pro ( gemini-3.1-pro-preview ) were re-run for this comparison with judge held fixed at GPT-5.4-nano. Direct QA spans 32.9–34.4 points across capacity for every model; NI spans at most 8.2 points within model and never tracks capacity.

Memory system	Direct QA	Ref.	NI
Summary/LC/RAG (7 cond., range)	44.9–78.8	5.6–8.0	22.2–27.8
Mem0	41.5	7.3	58.3
Letta (MemGPT)	61.4	10.8	56.9

Appendix

Table 19: Modern memory systems on MemUse (72 instances, 316 questions; GPT-4.1-mini generator; judge fixed at GPT-5.4-nano; all values %). Both raise Natural Integration (NI) well above the provisioning baselines, yet the Direct QA vs. Reference gap persists (§ 6.2 ).

Figure 10: Prompt interventions do not induce capacity-sensitivity in the integration channel. Top: Natural Integration (%) by memory condition for V0 (deployed) and four interventions, with Direct QA on the same axis as a retrieval baseline. Bottom: span across the 7 conditions (max

-

min, pts). Direct QA spans 32.9–33.9 pts in both models; every integration variant spans at most 21.9 pts, and the V0 deployed pipeline spans only 6.8–11.0 pts. Variants: V0 deployed pipeline (no extra prompt); V2-CoT chain-of-thought reasoning prepended; V5 one-line cue-aware patch; V6 two-step extract-then-integrate (a separate call extracts named details from context, then the deployed pipeline generates with those details handed in); V7 query rewrite + targeted RAG (the user trigger is rewritten into explicit search queries before retrieval). Per-condition numbers in § E.3.4 .

Variant	GPT-4.1-mini	GPT-5.5
V0 (deployed)	6.8	13.7
V1	9.6	8.2
V2-CoT	20.5	15.1
V3	9.6	13.7
V4	19.2	11.0
V5	9.6	21.9

Appendix

Table 20: Natural Integration-span diagnostic (capacity insensitivity). Each cell is max

-

min Natural Integration (%) across the 7 memory conditions for the indicated variant on the indicated model. † Same-model MemUse Direct QA capacity span across the 7 conditions, reported in § 6.2 : 32.9 / 33.9 / 34.4 pts for GPT-5.5 / GPT-4.1-mini / Gemini 3.1 Pro. Across every variant on either model, NI span remains

\leq 22

pts —well below the

\sim

33-pt Direct QA span—confirming that prompt-level interventions do not make the integration channel capacity-sensitive in the way the retrieval channel is. Full per-cell numbers (V0/variant NI,

\Delta

, McNemar

p

) are released with the analysis code.

Condition	V0	V5	$\Delta$	McNemar $p$
Summary-only	24.7	41.1	$+16.4$	.004
LC-10%	20.5	31.5	$+11.0$	.096
LC-50%	26.0	39.7	$+13.7$	.053
LC-100%	21.9	32.9	$+11.0$	.039
RAG-10%	24.7	31.5	$+6.8$	.302
RAG-50%	21.9	35.6	$+13.7$	.031

Appendix

Table 21: V5 cue-aware prompt vs. V0 deployed persona on Natural Integration % (GPT-4.1-mini,

N=73

instances per cell). Variant

\times

condition interaction

\chi^{2}=2.5

df=6

p=.86

(homogeneity of discordant-pair distribution). Reference rates barely move (

+0

+5

pts).

Cond.	V0 NI	V6 NI	$\Delta$ NI	V6 Ref.	McN. $p$
Summary-only	24.7	24.7	$+0.0$	10.0	1.00
LC-10%	20.5	23.3	$+2.7$	6.5	.77
LC-50%	26.0	28.8	$+2.7$	9.0	.82
LC-100%	21.9	27.4	$+5.5$	8.1	.29
RAG-10%	26.0	26.0	$+0.0$	7.5	1.00
RAG-50%	21.9	31.5	$+9.6$	8.7	.09

Appendix

Table 22: V6 two-step pipeline vs. V0 deployed persona (GPT-4.1-mini,

N=73

). Natural Integration and Reference in %. V6 is comparable to or worse than V2-CoT (

34.2\%

on LC-100%; Appendix E.3.4 ).

Step-1 found GT?	Step-2 referenced?	Count
Yes	Yes	11
Yes	No	37
No	Yes	9
No	No	16
No_relevant_context	—	2

Appendix

Table 23: V6 extraction-quality decomposition on LC-100% (

N=73

). Step-1 mean GT-coverage

=0.51

. When Step-1 successfully named the ground-truth facts, Step-2 still failed to integrate them in 37/48 = 77% of cases.

	NI (%)		Recall (%)
Cond.	V0	V7	V7	V0
Summary-only	24.7	26.0	8.5	9.0
LC-10%	26.0	24.7	29.9	32.7
LC-50%	26.0	24.7	70.1	75.4
LC-100%	23.3	24.7	67.3	78.7
RAG-10%	27.4	31.5	29.9	32.7

Appendix

Table 24: V7 query-rewrite RAG vs. V0 (GPT-4.1-mini,

N=73

). Natural Integration (NI) and retrieval recall (token-overlap

@0.5

between retrieved passages and ground-truth fact spans), both in %. V7 lifts NI modestly on RAG-50/100% (

+8.2/+6.8

pts; McNemar n.s.) but rewritten queries actually retrieve worse than raw user triggers at high

k\%

(LC/RAG-100% recall

67.3

vs.

78.7

Cond.	V0 NI	V2-CoT NI	$\Delta$	McN. $p$
Summary-only	54.8	71.2	$+16.4$	.023
LC-10%	47.9	64.4	$+16.4$	.023
LC-50%	61.6	71.2	$+9.6$	.19
LC-100%	47.9	72.6	$+24.7$	.0005
RAG-10%	50.7	67.1	$+16.4$	.023
RAG-50%	61.6	79.5	$+17.8$	.011

Appendix

Table 25: V2-CoT on GPT-5.5 across all 7 conditions (

N=73

). On LC-100%,

+24.7

pt lift closes

\sim

70% of the V0

\to

Direct QA gap (47.9

\to

72.6 vs. 83.0 ceiling), leaving

+10.3

pts of headroom. Variant

\times

condition interaction

\chi^{2}=2.2

df=6

p=.90

—lift is uniform across conditions including Summary-only where no facts are in context, replicating the cue-aware Summary-fragility pattern (Appendix E.3.8 ). Reference remains low (LC-100%

24.6\%

) confirming the dissociation between holistic Natural Integration and item-level Reference seen on GPT-4.1-mini.

Variant (LC-100%)	Overall	Probe	Re-prov
V0 baseline	23.6	40.0	21.0
V1 (light)	27.8	40.0	25.8
V2-CoT	33.3	70.0	27.4
V3 (failure-mode)	30.6	70.0	24.2
V4 (few-shot)	26.4	70.0	19.4
V5 (cue-aware)	31.9	70.0	25.8

Appendix

Table 26: Natural Integration % on LC-100%, split by event type (

n_{\text{probe}}=11

n_{\text{re-prov}}=62

; full 73-instance detection set, see Appendix E.1 ). Reasoning/framing prompts (V2-CoT, V3, V4, V5) lift probe Natural Integration by

\sim+30

pts but re-provision by

\leq+8

pts. The architectural V6 two-step shows the inverse pattern (mean across 7 conditions:

-5.7

pts on probes,

+4.4

pts on re-provisions). Probe

n=11

implies

\pm

25–30 pp 95% CIs per single cell; the pattern’s reliability comes from directional consistency across multiple variants and 7 conditions.

Figure 11: Predictors of

z

-scored satisfaction (

N=1{,}270

sessions, 29 users). Bars show 95% Wald CIs around point estimates. Three engagement metrics each independently predict higher satisfaction; memory condition does not.

Figure 12: Leave-one-user-out cross-validation of the memory

\times

engagement interaction. Each bar shows the interaction coefficient when one user is withheld. The effect is positive in all 16 iterations and significant (

p<.05

) in 15 of 16 (green). The vertical line marks the full-model estimate (

\beta=0.57

Figure 13: Cook’s distance for the memory

\times

engagement model (

N=53

). Four observations exceed the

4/N

threshold (dashed line). After removing all four, the interaction remains significant (

\beta=0.35

p=.018

Figure 14: Satisfaction by memory event type (session-level aggregate). Reactive events (user-initiated probes and re-provisions) are associated with higher satisfaction than proactive system recalls (

\bar{z}=+0.09

vs.

-0.14

p=.086

on sessions; § 6.3 refines this by scoring the recalls themselves).

	Grounding			Appropriateness
Score	0	1	2	0	1	2
$N$	1	20	43	8	25	31
$\bar{z}$	$+0.09$	$-0.14$	$-0.08$	$-0.48$	$+0.03$	$-0.10$

Appendix

Table 27: Mean

z

-scored satisfaction by Proactive- MemUse score bin (64 events, 24 users, 63 unique sessions with

\textsc{talking\_index}\leq 6

). The only pronounced effect is the low-appropriateness bin; better scores on either axis do not translate into higher satisfaction.

Condition	$\boldsymbol{N}$	Ground.	Approp.	$\boldsymbol{\bar{z}}$
Summary-only	3	1.67	1.33	$+0.14$
LC-10%	21	1.48	1.14	$-0.31$
LC-50%	17	1.76	1.24	$+0.12$
LC-100%	9	1.78	1.33	$-0.18$
RAG-10%	4	1.75	2.00	$+0.20$
RAG-50%	6	1.67	1.67	$-0.02$

Appendix

Table 28: Proactive- MemUse scores and mean

z

-scored satisfaction by deployed memory condition (event-level,

N{=}64

). Both axes are stable across conditions; RAG conditions achieve uniformly high appropriateness (mean

\approx 2.0

on RAG-10/100) without a corresponding satisfaction gain.

Conversational Memory

Conversational memory equips AI assistants to retain, update, and retrieve useful information from past interactions for coherent, personalized dialogue.

See all 72 papers

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents
Sep 3, 2026Wen-Yu Chang, Yun-Nung ChenConversational Memory LLM Agent Memory
National Taiwan University, Taipei, Taiwan
RUMBA: Russian User Memory Benchmark
Jul 23, 2026Elizaveta Shevtsova, Inna Glebkina, Mark Baushenko +2Temporal Reasoning in Language Models Temporal Reasoning
1DAIMLD
UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory
Aug 31, 2026Peijun Qing, Fobo Shi, Soroush VosoughiLLM Agent Memory Memory Agent Benchmarks
Dartmouth College · Wuhan University

Sep 3, 2026J/K move · Enter open · S save

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

Wen-Yu Chang, Yun-Nung Chen

National Taiwan University, Taipei, Taiwan

Conversational Memory LLM Agent Memory Memory Agent Benchmarks Long-Term Conversational Memory

Open paper AskPDF arXiv

Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.

Abstract

Conversational Memory

Explore similar work

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

RUMBA: Russian User Memory Benchmark

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory