Madeleine: Learning Involuntary Recall for Conversational Memory from Simulated Lives

Zhiyun Shi

cs.CLOct 1, 2026

Madeleine: Learning Involuntary Recall for Conversational Memory from Simulated Lives

Authors: Zhiyun Shi

Organizations: Nanyang Technological University, Singapore

Abstract

A long-term conversational assistant must recall the right memory at the right moment, yet the memory that matters most is often not similar to what the user says now. Current systems recover such associations by letting an LLM reason at write or read time, at a cost of hundreds to over a thousand LLM calls per memory bank and up to several thousand context tokens per query. We argue that association is a learnable relevance: the pointwise mutual information of memories under how human lives unfold. We introduce Madeleine, which learns amortized association: offline, an LLM life simulator writes simulated lives, whose cue-trigger pairs teach a query encoder a residual association on top of frozen similarity; online, it calls no LLM and plugs into any vector memory by replacing only the query encoder. On LoCoMo-Plus under the official protocol, Madeleine (I) reaches 66.6 when plugged into HyperMem, the highest among all systems evaluated under this protocol; (II) used alone, reaches the score of HyperMem as released (52.4 vs. 52.9) with zero LLM calls and about 1/21 of its answer context; and (III) lifts T-Mem by 26.2 points, significantly outperforms the same untrained backbone inside both systems, and leaves ordinary QA intact on the 4B backbone.

Figures & tables

Panel of Figure 1 — Figure 1: ( Left ) Associative recall: the trigger shares no content with its cue. A similarity retriever returns memories that share surface words with the trigger, whereas Madeleine recalls the cue without calling an LLM. ( Right ) LoCoMo-Plus score (official protocol) against LLM calls at write and read time (log scale). LLM-reasoning (System 2) memory systems lie to the right; standalone Madeleine reaches the score of HyperMem as released with zero LLM calls, and plugging it into HyperMem and T-Mem (arrows) raises both scores at unchanged cost.

Figure 2: Overview of Madeleine . ( Left, offline / System 2 ) An LLM life simulator writes simulated lives from three sources; each earlier statement becomes a cue and each later utterance a trigger that does not hint at it. The decontaminated pairs train a query-side LoRA adapter with InfoNCE over same-source facts. ( Right, online / System 1 ) Memories are stored verbatim and embedded once by the frozen encoder $E$ ; the trigger is encoded by $E$ and by the adapted $E_{\theta}$ , and the similarity and association scores are summed. A single query vector is the plug-in interface to a host system such as HyperMem.

Group	Method	Write	Read	Tokens	Causal	State	Goal	Value	All
	gemini-embedding	0	0	387	55.4	52.0	26.0	33.0	41.6
	text-embedding-3-large	0	0	396	58.4	45.0	30.0	31.0	41.1
	Qwen3-Embedding-4B	0	0	378	54.5	37.0	29.0	38.0	39.7
	Qwen3-Embedding-8B	0	0	386	50.5	36.0	28.0	35.0	37.4
(a) Embedding models	bge-m3	0	0	385	31.7	28.0	9.0	19.0	21.9
(b) Full context	Full dialogue	0	0	18,776	49.5	49.0	41.0	40.0	44.9

Table 1: Main results on LoCoMo-Plus (401 cognitive questions, official protocol: gpt-4.1-mini reader, gemini-2.5-flash judge); Causal – Value : the four relation types, All : overall. Write : LLM calls per memory bank; Read : LLM calls per query, excluding the answer call; Tokens : answer-input tokens per query. Retrievers pass

K{=}3

turns to the reader. Standalone Madeleine is the mean of three seeds; plug-in rows replace the host system’s retrieval encoder with Madeleine (seed 0), re-encoding T-Mem’s memories with the frozen backbone.

\uparrow

: gain over the same system without Madeleine (standalone: over the untrained backbone). Best per column in bold , second underlined . T-Mem and JustMem report 74.81 and 62.59 under their own, non-comparable protocols (Appendices C , E ).

Figure 3: Score versus answer-input tokens per query on LoCoMo-Plus (official protocol). The HyperMem curve shrinks its episode and fact budgets in its default 20:30 ratio; Madeleine is standalone (three-seed means,

K{=}3/5/10

System	Original	Untrained	Madeleine
Standalone	–	39.7	52.4 ${}^{**}_{{\color[rgb]{0.8516,0.2813,0.1094}\uparrow 12.7}}$
T-Mem, 3 scenes	33.7	53.9	59.9 ${}^{*}_{{\color[rgb]{0.8516,0.2813,0.1094}\uparrow 6.0}}$
HyperMem	52.9	60.6	66.6 ${}^{*}_{{\color[rgb]{0.8516,0.2813,0.1094}\uparrow 6.0}}$

Table 2: Same-backbone control on LoCoMo-Plus (official protocol). Within each row only the retrieval encoder changes: the system’s original one (bge-m3 for T-Mem, Qwen3-Embedding-4B without its query instruction for HyperMem), the untrained Qwen3-Embedding-4B with the instruction, or Madeleine (seed 0) on that backbone; memories are encoded by the frozen encoder of each column.

\uparrow

: gain over the untrained encoder;

{}^{*}p<0.05

{}^{**}p<10^{-4}

(exact McNemar). All gains over the original encoder:

p<10^{-6}

Figure 4: LoCoMo-Plus by relation type (

\approx

100 items each): (a) cues missing from the top 10; (b) end-to-end score (

K=3

, three-seed mean for Madeleine ).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Generator	Calls	Cost ($)	Statements	Pair cos	Rows kept (removed)
v2 (static facts)	gpt-4.1-mini	800	1.61	8,000	0.422	31,697 ( $\approx$ 1%)
v3 (trajectories)	DeepSeek-V4.1-Flash	800	0.79	7,975	0.398	32,419 (49)
v2d (v2 prompt)	DeepSeek-V4.1-Flash	800	0.73	8,136	–	31,697 (1,092)

Appendix

Table 3: The three life-simulator sources. Pair cos is the mean cosine (untrained Qwen3-Embedding-4B) between an earlier statement and its later messages on a 150-pair pilot; lower means less similar. Rows are training pairs (later message

\to

earlier statement, plus one direct question per statement) after decontamination; they include every later message the simulator returned, and some statements received more than three. v2 and v2d happen to keep the same number of rows.

Item	Value
Backbone	Qwen3-Embedding-4B (also 0.6B, 8B) ( Zhang et al., 2025 )
Adapter	LoRA ( Hu et al., 2022 ) , $r=32$ , $\alpha=64$ , dropout 0.05
LoRA targets	q, k, v, o, gate, up, down projections
Trainable parameters (4B)	66.1M
Loss	InfoNCE on the summed score, $\tau=0.05$
Negatives	all same-source training facts

Appendix

Table 4: Hyperparameters of the query-side association adapter (identical for all backbones and seeds unless stated).

Retriever	$K$ =3	$K$ =5	$K$ =10
Madeleine (4B), seed 0	52.4	53.1	54.1
Madeleine (4B), seed 1	51.6	53.9	51.9
Madeleine (4B), seed 2	53.1	54.4	53.9
Madeleine (4B), mean	52.4	53.8	53.3
Untrained Qwen3-Embedding-4B	39.7	47.4	49.9
gemini-embedding	41.6	45.9	46.4

Appendix

Table 5: LoCoMo-Plus under the official protocol (gpt-4.1-mini reader unless stated) for

K=3,5,10

(official gemini-2.5-flash judge). FullText has no

K

System	Query encoder	Score
HyperMem	original	52.9
HyperMem	untrained 4B (with instruction)	60.6
HyperMem	Madeleine	66.6
T-Mem, 3 scenes	original (bge-m3)	33.7
T-Mem, 3 scenes	untrained 4B	53.9
T-Mem, 3 scenes	Madeleine	59.9

Appendix

Table 6: Plug-in results under the official protocol, including T-Mem at its default of 10 scenes. Only the retrieval encoder differs within each system (T-Mem’s memories are re-encoded by the frozen encoder); Madeleine is seed 0.

Retriever in T-Mem	All	Causal	Goal	State	Value	Cue@10	3 scenes (Cue@3)
bge-m3 (original)	51.9	64.4	44.0	59.0	40.0	70.1	41.4 (40.4)
Untrained Qwen3-Embedding-4B	67.6	78.2	62.0	67.0	63.0	89.3	64.1 (72.6)
Madeleine	70.6	78.2	67.0	72.0	65.0	94.5	72.6 ( 84.3 )

Appendix

Table 7: T-Mem under its own released code and evaluation protocol (401 LoCoMo-Plus items, GPT-4o reader, T-Mem’s answer and judge prompts). Only the embedding used by its retrieval stage changes; the memory bases are shared by all arms. Cue@k : the scene containing the cue is among the

k

retrieved scenes.

Benchmark	Metric	Untrained 4B	Madeleine	text-emb-3-large
LoCoMo-Conv (1,540)	Recall@10 (paper metric)	46.2	48.1	49.6
LoCoMo-Conv	answer score, $K$ =10	0.440	0.456	0.453
LoCoMo-Conv, user turns only	answer score, $K$ =10	0.456	0.482	0.489
InMind single fact (125)	indirect query	5.6	27.7	0.8
InMind single fact	direct query	96.8	93.1	80.8

Appendix

Table 8: Other benchmarks. Retrieval rows are Recall@10 (%); end-to-end rows use each benchmark’s own protocol. Madeleine is the three-seed mean where three seeds exist and seed 0 otherwise.

Retriever	Causal	State	Goal	Value	All
Untrained 0.6B	50.5	41.0	33.0	19.0	35.9
0.6B, v2 only	72.3 ∗	61.0 ∗	53.0 ∗	51.0 ∗	59.4 ∗
Untrained 4B	89.1	65.0	67.0	63.0	71.1
4B, v2 only	90.1	82.0 ∗	72.0	71.0 ∗	78.8 ∗
4B, full recipe	90.1	85.0 ∗	79.0 ∗	77.0 ∗	82.8 ∗

Appendix

Table 9: Cue Recall@10 (%) on LoCoMo-Plus by relation type. v2 is a single simulator source (gpt-4.1-mini, concrete facts followed by requests to an assistant); the full recipe mixes v2, v3 and v2d (seed 0). ∗ Significantly above the untrained backbone of the same size (

p<0.05

, exact McNemar).

Variant	LP	LoCoMo	LoCoMo QA $\Delta$
(a) Which side to train, and which score to train on [0.6B, v2 data]
Untrained 0.6B	35.9	67.3	–
Both sides, full fine-tuning	49.6	55.1	–
Both sides, LoRA	50.1	56.1	–
Query side, association term trained alone	54.1	64.5	–
Query side, residual (summed) score	59.4	60.8	–

Appendix

Table 10: Ablations of the design choices in Section 4 . Columns 2–3 are retrieval Recall@10 (%); LP (LoCoMo-Plus) tests associative recall and LoCoMo ordinary lookup. The last column is the end-to-end change on ordinary LoCoMo QA relative to the untrained backbone (mean over seeds where three exist). Shaded rows are the configuration used in the paper for that block; bold marks the best LP Recall@10 in each block.

Method	Read	Score	$\Delta$	$p$
Madeleine (seed 0)	0	52.4	–	–
Untrained Qwen3-Embedding-4B	0	39.7	$\downarrow$ 12.7	$1.2{\times}10^{-5}$
(i) Read-time LLM rewriting or guessing
QueryLink, original config.	2	23.7	$\downarrow$ 28.7	$1.6{\times}10^{-19}$
QueryLink, native union	2	35.9	$\downarrow$ 16.5	$1.3{\times}10^{-7}$
QueryLink, rewriting + Qwen3-4B	2	34.4	$\downarrow$ 18.0	$9.0{\times}10^{-9}$

Appendix

Table 11: Routes that do not learn association, on LoCoMo-Plus (official protocol,

K{=}3

). Read : LLM calls per query at retrieval time.

\downarrow

: score drop relative to Madeleine (seed 0), computed before rounding;

p

: exact McNemar test against it. Rerankers re-score the top 50 candidates of Madeleine .

Explore similar work

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents
Apr 20, 2026Shuqi Cao, Jingyi He, Fei TanConversational Memory Multi-Turn Large Language Model Conversations
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Remember by Asking: Retrieval-Induced Memory Evolution for LLM Agents
Sep 28, 2026Wanqi Zhou, Jiawei Lu, Yang Wang +4Memorization
Chang’an University
RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory
Aug 13, 2026Jingbo Ji, Lingyi Li, Xilong Cheng +4Long-Term Agent Memory Long-Term Memory
Communication University of China, Beijing, China. · Zhilian Yinghe Technology Co., Ltd., Beijing, China · State Key Laboratory of Media Convergence and Communication, Beijing, China

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Shuqi Cao, Jingyi He, Fei Tan

East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Conversational Memory Multi-Turn Large Language Model Conversations Summarization Semantic Anchor Hierarchical

Open paper AskPDF arXiv

Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.

Abstract

Explore similar work

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Remember by Asking: Retrieval-Induced Memory Evolution for LLM Agents

RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory