Structured Distillation for Personalized Agent Memory: 11x Token Reduction with Retrieval Preservation

Sydney Lewis

cs.AIMar 13, 2026

Structured Distillation for Personalized Agent Memory: 11x Token Reduction with Retrieval Preservation

Authors: Sydney Lewis

Abstract

Long conversations with an AI agent create a simple problem for one user: the history is useful, but carrying it verbatim is expensive. We study personalized agent memory: one user's conversation history with an agent, distilled into a compact retrieval layer for later search. Each exchange is compressed into a compound object with four fields (exchange_core, specific_context, thematic room_assignments, and regex-extracted files_touched). The searchable distilled text averages 38 tokens per exchange. Applied to 4,182 conversations (14,340 exchanges) from 6 software engineering projects, the method reduces average exchange length from 371 to 38 tokens, yielding 11x compression. We evaluate whether personalized recall survives that compression using 201 recall-oriented queries, 107 configurations spanning 5 pure and 5 cross-layer search modes, and 5 LLM graders (214,519 consensus-graded query-result pairs). The best pure distilled configuration reaches 96% of the best verbatim MRR (0.717 vs 0.745). Results are mechanism-dependent. All 20 vector search configurations remain non-significant after Bonferroni correction, while all 20 BM25 configurations degrade significantly (effect sizes |d|=0.031-0.756). The best cross-layer setup slightly exceeds the best pure verbatim baseline (MRR 0.759). Structured distillation compresses single-user agent memory without uniformly sacrificing retrieval quality. At 1/11 the context cost, thousands of exchanges fit within a single prompt while the verbatim source remains available for drill-down. We release the implementation and analysis pipeline as open-source software.

Figures & tables

Component	Source	Content	Analogy
exchange_core	LLM	What was accomplished (1–2 sentences)	Commit message
specific_context	LLM	One distinguishing technical detail	The diff
room_assignments	LLM	1–3 thematic rooms (type, key, label)	Directory tree
files_touched	Regex	File paths referenced in exchange	Changed files

Table 1: Components of a distilled object.

Statistic	Value
Total conversations	4,182
Total exchanges	14,340
Distilled objects	12,427
Avg. verbatim length	371 tokens
Avg. distilled length	38 tokens
Compression ratio	11 $\times$ ∗

Table 2: Corpus statistics from 6 months of production AI coding sessions across 6 projects. Token counts based on cl100k_base encoding, measured on corpus totals (overlapping conversations only).

Family	Configuration	MRR
Best verbatim	Full Text / BM25-FTS	0.745
Best pure distilled	Distill Core+Rooms / Exact / Weighted	0.717
Best cross-layer	Cross: BM25(V)+HNSW(D) / BM25-FTS + HNSW / CombMNZ	0.759

Table 3: Headline retrieval results. Full configuration-level results appear in Appendix A .

Figure 1: MRR heatmap by mode and retrieval mechanism (best fusion per cell). Darker cells indicate higher MRR.

Figure 2: Effect sizes (Cohen’s

d

) for per-mechanism comparisons (verbatim vs each distilled mode, averaging across fusion strategies within each mode) with 95% bootstrap confidence intervals. Grouped by mechanism.

Figure 3: Distillation impact by query type. Heatmap showing mean grade delta relative to pooled verbatim baseline; negative values (red) indicate degradation, positive values (blue) indicate improvement.

Figure 4: Grade composition by search mode. Bars show the proportion of pooled results at grades 0–3 for each mode. Grade 3 is the largest bucket in every mode, but the overall profiles remain similar, with most mass concentrated in grades 1 and 3.

Measure	Value
Fleiss’ $\kappa$ across all graders	0.175 (Slight), over 3,672 items
Best pairwise $\kappa$	Phi-3.5-Mini / Qwen3-8B: 0.260 (Fair)
Worst pairwise $\kappa$	Mistral-7B / Phi-3.5-Mini: 0.131 (Slight)

Table 4: Agreement summary. The full pairwise agreement table appears in Appendix A .

Figure 5: Pairwise proportion agreement heatmap among 5 LLM graders. Values show the fraction of items where two graders assigned identical grades. Cohen’s $\kappa$ values (Table 4 and Appendix A ) are lower due to chance correction.

Mechanism	Significant	Non-significant
HNSW	0/10	10/10
Exact	0/10	10/10
BM25-Okapi	10/10	0/10
BM25-FTS	10/10	0/10

Table 5: Statistical summary by retrieval mechanism. The full comparison table appears in Appendix A .

Figure 6: Query coverage overlap between Full Text and Distilled configurations (grade-3, P@1). Left: best single config from each family, showing near parity at 11

\times

compression. Right: oracle over all configs per family, showing complementary retrieval pathways.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Mode	Mech	Fus	MRR	Grade	95% CI	P@1	nDCG
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	CMZ	0.759	2.237	[2.20, 2.28]	0.622	0.667
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W70	0.758	2.261	[2.22, 2.30]	0.622	0.675
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W90	0.757	2.235	[2.20, 2.27]	0.632	0.667
Cross: BM25(V)+HNSW(D)	BM25-Okapi + HNSW	W70	0.754	2.182	[2.14, 2.22]	0.642	0.650
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W95	0.751	2.234	[2.20, 2.27]	0.617	0.666
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	Add	0.750	2.254	[2.22, 2.29]	0.612	0.671

Appendix

Table 6: Retrieval quality across 107 configurations: verbatim baselines vs. distilled modes at 11

\times

compression (201 queries, 0–3 relevance scale). Sorted by MRR descending. V = Full Text, D = Distill, F = Files, R = Rooms. Add = Additive, Wt = Weighted.

Query type	Stronger setup	Mean grade	Comparison setup	Mean grade
Exact term	Core+Files / HNSW / Wt	2.08	Full Text / HNSW	1.93
Conceptual	Full Text / HNSW	2.08	Core+Files / HNSW / RRF	1.96

Appendix

Table 7: Representative query-type contrasts.

Mode	Mechanism	Fusion	Conceptual	Phrase	Exact Term
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	CombMNZ	$2.26\pm 0.05$	$2.12\pm 0.07$	$2.36\pm 0.08$
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W70	$2.29\pm 0.05$	$2.13\pm 0.07$	$2.40\pm 0.08$
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W90	$2.25\pm 0.05$	$2.10\pm 0.08$	$2.40\pm 0.08$
Cross: BM25(V)+HNSW(D)	BM25-Okapi + HNSW	W70	$2.17\pm 0.06$	$2.10\pm 0.08$	$2.31\pm 0.08$
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	W95	$2.25\pm 0.05$	$2.10\pm 0.08$	$2.38\pm 0.08$
Cross: BM25(V)+HNSW(D)	BM25-FTS + HNSW	Additive	$2.27\pm 0.05$	$2.13\pm 0.07$	$2.39\pm 0.08$

Appendix

Table 8: Retrieval quality disaggregated by query type (

\pm

95% CI). Row order follows Table 6 (MRR descending). Query types: conceptual (abstract topics), phrase (multi-word patterns), exact term (specific identifiers).

Measure	Value
Fleiss’ $\kappa$ (all 5 graders)	0.175 (Slight)
Mean pairwise Cohen’s $\kappa$	0.192 (Slight)
Best pair (Phi-3.5-Mini $\leftrightarrow$ Qwen3-8B)	0.260 (Fair)
Worst pair (Mistral-7B $\leftrightarrow$ Phi-3.5-Mini)	0.131 (Slight)
Items rated by all 5 graders	3,672

Appendix

Table 9: Inter-rater agreement across 5 LLM graders. Despite low absolute

\kappa

, the mechanism-dependent preservation finding is directionally consistent across all graders. Interpretation follows Landis & Koch (1977):

\kappa<0.20

= Slight,

0.20

–

0.40

= Fair.

Mechanism	Mode	Fusion	$\Delta$	$t$	$p$	$d$
HNSW
	D Core †	—	$-$ 0.052	$-$ 1.93	0.055	$-$ 0.14N
	D Core+F	Add	+0.105	3.07	0.0025**	0.22S
	D Core+F	RRF	+0.089	2.73	0.0068**	0.19N
	D Core+F †	Wt	$-$ 0.028	$-$ 1.00	0.319	$-$ 0.07N
	D Core+R	Add	+0.014	0.52	0.601	0.04N

Appendix

Table 10: Mechanism-dependent quality preservation: per-mechanism pairwise comparisons (40 total). Baseline: Full Text (trivial fusion). Vector search (HNSW, Exact) shows non-significant degradation; keyword search (BM25) shows consistent significant degradation. Bonferroni-corrected

\alpha=0.00125

. † = best distilled MRR for this mechanism. *** = significant after Bonferroni (

p<0.00125

). ** = significant at nominal

\alpha=0.05

but not after Bonferroni correction. V = Full Text, D = Distill, F = Files, R = Rooms. Add = Additive, Wt = Weighted. Effect: N = negligible, S = small, M = medium.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

Jul 17, 2026

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.

Jing Yu, Yibo Zhao, Jiaming Zhang +1

School of Data Science and Engineering, East China Normal University

Persistent Memory for Language Models Long-Term Conversational Memory Agent Memory Management Memory Management for LLM Agents

Ask PDF

Sep 28, 2026

Stashbird: Episode-Grounded Lifecycle Reconciliation for Conversational Agent Memory

AI agents require memory that remains consistent as conversational evidence changes. We present Stashbird, an episode-grounded memory system that uses provenance to reconcile derived graph and retrieval state during updates and deletion. Repeated evidence extends support, corrections supersede prior active relations, and deletion removes unsupported state while preserving relations with remaining support. Under fixed semantic decisions, reconciliation produces correct final states in all 20 update sequences, compared with eight without reconciliation, and satisfies the defined checks across 263 deletion interventions. Across four long-term memory benchmarks, Stashbird achieves competitive question-answering accuracy and strong model efficiency. On LoCoMo, it uses 76.4x fewer ingestion prompt tokens than Graphiti and 8.1x fewer retrieval prompt tokens than reproduced Hindsight

Chidera Biringa, Lucas Yannul, Xiaowen Wang +5

Microsoft

Persistent Memory for Language Models Graph-Based Agent Memory Long-Term Conversational Memory Agent Memory Management

Ask PDF

Jun 11, 2026

MemRefine: LLM-Guided Compression for Long-Term Agent Memory

Large language model (LLM) agents are increasingly expected to operate over long-term interactions, where information from past dialogues must be preserved and recalled to support future tasks. However, as interactions accumulate, the memory store grows without bound and fills with redundant entries that inflate storage cost and degrade retrieval by crowding out the most useful evidence. Furthermore, this is especially limiting on resource-constrained platforms with hard memory budgets, motivating us to formulate storage-budgeted memory management, the task of keeping an already constructed memory store within a fixed budget while preserving information useful for future interactions. To this end, we then propose MemRefine, an LLM-guided framework that, since surface similarity poorly reflects factual value, uses similarity only to propose candidate pairs and defers delete, merge, and preserve decisions to an LLM judge based on factual content, iterating until the budget is met. Across multiple memory frameworks and long-term conversation benchmarks, MemRefine consistently meets target budgets while preserving downstream performance and outperforming rule-based baselines under tight budgets.

Minjae Kim, Jinheon Baek, Soyeong Jeong +1

Korea University · KAIST

Persistent Memory for Language Models LLM Context Compression Long-Term Conversational Memory Agent Memory Management Memory Management for LLM Agents

Ask PDF

Abstract

LLM Agent Memory

Explore similar work

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

Stashbird: Episode-Grounded Lifecycle Reconciliation for Conversational Agent Memory

MemRefine: LLM-Guided Compression for Long-Term Agent Memory