Grounding Memory Summarization in Utility Intent

Zhenyu Lei; Mingjia Shi; Xingbo Fu; Haoyu He; Qi R. Wang; Jundong Li

cs.CLSep 27, 2026

Grounding Memory Summarization in Utility Intent

Authors: Zhenyu Lei, Mingjia Shi, Xingbo Fu, Haoyu He, Qi R. Wang, Jundong Li

Organizations: University of Virginia · Northeastern University

Abstract

Existing summarizers for memory systems are typically optimized for human-facing criteria such as faithfulness, which misaligns with their true objective: preserving the evidence needed to support future queries. We show that conditioning summarization on query-answer pairs substantially improves answer quality, and that this utility-aware behavior is transferable across queries. Motivated by these findings, we propose MemSuit, a self-distillation framework in which a teacher summarizer, conditioned on observed query-answer pairs, produces utility-aware memory entries that a student learns to reproduce from the raw conversation alone. To prevent collateral erasure where conditioning on a single query-answer pair discards evidence relevant to other plausible queries, the teacher decomposes each block into multiple self-contained entries that preserve distinct query-relevant facets as independently retrievable units. To align the retriever with the compact, fact-dense style of teacher entries, we further fine-tune the embedding model with a contrastive objective supervised by teacher entries. Across a diverse suite of conversational query types, MemSuit consistently outperforms state-of-the-art baselines, confirming the value of grounding memory in downstream utility.

Figures & tables

Panel of Figure 1 — Figure 1: (a) A short conversation summarized by different methods. Neither the rule-based nor the vanilla LLM summary retains enough details to answer the two example queries. (b) Evidence coverage on LoCoMo (Qwen2.5-3B), measured as the fraction of query-supporting facts preserved in the summary. Vanilla LLM summaries lose substantial evidence for temporal and open-domain queries.

Panel of Figure 2 — Figure 2: (a) Granting the summarizer access to the downstream query-answer pair ( Utility ) substantially outperforms the query-agnostic baseline ( Vanilla ). (b) In-context utility-aware demonstrations surpasses the vanilla baseline. Gains peak when demonstration and evaluation types match and persist under mixing, indicating that utility-aware patterns transfer across queries

Figure 3: Overview of MemSuit. Left: A teacher summarizer conditioned on observed Q-A pairs decomposes each conversation into utility-aware memory entries; a student is distilled to reproduce these entries from the raw conversation alone. Right: A contrastive objective calibrates the retriever’s embedding space, pulling each query toward its grounding entries and pushing it away from entries grounding other queries.

Model	Method	Multi		Temp		Open		Single		Avg
Model	Method	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Llama-3.1-8B-Instruct	LoCoMo	21.33	16.42	17.46	14.41	10.60	6.69	30.82	23.41	20.05	15.23
	MemoryBank	24.60	19.46	12.01	9.56	10.47	8.18	38.45	34.34	21.38	17.89
	A-Mem	17.08	13.21	24.00	18.26	11.93	10.48	29.62	21.28	20.66	15.81
	Mem0	28.98	23.35	19.93	15.18	19.73	16.41	45.76	39.79	28.60	23.68
	SimpleMem	26.92	17.66	35.11	23.58	22.41	16.89	35.92	30.83	30.09	22.24

Table 1: Main Results. MemSuit achieves the best average performance.

Model	Variant	Multi		Temp		Open		Single		Avg
Model	Variant	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Llama-3.1-8B-Instruct	w/o Rtr-Cal	26.02	18.36	41.03	28.19	21.16	17.29	38.86	33.17	31.77	24.25
	w/o Self-Dist	30.92	21.68	40.24	28.38	21.36	16.72	40.92	36.02	33.36	25.70
	w/o Cal & Dist	23.16	15.51	30.88	21.30	17.53	14.50	33.38	28.62	26.24	19.98
	MemSuit	31.18	23.55	40.77	27.94	25.42	20.83	44.89	38.61	35.56	27.73
Qwen2.5-3B-Instruct	w/o Rtr-Cal	21.23	16.22	26.65	18.57	14.58	11.56	30.09	26.58	23.14	18.23

Table 2: Ablation study of MemSuit. Removing either component hurts performance.

Figure 4: Decomposition granularity ablation on LoCoMo (Qwen2.5-3B). Fixed schemes (1/5/10 turns per entry) underperform MemSuit’s adaptive decomposition across all categories of queries.

Model		Recall@1	Recall@5	Recall@10
Llama	Before	25.77	54.64	64.43
	After	35.05	63.40	69.59
Qwen	Before	21.78	44.06	51.49
	After	26.73	48.02	55.94

Table 3: Recall of grounding-evidence entries before and after Entry-Aware Retrieval Calibration.

Figure 5: Cosine similarities of positive and negative query-entry pairs before and after calibration. Calibration mainly lifts the tail of the positive distribution.

Longmemeval	F1	Bleu1
Mem0	8.92	7.01
SimpleMem	28.91	21.40
MemSuit	28.16	20.84

Table 4: Generalization on a 50-example LongMemEval subset with MemSuit trained on LoCoMo. MemSuit matches the strongest baseline despite distribution shift.

Figure 6: F1 vs. summary tokens per conversation on LoCoMo. MemSuit attains the highest F1 at a competitive token budget.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Qwen2.5-3B	Llama-3.1-8B
Multi-F1	22.63	22.89
Multi-BLEU	16.36	15.22
Temp-F1	28.66	27.78
Temp-BLEU	20.40	18.39
Open-F1	15.40	17.15
Open-BLEU	13.15	13.96

Appendix

Table 5: Sensitivity of MemSuit (Qwen2.5-3B backbone) to the choice of grounding-entry judge

\psi

. Results are stable across judge models.

Metric	MemSuit	Adaptive (non-target)
Multi-F1	22.63	19.62
Multi-BLEU	16.36	13.82
Temp-F1	28.66	24.75
Temp-BLEU	20.40	19.57
Open-F1	15.40	15.85
Open-BLEU	13.15	12.10

Appendix

Table 6: Ablation isolating adaptive decomposition from query conditioning (Qwen2.5-3B backbone). The non-target variant performs adaptive decomposition without conditioning on the query–answer pair.

LongMemEval	F1	BLEU-1
Mem0	8.92	7.01
SimpleMem	28.91	21.40
MemSuit (LoCoMo-trained, OOD)	28.16	20.84
MemSuit (LongMemEval-trained, ID)	30.89	24.32

Appendix

Table 7: In-domain training on LongMemEval. MemSuit trained directly on LongMemEval surpasses both the out-of-domain (LoCoMo-trained) variant and the strongest baseline.

Metric	LoCoMo	MemoryBank	A-mem	Mem0	SimpleMem	MemSuit
Multi-F1	8.51	9.91	18.22	15.44	17.56	19.88
Multi-BLEU	6.96	7.21	14.91	11.27	12.77	15.22
Temp-F1	6.34	12.07	23.86	16.08	22.39	23.87
Temp-BLEU	5.14	9.04	18.60	12.20	15.15	15.55
Open-F1	9.48	6.97	15.09	15.35	16.27	22.88
Open-BLEU	7.36	5.24	12.83	12.25	12.70	19.52

Appendix

Table 8: Results on LoCoMo with a newer-generation backbone, Qwen3-1.7B , for both MemSuit and all baselines. MemSuit maintains the best average F1 and BLEU.

Explore similar work

JustMem: Just-Enough Memory Access for Long-Term Conversations
Sep 17, 2026Guanhua Chen, Yanting Wang, Wenjing Zhi +1Conversational Memory Peak Memory
Beihang University
Beyond Static Summarization: Proactive Memory Extraction for LLM Agents
Jan 8, 2026Chengyuan Yang, Zequn Sun, Wei Wei +1Large Language Model Agents Summarization
State Key Laboratory for Novel Software Technology, Nanjing University, China · National Institute of Healthcare Data Science, Nanjing University, China
Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering
Sep 7, 2026Yifan Wang, Xinkui Lin, Yongxiu Xu +9Conversational Memory Peak Memory
University of Electronic Science and Technology of China, Chengdu, China 4State Key Laboratory of Internet Architecture, Tsinghua University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 3School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · 5Independent Researcher

Sep 17, 2026, cs.CLJ/K move · Enter open · S save

JustMem: Just-Enough Memory Access for Long-Term Conversations

Guanhua Chen, Yanting Wang, Wenjing Zhi, Lei Sha

Beihang University

Conversational Memory Peak Memory

Open paper AskPDF arXiv

Efficient long-term conversational memory requires retrieving sufficient evidence without indiscriminately expanding the context presented to the language model. This is challenging because relevant evidence may be distributed across multiple sessions, while compression may discard details needed for answering. Different queries therefore require different forms of memory access. To capture these demands, we formulate memory access along two dimensions: discovery breadth, which controls how broadly evidence is searched, and reading fidelity, which controls whether evidence is read in compact form or recovered from the original conversation. Based on this formulation, we introduce JustMem, which stores conversation history as compact atomic memories and adapts memory access along these two dimensions to each query. Specifically, LOOKUP handles local evidence, COMPOSE broadens discovery for distributed evidence, and REPLAY increases reading fidelity for fidelity-sensitive evidence. On LoCoMo and LongMemEval-S, JustMem achieves the highest mean accuracy and retrieval recall among the compared memory systems while using substantially fewer generative-model tokens for memory construction and inference.

Abstract

Explore similar work

JustMem: Just-Enough Memory Access for Long-Term Conversations

Beyond Static Summarization: Proactive Memory Extraction for LLM Agents

Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering