Beyond Memory Construction: Rethinking Memory Access for LLM-based Conversational Agents

Donghua Cai; Yongheng Deng; Yifei Wang; Zijun Shen; Ju Ren

cs.CLSep 27, 2026

Beyond Memory Construction: Rethinking Memory Access for LLM-based Conversational Agents

Authors: Donghua Cai, Yongheng Deng, Yifei Wang, Zijun Shen, Ju Ren

Organizations: Tsinghua University · Beijing Institute of Technology · Nanjing University

Abstract

Memory is a core component of conversational agents, enabling coherent and context-aware behavior over long interactions. Recent approaches commonly rely on LLM-based memory construction, where raw interactions are rewritten into structured memory units and later retrieved via a RAG pipeline. While effective in controlled settings, we show that this paradigm breaks down in long-horizon, high-entropy conversations: memory construction becomes increasingly lossy and unstable as context length and information complexity grow, and incurs prohibitive cost due to repeated LLM invocation. To address these limitations, we propose Threader, a memory system that shifts the focus from memory construction to efficient, structure-aware access over raw interactions. Instead of rewriting interactions, Threader preserves them as first-class memory, organizes them into topic-coherent segments via lightweight incremental segmentation, and enables accurate retrieval through multi-view representation. At query time, it performs multi-signal retrieval that combines segment-level access with localized evidence matching, ensuring both completeness and coherence. Extensive experiments demonstrate that Threader consistently improves answer accuracy and evidence recall, while significantly reducing the memory construction overhead.

Figures & tables

Panel of Figure 1 — Figure 1: Limitations of existing LLM-based memory construction methods. As interaction length and information entanglement increase (simulated via session concatenation with LongMemEval), representative methods exhibit (a) degraded QA accuracy and shrinking memory size, (b) increasing loss of query-relevant evidence, and (c) substantial token amplification during memory construction.

Figure 2: Overview of Threader . It incrementally segments interaction streams into topic-coherent units, encodes them with multi-view representations, and retrieves related evidence with multi-signals.

Method	SS-User	SS-Assi.	SS-Pref.	Multi-S.	Know.-U.	Temp-R.	Overall
GPT-5-nano
Self-RAG	87.14	94.64	43.33	56.39	73.08	73.68	71.40
HippoRAG2	71.43	94.64	53.33	60.15	76.92	69.92	70.40
Mem0	80.00	46.43	26.67	39.85	69.23	58.65	55.00
MemU	81.43	78.57	43.33	55.64	53.85	68.42	64.20
A-Mem	84.29	82.14	20.00	30.83	56.41	27.82	46.60

Table 1: Main results on LongMemEval. Threader retrieves the top-12 segments for LongMemEval. SS-User, SS-Assi., SS-Pref., Multi-S., Know.-U., and Temp.-R. denote Single-Session-User, Single-Session-Assistant, Single-Session-Preference, Multi-Session, Knowledge-Update, and Temporal-Reasoning, respectively. Results are grouped by backbone. Within each backbone block, the best result in each column is shown in bold , the second-best distinct result is underlined .

Method	Generalize	Preference	User-	User-	Update-	New-	Preference	Overall
Method	-New-Scn.	-Align	Shared	Mentioned	Reason	Ideas	Evolution	Overall
GPT-5-nano
Self-RAG	29.82	52.73	16.28	47.06	67.68	38.71	48.20	41.60
HippoRAG2	56.14	65.45	53.49	70.59	69.70	39.78	51.08	55.35
Mem0	45.61	43.64	31.01	47.06	70.71	39.78	55.40	47.88
MemU	35.09	41.82	47.29	41.18	58.59	27.96	51.80	45.33

Table 2: Main results on PersonaMem. Threader retrieves the top-8 segments for PersonaMem. Generalize-New-Scn., Preference-Align, User-Shared, User-Mentioned, Update-Reason, New-Ideas, and Preference-Evolution denote generalizing-to-new-scenarios, provide-preference-aligned-recommendations, recalling-user-shared-facts, recalling-facts-mentioned-by-the-user, recalling-reasons-behind-previous-updates, suggesting-new-ideas, and tracking-full-preference-evolution, respectively. Results are grouped by backbone. Within each backbone block, the best result in each column is shown in bold , the second-best distinct result is underlined .

Variant Top-5 Top-12 ACC Full-R Mean-R ACC Full-R Mean-R Segmentation ablation Threader 82.60 82.55 90.26 90.60 93.40 96.50 w/o Seg. 82.00 – – 87.00 – – Chunk-500 82.40 – – 85.60 – – Chunk-1K 74.20 – – 86.40 – – Chunk-2K 63.60 – – 79.80 – – Multi-view representation ablation Emb. w/o Msg. 71.20 67.45 77.34 85.40 86.81 91.65 Emb. - Full 70.20 64.68 75.52 84.00 82.55 88.72 Emb. - User 68.20 61.91 73.45 83.00 82.76 89.07 Emb. - Asst. 62.40 55.11 68.59 75.40 73.40 82.88

Variant Top-5 Top-12 ACC Full-R Mean-R ACC Full-R Mean-R Retrieval signal ablation Threader 82.60 82.55 90.26 90.60 93.40 96.50 w/o Emb. 79.00 79.15 87.40 84.60 86.17 92.19 w/o BM25 81.60 80.43 88.21 88.20 91.49 94.81 w/o Rerank. 78.40 77.23 86.43 88.20 91.49 95.08 w/o Emb.&BM25 80.40 76.17 85.31 85.60 88.51 93.54 w/o Emb.&Rerank. 74.80 73.19 81.83 83.20 84.68 91.06 w/o BM25&Rerank. 72.00 69.57 80.63 87.00 89.15 93.76 w/o Msg.-Emb. 81.80 81.28 88.72 89.20 93.19 96.29 Retrieval granularity ablation Segment Unit 82.60 82.55 90.26 90.60 93.40 96.50 Message Unit 82.00 – – 85.80 – –

Table 3: Ablation study on the key design axes of Threader . We evaluate the impact of segmentation, multi-view representation, multi-signal retrieval, and retrieval granularity on answer accuracy and evidence recall under Top-5 and Top-12 settings. Full-R denotes the fraction of queries with complete evidence coverage, and Mean-R denotes average evidence recall. Emb.-User, Emb.-Asst., and Emb.-Full correspond to using only user, assistant, or full interaction views, respectively.

Panel of Figure 3 — Figure 3: Top-K sensitivity analysis on LongMemEval and PersonaMem.

Method	Memory Construction					Query-Time
Method	Time	#Calls	In Tok.	Out Tok.	Cost $	Latency	#Calls	In Tok.	Out Tok.	Cost $
Self-RAG	13.18	0	0	0	0	$93.52$	$6.55$	2.3 K	0.4 K	0.0016
HippoRAG2	$110.84$	$95.26$	249 K	38 K	0.1604	$51.02$	$3.03$	17.5 K	0.7 K	0.0081
Mem0	$1239.62$	$47.94$	231 K	64 K	0.1948	8.38	0	0	0	0
MemU	2448.94	$305.65$	936 K	121 K	0.5680	199.47	19.09	16.1 K	2.5 K	0.0112
A-Mem	$2032.45$	1415.19	2107 K	79 K	0.9692	$12.35$	$1.00$	9.9 K	0.9 K	0.0054

Table 4: Cost comparison of agent memory systems across offline memory construction and online query-time serving. MemOS is omitted due to its black-box interface, which prevents reliable cost attribution. We report construction cost per 100K input tokens and query-time cost per query, covering runtime/latency (s), LLM calls, input and output token consumption, and monetary cost ($). The best and worst reported results are highlighted in green and red , respectively.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# QA	# Sample	Sess./Sample	Tok./Sess.	Turn/Sess.	Tok./Turn
LongMemEval	500	500	47.7	2,160.73	5.12	421.88
PersonaMem	589	37	1	26,165.38	174.65	149.82

Appendix

Table 5: Dataset statistics. We report the number of QA pairs, the number of samples, and dialogue-level statistics for each benchmark. All metrics with “/” denote average values, including sessions per sample, tokens per session, turns per session, and tokens per turn.

Name	Setting	Sessions / Sample	Tokens / Session
LME	Original	47.65	2,311.21
LME-M2	2 session concatenation	23.55	4,676.53
LME-M4	4 session concatenation	11.50	9,576.78
LME-M8	8 session concatenation	5.48	20,085.12

Appendix

Table 6: Statistics of the constructed motivation data. We progressively concatenate adjacent sessions in LongMemEval to create longer-session settings. The number of QA examples and the total dialogue content are kept unchanged, while the average number of sessions per sample decreases and the average session length increases.

Panel of Figure 4 — Figure 4: Limitations of existing LLM-based memory construction methods (MemU). As interaction length and information entanglement increase (simulated via session concatenation with LongMemEval), MemU exhibits (a) degraded QA accuracy and shrinking memory size, (b) increasing loss of query-relevant evidence.

Figure 5: Case study of memory degradation. As the concatenation factor increases, Mem0 and MemU construct fewer memory items and fewer words, and progressively lose query-relevant evidence. MI denotes the number of memory items, and W denotes the total number of words in the constructed memory.

Figure 6: Case study of uncontrolled memory retention. Rewrite-based memory construction does not preserve evidence monotonically across session-granularity settings. The same evidence may be covered, partially retained, or missing under different concatenation factors. The emoji markers denote Covered , Partial , and Missing , respectively. MI denotes the number of memory items, and W denotes the total number of words in the constructed memory.

Base Model	Threshold ( $\tau$ )	LongMemEval Dataset (50 Samples)
Base Model	Threshold ( $\tau$ )	$F\_1$ $\uparrow$	W- $F\_1$ $\uparrow$	BOR $\rightarrow 1$	Pur. $\uparrow$	Cov. $\uparrow$
BERT Family
BERT	0.15	0.497	0.600	1.266	0.777	0.682
BERT	0.20	0.483	0.594	1.169	0.763	0.708
BERT	0.25	0.475	0.592	1.100	0.754	0.727
RoBERTa Family

Appendix

Table 7: Performance comparison of varying architectures and thresholds on a randomly sampled subset of 50 evaluation instances from the LongMemEval dataset. The arrows (

\uparrow

\rightarrow 1

) indicate the optimal direction for each metric. Best and second-best results within each model family are highlighted in bold and underlined , respectively. The highlighted row indicates our final selected segmenter.

Relative training size	F1 $\uparrow$	BOR $\rightarrow 1$
50%	0.399	0.914
80%	0.471	1.114
90%	0.454	1.054
100% (original)	0.483	1.169
110%	0.486	1.247
120%	0.478	1.205

Appendix

Table 8: Effect of training data size on segmentation quality. Training sizes are expressed relative to the original setting. The original configuration is shown in bold.

Class ratio (C:S)	F1 $\uparrow$	BOR $\rightarrow 1$
$1:1$	0.421	2.228
$2:1$	0.477	1.276
$3:1$	0.484	1.511
$4:1$ (original)	0.483	1.169

Appendix

Table 9: Effect of the training class ratio on segmentation quality. C and S denote CONTINUE and SEGMENT , respectively. The original configuration is shown in bold.

Supervision setting	F1 $\uparrow$	BOR $\rightarrow 1$
Hard-only	0.226	0.505
Hard:Soft $=2:1$	0.321	0.674
Hard:Soft $=1:2$	0.459	0.816
Soft-only	0.472	1.907
Original mix ( $0.902:1$ )	0.483	1.169

Appendix

Table 10: Effect of the hard/soft supervision mixture on segmentation quality under fixed training hyperparameters. The original hard-to-soft ratio is

0.902:1

. The original configuration is shown in bold.

Method	SS-User	SS-Assi.	SS-Pref.	Multi-S.	Know.-U.	Temp-R.	Overall
Qwen3.5-plus
Self-RAG	90.00	94.64	60.00	69.17	79.49	80.45	79.00
HippoRAG2	77.14	96.43	80.00	72.93	88.46	71.43	78.60
Mem0	72.86	21.43	26.67	40.60	48.72	56.39	47.60
MemU	98.57	82.14	40.00	82.71	87.18	45.86	73.20
A-Mem	92.86	98.21	26.67	54.89	78.21	64.66	69.60

Appendix

Table 11: Main results on LongMemEval with Qwen-family models. Threader retrieves the top-12 segments for LongMemEval. SS-User, SS-Assi., SS-Pref., Multi-S., Know.-U., and Temp.-R. denote Single-Session-User, Single-Session-Assistant, Single-Session-Preference, Multi-Session, Knowledge-Update, and Temporal-Reasoning, respectively. Results are grouped by backbone. Within each backbone block, the best result in each column is shown in bold , the second-best distinct result is underlined .

Method	Generalize	Preference	User-	User-	Update-	New-	Preference	Overall
Method	-New-Scn.	-Align	Shared	Mentioned	Reason	Ideas	Evolution	Overall
Qwen3.5-plus
Self-RAG	29.82	63.64	10.85	52.94	77.78	59.14	72.66	52.29
HippoRAG2	78.95	80.00	79.84	58.82	83.84	46.24	74.10	73.17
Mem0	64.91	54.55	37.21	58.82	83.84	39.78	86.33	61.97
MemU	61.40	72.73	81.39	64.71	85.86	38.71	76.26	70.97

Appendix

Table 12: Main results on PersonaMem with Qwen-family models. Threader retrieves the top-8 segments for PersonaMem. Generalize-New-Scn., Preference-Align, User-Shared, User-Mentioned, Update-Reason, New-Ideas, and Preference-Evolution denote generalizing-to-new-scenarios, provide-preference-aligned-recommendations, recalling-user-shared-facts, recalling-facts-mentioned-by-the-user, recalling-reasons-behind-previous-updates, suggesting-new-ideas, and tracking-full-preference-evolution, respectively. Results are grouped by backbone. Within each backbone block, the best result in each column is shown in bold , and the second-best distinct result is underlined .

Method	Accuracy (%)
Threader w/o multi-view representation	78.4
Threader (full pipeline)	81.2

Appendix

Table 13: Full-pipeline ablation of multi-view representation on LongMemEval , with

K_{1}

reduced from 40 to 10. Both variants retain BM25 retrieval, message-level dense scoring, and cross-encoder reranking.

Variant ACC Segmentation ablation Threader 69.95 Chunk-300. 65.70 Chunk-500 67.40 Chunk-700 63.67 Chunk-1K 59.76 Multi-view representation ablation Emb. w/o Msg. 58.40 Emb.-Full 58.23 Emb.-User 58.23 Emb.-Asst. 54.67

Variant ACC Retrieval signal ablation Threader 69.95 w/o Emb. 67.40 w/o BM25 65.87 w/o Rerank. 66.04 w/o Emb.&BM25 62.82 w/o Emb.&Rerank. 66.55 w/o BM25&Rerank. 62.14 w/o Msg.-Emb. 65.70 Retrieval granularity ablation Segment Unit 69.95 Message Unit 67.06

Appendix

Table 14: Ablation study on the key design axes of Threader on Personamem. We evaluate the impact of segmentation, multi-view representation, retrieval signals, and retrieval granularity on answer accuracy. Emb.-User, Emb.-Asst., and Emb.-Full correspond to using only user, assistant, or full interaction views, respectively.

$(w_{u},w_{a},w_{f})$	Top-5		Top-8		Top-10		Top-12
$(w_{u},w_{a},w_{f})$	Full-R	Mean-R	Full-R	Mean-R	Full-R	Mean-R	Full-R	Mean-R
$(0.3,0.35,0.35)$	0.6596	0.7644	0.7638	0.8464	0.8064	0.8758	0.8489	0.9041
$(0.4,0.3,0.3)$	0.6660	0.7666	0.7723	0.8541	0.8128	0.8807	0.8617	0.9129
$(0.5,0.25,0.25)$	0.6745	0.7734	0.7723	0.8534	0.8170	0.8817	0.8681	0.9165
$(0.6,0.2,0.2)$	0.6553	0.7625	0.7723	0.8534	0.8213	0.8832	0.8638	0.9141
$(0.7,0.15,0.15)$	0.6511	0.7509	0.7702	0.8524	0.8277	0.8856	0.8596	0.9141

Appendix

Table 15: Segment-view weight analysis. We evaluate different dense segment-view weights

(w_{u},w_{a},w_{f})

using segment-level dense retrieval only, with BM25 matching and message-level dense matching disabled. Full-R denotes the fraction of queries with complete evidence coverage, and Mean-R denotes average evidence recall.

Figure 7: Fusion weight sensitivity analysis. Performance heatmaps of different BM25 interpolation weights $\beta$ and final fusion weights $\alpha$ under Top-5 and Top-12 retrieval settings.

Memory method	Response model	SS-Pref.	New-Ideas	Preference Evolution	Overall
Mem0	GPT-5-mini	26.67	27.96	66.91	48.47
Mem0	GPT-5.6-sol	46.67	48.39	58.99	53.82
EverMemOS	GPT-5-mini	96.67	26.88	64.03	54.58
EverMemOS	GPT-5.6-sol	100.00	59.14	79.14	74.43
Threader	GPT-5-mini	96.67	45.16	63.31	60.69
Threader	GPT-5.6-sol	100.00	65.59	77.70	75.95

Appendix

Table 16: Accuracy (%) on three selected abstraction-related question types. SS-Pref. is from LongMemEval; New-Ideas and Preference Evolution are from PersonaMem. Overall denotes accuracy pooled across the 262 questions in these three categories. GPT-5.6-sol uses maximum reasoning effort. The best result in each metric is shown in bold.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents
Apr 20, 2026Shuqi Cao, Jingyi He, Fei TanPersistent Memory for Language Models Hierarchical Memory
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Cognis: Context-Aware Memory for Conversational AI Agents
Mar 27, 2026Parshva Daftari, Khush Patel, Shreyas Kapale +2Agent Memory Management Memory-Augmented Language Model Agents
Lyzr Research
Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation
Apr 13, 2026Yuqian Wu, Wei Chen, Zhengjun Huang +5Conversational Memory Long-Term Conversational Memory
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · National University of Singapore

Apr 20, 2026J/K move · Enter open · S save

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Shuqi Cao, Jingyi He, Fei Tan

East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Persistent Memory for Language Models Hierarchical Memory Long-Term Conversational Memory Agent Memory Management

Open paper AskPDF arXiv

Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.

Abstract

LLM Agent Memory

Explore similar work

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Cognis: Context-Aware Memory for Conversational AI Agents

Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation