Agent Memory as Recommendation: Recommender-System Design for Long-Term Memory of LLM Agents

Xinyu Zhang; Weiwei Sun

cs.IROct 26, 2026

Agent Memory as Recommendation: Recommender-System Design for Long-Term Memory of LLM Agents

Authors: Xinyu Zhang, Weiwei Sun

Organizations: Fudan University

Abstract

Large language model (LLM) agents that serve users across many sessions must choose, for every request, the few stored memories that enter a limited context window. This choice largely decides whether the agent answers correctly, and it runs on every request at Web scale. This selection faces three challenges: a large and growing store, a per-request cost budget, and multi-session questions that need all of their evidence. Many memory systems build notes or graphs with hundreds to thousands of LLM calls per history, and each is designed on its own, so it is unclear which component yields the accuracy. These challenges mirror the properties of recommendation, so we formalize agent memory as budgeted slate recommendation, with the request as the user, stored memories as items and the context window as a slate. The formalization places memory systems in one design space of recommender cascades and factorizes accuracy into the completeness that each stage controls. Guided by it, we propose AgentMemRec, which adapts candidate generation, learning to rank and slate construction from recommender systems and selects memories without any LLM call. On multi-session questions of LongMemEval, AgentMemRec outperforms memory and context-selection methods, including A-Mem, HippoRAG 2, DynamicRAG and T-Mem, by 10 to 31 points of accuracy. It matches an LLM reranker that makes 64 calls per query at 3.2 times lower serving cost, and on LoCoMo it outperforms Mem0 and Zep without any LLM call at write time. The factorization attributes the gains to slates that more often hold all of the evidence, so recommender-system design offers an effective and efficient foundation for the long-term memory of LLM agents.

Figures & tables

Panel of Figure 1 — Figure 1. Agent memory as recommendation. Evidence for the request lies in two of 50 sessions, and a selector with no LLM call fills a slate of stored turns under a token budget $B$ , from which the agent’s LLM answers. Orange labels give the recommender role of each element. A long-term memory of 50 dated sessions is drawn as a row of chat cards with user and assistant turns. Sessions 3 and 17 are highlighted because they hold the evidence, pottery on Tuesdays and moved to Thursdays. A user asks on June 20 which day the pottery class is now. AgentMemRec, which recalls, ranks and fills with no LLM call, selects a slate of at most B tokens that contains both evidence turns. An answering LLM reads the slate and answers Thursdays, which is judged correct. Orange labels mark the history as items, the request as the user, the context as the slate and the judged answer as feedback.

Panel of Figure 2 — Figure 2. Long-term agent memory as budgeted slate recommendation. (a) Memory construction, recall, ranking and context assembly narrow the stored turns of a long-term memory to the agent’s context under a token budget $B$ , and each is a recommender stage (orange). (b) The compared systems by their choice at each stage, where marks an LLM call and the last column summarizes the stages that call an LLM. AgentMemRec learns its ranker over cheap features and makes no LLM call at any stage. Two panels share four stage columns: write-time map, candidate generation, ranker and slate builder. Panel a shows a funnel that narrows from left to right. A request and a history enter it. The first segment holds the dated sessions of turns, among them two gold evidence units and some optional records. The second holds a smaller pool that still contains both gold units. The third holds a ranking with the gold units on top, and the fourth a slate under the token budget B. An answering LLM reads the slate, and the judged answer is the reward. Braces under the funnel state that the first two stages decide whether the pool keeps all evidence and that the ranker decides whether the slate keeps the pooled evidence. The slate builder fills the budget with the ranked turns that fit, and the judged answer gives the accuracy. Panel b lists the compared systems in three groups: LLM-built memories, context selectors and an LLM reranker. Each row has one chip per stage and a last column, set apart by a dashed line, that summarizes the stages at which the system calls an LLM, none for Adaptive-k and AgentMemRec. LongMemEval design, HippoRAG 2, A-Mem and T-Mem spend LLM calls on the write-time map, and HippoRAG 2 and A-Mem also at candidate generation. Adaptive-k and DynamicRAG use per-question cutoffs, and DynamicRAG and Ref call an LLM in the ranker. AgentMemRec uses turns with optional records, a fused lexical and dense pool of 128 units, a ranker learned over cheap features and a greedy fill, with no LLM call.

LongMemEval-S (50 sessions, 100k tokens per history)
	LLM calls $\downarrow$			Accuracy (%) $\uparrow$
System	Write	Sel.	Tokens	Qwen	DeepSeek
Adaptive-k ( Taguchi et al., 2025 )	0	0	1,470	69.1	76.6
DynamicRAG-8B ( Sun et al., 2025 )	0	1	2,140	67.0	74.2
HippoRAG 2 ( Gutiérrez et al., 2025 )	442	1	2,520	59.9	63.2
A-Mem ( Xu et al., 2025 )	979	1	2,540	54.4	58.1

Table 1. Main results on the test questions of LongMemEval-S and LongMemEval-M with evidence in at least two sessions (

\kappa\geq 2

n=209

each). Gains are paired margins. LLM calls are per question, at write time (Write), amortized over the histories each build covers, and at selection time (Sel.). Tokens are mean slate lengths. Best in bold, second underlined.

Panel of Figure 3 — Figure 3. Accuracy on the LongMemEval-S test questions with evidence in at least two sessions, for Qwen (filled) and DeepSeek (hollow). (a) Every compared system by its LLM calls per question at write and selection time ( $n=209$ ). (b) The questions whose evidence spans three or more sessions. AgentMemRec reaches the accuracy of the 64-call reranker with no LLM call, and it is the most accurate system on the questions with the most scattered evidence. Two panels of accuracy on LongMemEval-S. Panel a plots accuracy against LLM calls per question: AgentMemRec at zero calls is as accurate as the reranker with 64 calls, and the LLM-built memories with hundreds to thousands of calls are lower. Panel b shows accuracy on questions whose evidence spans three or more sessions, where AgentMemRec has the highest accuracy for both answering models.

LongMemEval-S
System	Acc. $\uparrow$
Zep ∗ ( Rasmussen et al., 2025 )	63.8
RecMem ∗ ( Dai et al., 2026 )	69.2
TiMem ∗ ( Li et al., 2026a )	76.9
MemOS ∗ ( Li et al., 2025 )	77.8
AgentMemRec	83.2

Table 2. Accuracy (%) under the shared protocols of memory papers, with GPT-4o-mini answering. AgentMemRec scores above every reported system on LongMemEval-S and is at parity with our T-Mem reproduction on LoCoMo. ∗ Copied (Appendix D ), reprod. our reproduction, and † with the optional write-time records. Best in bold, second underlined.

		Qwen		Acc. $\uparrow$
System	$r_{\mathrm{slate}}$ $\uparrow$	$a_{\mathrm{c}}$	$a_{\mathrm{i}}$	Qwen	DeepSeek
Adaptive-k	0.684	87.1	30.3	69.1	76.6
DynamicRAG	0.630	86.3	34.0	67.0	74.2
HippoRAG 2	0.502	89.5	30.0	59.9	63.2
A-Mem	0.502	86.2	22.4	54.4	58.1
Ref (64 calls)	0.915	83.8	36.6	79.8	89.1

Table 3. Stage factorization on LongMemEval-S test,

\kappa\geq 2

(

n=209

), accuracies in %, so

\mathrm{Acc}=a_{\mathrm{i}}+(a_{\mathrm{c}}-a_{\mathrm{i}})\,r_{\mathrm{slate}}

. Slate completeness separates the systems, and AgentMemRec has the most complete slates with no LLM call. Lower rows rank its 128-turn pool (

r_{\mathrm{pool}}=0.981

, against 0.938 for the 64-turn pool of Ref). Best in bold, second underlined.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Features	$r_{\mathrm{slate}}$ , $\kappa\geq 2$	$r_{\mathrm{slate}}$ , $\kappa\geq 3$
All ten	0.949	0.852
without turn context	0.815	0.541
retrieval signals only	0.813	0.541
cross-encoder only	0.676	0.352
key overlap only	0.644	0.219
turn context only	0.537	0.582

Appendix

Table 4. Feature-group ablation of the ranker (label-only, LongMemEval-S test, each variant retrained). The learned combination exceeds every single feature group by 13 to 41 points of slate completeness at

\kappa\geq 2

	$\kappa=2$ ( $n=160$ )		$\kappa\geq 3$ ( $n=49$ )
System	Qwen $\uparrow$	DeepSeek $\uparrow$	Qwen $\uparrow$	DeepSeek $\uparrow$
Adaptive-k	78.4	85.0	38.8	49.0
DynamicRAG-8B	73.6	81.2	45.4	51.0
HippoRAG 2	67.8	72.3	34.2	33.2
A-Mem	59.8	65.9	36.7	32.7
T-Mem	74.4	78.1	57.1	57.1

Appendix

Table 5. Accuracy (%) on the LongMemEval-S test questions by the number of evidence sessions. AgentMemRec is the most accurate system at

\kappa\geq 3

, and at

\kappa=2

it exceeds every memory and context-selection method by 6.1 to 27.0 points, all gains. Best in bold, second underlined.

	Concerts		Festivals		Investment
System	Ev. $\uparrow$	OK $\uparrow$	Ev. $\uparrow$	OK $\uparrow$	Ev. $\uparrow$	OK $\uparrow$
Adaptive-k	2/5	0/4	3/6	0/4	0/1	0/4
DynamicRAG-8B	2/5	0/4	5/6	0/4	0/1	0/4
HippoRAG 2	0/5	0/4	3/6	0/4	0/1	0/4
A-Mem	1/5	0/4	4/6	2/4	0/1	0/4
T-Mem	own	0/1	own	1/1	own	0/1

Appendix

Table 6. Three LongMemEval-S test questions. Concerts : What is the order of the concerts and musical events I attended in the past two months, starting from the earliest? Festivals : How many movie festivals that I attended? Investment : I mentioned an investment for a competition four weeks ago? What did I buy? Ev. counts evidence turns in the 2,700-token slate, and OK counts correct slates per answering model (T-Mem answers once from its own memory). AgentMemRec keeps all evidence and answers correctly in all three, and bold marks every evidence turn kept and every slate correct.

System	Pool / ranker	LLM calls	LLM calls	Tokens	Qwen	DeepSeek	Qwen	DeepSeek
		Write	Sel.		Accuracy $\uparrow$		$\Delta$
Adaptive-k ( Taguchi et al., 2025 )	history / largest gap	0	0	1,180	57.8	60.8	$+12.2$ G	$+20.6$ G
DynamicRAG-8B ( Sun et al., 2025 )	fused 40 / listwise 8B	0	1	2,150	59.3	62.1	$+10.6$ G	$+19.3$ G
LongMemEval design ( Wu et al., 2025 )	rounds / dense, fact keys	251,622	0	2,530	56.3	60.3	$+13.6$ G	$+21.1$ G
native top-10, uncapped	rounds / dense, fact keys	251,622	0	4,945	64.1	65.6	$+5.9$ ref.	$+15.8$ ref.
HippoRAG 2 ( Gutiérrez et al., 2025 )	rounds / fact graph, PPR	$\approx$ 512k	1	2,520	46.1	48.1	$+23.9$ G	$+33.3$ G

Appendix

Table 7. LongMemEval-M test,

\kappa\geq 2

(

n=209

), with the GPT-4o judge and four perturbed slates per question. Accuracies are in %. LLM calls are generations per question at selection time, or in total at write time, and tokens are approximate mean slate lengths.

\Delta

is AgentMemRec minus the row in points, with its label: G gain, P parity, ref. a reference arm without a label. Best accuracy in bold, second underlined.

	AgentMemRec	Ref	Ratio
LLM calls per request	0	64	n/a
LLM tokens per request	0	20,861	n/a
GPU time per request (ms)	280	891	3.2 $\times$
Dollars per million requests	69	220	3.2 $\times$
Median latency, S (ms)	277	924	3.3 $\times$
90th-percentile latency, S (ms)	321	1,156	3.6 $\times$

Appendix

Table 8. Selection-stage serving cost on one A100 80 GB GPU for the 209 LongMemEval-S test questions with

\kappa\geq 2

. AgentMemRec makes no LLM call and needs about a third of the GPU time and latency of the 64-call reranker.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents
Jun 19, 2026Xingyu Chen, Rui Wang, Zhaopeng Tu +1LLM Agent Memory Persistent Memory for Language Models
Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent
LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory
Jul 17, 2026Jing Yu, Yibo Zhao, Jiaming Zhang +1Persistent Memory for Language Models Long-Term Conversational Memory
School of Data Science and Engineering, East China Normal University
FlexA-Mem: Budget-Adaptive Agentic Memory for LLM Agents
Oct 23, 2026Wujiang Xu, Jiaojiao Han, Xi Zhu +5Memory Management for LLM Agents Long-Term Conversational Memory
Rutgers University · Independent Researcher · Stanford University +2

Jun 19, 2026J/K move · Enter open · S save

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo

Shanghai Jiao Tong University, Tencent · Shanghai Jiao Tong University · Tencent

LLM Agent Memory Persistent Memory for Language Models Agent Memory Management Personal AI Agents Memory Management for LLM Agents

Open paper AskPDF arXiv

Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user's current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user's memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available: https://github.com/galaxyChen/AdaMem

Abstract

LLM Agent Memory

Explore similar work

AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

FlexA-Mem: Budget-Adaptive Agentic Memory for LLM Agents