MemComposer: Learning to Assemble Memories for LLM Agents via Reinforcement Learning

Yixuan Xie; Xueqiang Xu; Xinyi Fan; Xinxi Lyu; Tao Feng; Xing Fan; Chenlei Guo; Jie Hao

cs.AIOct 15, 2026

MemComposer: Learning to Assemble Memories for LLM Agents via Reinforcement Learning

Authors: Yixuan Xie, Xueqiang Xu, Xinyi Fan, Xinxi Lyu, Tao Feng, Xing Fan, Chenlei Guo, Jie Hao,

Organizations: University of Illinois Urbana-Champaign · Amazon

Abstract

Agent memory enables large language model (LLM) agents to reuse past experiences without updating their parameters. However, semantic similarity alone does not guarantee memory usefulness: downstream performance depends on how memories are selected and presented to the agent. We propose MemComposer, a memory composer that jointly examines the current query and retrieved candidates to decide which experiences to include, how many to select, and how to order them. We train the composer using Experience Policy Optimization (ExPO), a GRPO-based framework that evaluates assembled memory contexts through complete agent interactions and uses task-success feedback to optimize the assembly policy. Across three $τ^2$ -bench domains with two frozen executors, MemComposer outperforms the strongest baseline for each executor while reducing average memory-token usage by 16.9% and 4.9%, respectively. The learned policy also transfers to unseen domains, executors, and memory banks, and benefits from memories generated by stronger models without further training. These results demonstrate the value of learning query-specific memory assembly using downstream task feedback.

Figures & tables

Figure 1: Semantic Similarity $\neq$ Memory Usefulness. The more similar memory misses the permission issue; the less similar one leads to success.

Panel of Figure 2 — Figure 2: Both the number and the order of memory matter. Frozen Qwen3-4B-Instruct agent on $\tau^{2}$ -bench pass 1 averages $c/n$ across tasks; pass 4 averages $\binom{c}{4}/\binom{n}{4}$ , where $c$ of $n$ trials succeed. Dashed lines denote no-memory performance. (a) Increasing $k$ does not consistently improve performance, and the best $k$ differs across domains. (b) Reordering the same memory set changes per-task outcomes, though no single order is best on average.

Figure 3: Overview of MemComposer . Top: Self-generated trajectories are reviewed and curated into a fixed memory bank. Bottom: The composer samples $G$ variable-size, ordered memory sequences per query, each evaluated over $J$ executor trials; ExPO updates the composer with their group-relative advantages, while the executor and memory bank stay fixed.

Qwen3-4B-Instruct
Method	Retail		Airline		Telecom		Avg.
Method	pass 1	pass 4	pass 1	pass 4	pass 1	pass 4	pass 1	pass 4
No Memory	45.6	30.0	32.5	10.0	22.5	2.5	33.5	14.2
Similarity-based	53.7	30.0	50.0	25.0	29.4	5.0	44.4	20.0
Complexity-based	51.9	37.5	40.0	30.0	37.5	15.0	43.1	27.5
Qwen3-Reranker-8B	54.4	35.0	46.2	30.0	31.9	7.5	44.2	24.2

Table 1: Results on

\tau^{2}

-bench with two frozen executor backbones. pass k is the probability that a task is solved in all

k

independent trials ( Yao et al., 2024 ) : pass 1 is the average success rate and pass 4 reflects consistency. Heuristic baselines use the

k\in\{1,3,5,7\}

with the highest average pass 1 on the test set. Avg. is the macro average over domains. Bold and underline mark the best and second-best results within each backbone; ties share the marking.

Figure 4: Lower-ranked candidates account for a larger share of selections after training . Dashed lines indicate the mean rank of selected memories.

Figure 5: Training weakens the preference for successful experiences. Fraction of candidates selected within each rank band, for the (a) untrained and (b) trained selectors on retail.

Alternative order	Mean $\Delta$	$\Delta$ pass 1 (points)
Similarity	+0.220	+5.5
Similarity reversed	+0.340	+8.5
Length $\uparrow$	+0.270	+6.8
Length $\downarrow$	+0.150	+3.8
Average	+0.245	+6.1

Table 2: Advantage of the composer’s order over alternative orderings of the same memories.

\Delta

: mean difference in successful trials over 100 tasks;

\Delta

pass

{}^{1}=25\Delta

points. Positive values favor the composer.

Figure 6: Effect of the memory source. Only the agent that generated the memory pool varies along the x-axis. Qwen3-4B-Instruct is the executor’s own memory, on which MemComposer was trained. Both methods draw from the same top-10 candidates per task: similarity top-5 keeps the five most similar, while MemComposer selects its own subset. We report pass 4 on retail (40 tasks), airline (20 tasks), and their unweighted average. Shading marks where MemComposer is ahead (blue) or behind (red). Markers are measured values; curves are a visual guide only.

MemComposer
Method	Airline		Telecom
Method	pass 1	pass 4	pass 1	pass 4
No Memory	32.5	10.0	22.5	2.5
Similarity †	50.0	25.0	29.4	5.0
Complexity †	40.0	30.0	37.5	15.0
Vanilla selector	43.8	30.0	34.4	10.0

Table 3: The retail-trained composer transfers to unseen domains. Qwen3-4B-Instruct executor. In-domain : trained on the target domain; Retail

\rightarrow

: trained on retail only. † As in Table 1 .

Method	pass 1	pass 4
No Memory	58.1	32.5
Similarity †	68.1	40.0
Complexity †	65.6	45.0
Vanilla selector	64.4	30.0
MemComposer	75.0	47.5
MemComposer (4B)	70.6	50.0

Table 4: The learned policy transfers to a different executor. Retail, Ministral-3-8B-Instruct executor with its own memory bank. MemComposer : trained with Ministral-3-8B-Instruct; MemComposer (4B): trained with Qwen3-4B-Instruct. † As in Table 1 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-4B-Instruct
Baseline	$\Delta$ pass 1	$\Delta$ pass 4
No memory	+17.3 [9.2, 25.6]	+18.3 [9.2, 28.3]
Similarity ( $k{=}7$ )	+6.5 [0.2, 13.1]	+12.5 [3.3, 21.7]
Complexity ( $k{=}3$ )	+7.7 [0.0, 15.4]	+5.0 [ $-$ 3.3, 13.3]
Qwen3-Reranker	+6.6 [0.2, 13.1]	+8.3 [ $-$ 0.8, 17.5]
MemRL	+6.3 [ $-$ 0.4, 12.7]	+15.0 [5.0, 25.0]

Appendix

Table 5: Gap between MemComposer and each baseline per executor, averaged over the three domains, with 95% paired-bootstrap CIs over test tasks in brackets. Bold marks intervals that exclude zero. Heuristic baselines use the budget

k

reported in Table 1 .

Baseline	$\Delta$ pass 1	$\Delta$ pass 4
No memory	+19.2 [13.2, 25.2]	+17.9 [10.8, 25.0]
Similarity †	+6.6 [2.0, 11.3]	+8.8 [2.5, 15.0]
Complexity †	+6.4 [1.1, 11.6]	+5.0 [ $-$ 2.1, 12.1]
Qwen3-Reranker	+8.8 [4.5, 13.2]	+11.2 [5.0, 17.5]
MemRL	+5.4 [1.0, 9.9]	+8.3 [1.7, 15.4]
Memento	+5.8 [1.0, 10.8]	+5.8 [ $-$ 1.2, 12.9]

Appendix

Table 6: Gap between MemComposer and each baseline pooled over both executors (six domain–executor cells, equal weight), with 95% paired-bootstrap CIs in brackets. Bold marks intervals that exclude zero. † Each executor uses its budget

k

from Table 1 .

Agent	Domain	#	Clean	Fake	Fail
Qwen3-4B	retail	143	37.8	2.8	59.4
	airline	58	19.0	19.0	62.1
	telecom	141	12.8	8.5	78.7
Ministral-3-8B	retail	144	27.1	11.1	61.8
	airline	57	15.8	14.0	70.2
	telecom	144	22.9	6.9	70.1

Appendix

Table 7: Memory bank statistics: number of memories (#) and the share (%) of clean passes, fake passes, and failures.

Qwen3-4B-Instruct
Parameter	Value
LoRA
Rank / scaling factor	64 / 128
Dropout	0
Rollout sampling
Candidates per prompt	10

Appendix

Table 8: Training hyperparameters for the Qwen3-4B-Instruct composer.

Ministral-3-8B-Instruct
Parameter	Value
LoRA
Rank / scaling factor	64 / 128
Dropout	0
Rollout sampling
Candidates per prompt	10

Appendix

Table 9: Training hyperparameters for the Ministral-3-8B-Instruct composer.

Qwen3-4B-Instruct
Method	Retail		Airline		Telecom		Avg.
Method	pass 1	pass 4	pass 1	pass 4	pass 1	pass 4	pass 1	pass 4
Similarity-based
$k=1$	49.4	30.0	43.8	20.0	25.6	10.0	39.6	20.0
$k=2$	46.3	27.5	40.0	20.0	36.2	12.5	40.8	20.0
$k=3$	51.2	42.5	38.8	15.0	37.5	20.0	42.5	25.8

Appendix

Table 10: Full retrieval-budget sweep for similarity-based and complexity-based reranking with two frozen executors. Avg. denotes the macro-average across the three domains.

Figure 7: Performance–memory-token trade-offs across 2 executor backbones and all domains. Each point represents a method or retrieval-budget configuration. The dashed line shows the baseline Pareto frontier: the highest observed baseline pass 1 within each memory-token budget. Memory tokens count the selected context supplied to the executor, averaged per task.

	retail	airline	telecom
Training tasks	74	30	74
Test tasks	40	20	40
Trials per task	4	4	4
Test episodes	160	80	160

Appendix

Table 11: Official train/test split of each

\tau^{2}

-bench domain. Memory banks are built from the training tasks only, and all results are reported on the test tasks.

Figure 8: Retail trade-off for Qwen3-4B-Instruct. Same layout as Figure 7 for a single domain.

Panel of Table 12 — Table 12: An airline case illustrating complementary guidance from two trajectory memories.

Panel of Table 13 — Table 13: A failed memory contains the correct exchange action for a related task involving the same customer. Content is abbreviated; starred boxes highlight the historical error and the reusable action.

Panel of Table 14 — Table 14: Same five memories, different order. A retail case where the executor succeeds in 4/4 trials under MemComposer ’s ordering and 0/4 under similarity ordering. Starred memories are the two the executor acts on.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 428 papers

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University
ElasticMem: Latent Memory as a Learnable Resource for LLM Agents
May 29, 2026Tao Feng, Chongrui Ye, Tianyang Luo +5LLM Agent Memory Persistent Memory for Language Models
University of Illinois Urbana-Champaign · Nanyang Technological University
Mem- $π$ : Adaptive Memory through Learning When and What to Generate
May 20, 2026Xiaoqiang Wang, Chao Wang, Hadi Nekoei +5LLM Agent Memory RL for LLM Agents
1ServiceNow AI Research · 2Mila – Quebec AI Institute · 3Université de Montréal +3

Jul 15, 2026J/K move · Enter open · S save

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Levina Li+10

University of California Los Angeles · University of Washington · Northwestern University

Persistent Memory for Language Models Memory Management for LLM Agents Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

ElasticMem: Latent Memory as a Learnable Resource for LLM Agents

Mem-πππ: Adaptive Memory through Learning When and What to Generate

Mem- $π$ : Adaptive Memory through Learning When and What to Generate