M$^\star$: Every Task Deserves Its Own Memory Harness

Wenbo Pan; Shujie Liu; Xiangyang Zhou; Shiwei Zhang; Wanlu Shi; Mirror Xu; Xiaohua Jia

cs.PLApr 10, 2026

M $^\star$ : Every Task Deserves Its Own Memory Harness

Authors: Wenbo Pan, Shujie Liu, Xiangyang Zhou, Shiwei Zhang, Wanlu Shi, Mirror Xu, Xiaohua Jia

Organizations: City University of Hong Kong · Microsoft

Abstract

Large language model agents rely on specialized memory systems to accumulate and reuse knowledge during extended interactions. Recent architectures typically adopt a fixed memory design tailored to specific domains, such as semantic retrieval for conversations or skills reused for coding. However, a memory system optimized for one purpose frequently fails to transfer to others. To address this limitation, we introduce M $^\star$ , a method that automatically discovers task-optimized memory harnesses through executable program evolution. Specifically, M $^\star$ models an agent memory system as a memory program written in Python. This program encapsulates the data Schema, the storage Logic, and the agent workflow Instructions. We optimize these components jointly using a reflective code evolution method; this approach employs a population-based search strategy and analyzes evaluation failures to iteratively refine the candidate programs. We evaluate M $^\star$ on four distinct benchmarks spanning conversation, embodied planning, and expert reasoning. Our results demonstrate that M $^\star$ improves performance over existing fixed-memory baselines robustly across all evaluated tasks. Furthermore, the evolved memory programs exhibit structurally distinct processing mechanisms for each domain. This finding indicates that specializing the memory mechanism for a given task explores a broad design space and provides a superior solution compared to general-purpose memory paradigms.

Figures & tables

Figure 2: System overview of Mstar . Numbered stages show the workflow from seed memory program ( 0 ) to final test evaluation ( 5 ): population pool ( 1 ), task evaluation ( 2 ), reflective code evolution ( 3 ), and compile/runtime quality checks ( 4 ).

Retrieval-based Systems (Memorizing Episodes)
	LoCoMo		ALFWorld		HealthBench		PRBench
Method	F1	L-J	Unseen	Seen	Data	Emerg.	Legal	Finance
No Memory	0.036	0.030	0.738	0.640	0.242	0.429	0.431	0.269
Vector Search	0.256	0.400	0.643	0.720	0.264	0.400	0.466	0.308
G-Memory	0.224	0.380	0.690	0.560	0.309	0.447	0.499	0.318
Mem0	0.373	0.540	0.738	0.640	0.216	0.413	0.450	0.321

Table 1: Main results across four task families. Columns report LoCoMo token F1 and LLM-judge score (L-J), ALFWorld success rate, HealthBench rubric score, and PRBench rubric score. Baselines are grouped by memory paradigm. Best per column in bold , second best in green .

Figure 3: ALFWorld evolution trajectory. The main panel plots each child program’s validation score and the best score so far across 20 evolution iterations, with crosses marking compile failures and the dashed line marking the Vanilla RAG baseline. Numbered panels highlight representative code changes that structure task records, reweight retrieval, and add mismatch penalties and execution checklists for selecting the next action.

Variant	F1	$\Delta$
Mstar	0.459	—
$-$ Instruction	0.353	$-$ 0.106
$-$ Code	0.256	$-$ 0.203
Max parent selection	0.381	$-$ 0.078
$-$ Diversity	0.318	$-$ 0.141

Table 2: Ablation study on LoCoMo. Rows report the full system and variants that remove or alter one design choice; metric is token F1.

\Delta

reports absolute change from the full system.

Figure 4: Program embedding landscape. Each evolved program is embedded with a code embedding model and projected to 2D via t-SNE [ 26 ] . (a, b) Population-based search ( Mstar ) and linear search, with colored edges tracing parent-child lineage. (c) All programs from the evaluated configurations and the LoCoMo linear-search ablation, colored by configuration. Marker shapes denote the storage architecture discovered by evolution: circles for vector + LLM, triangles for vector-only, squares for SQL + vector, rotated squares for SQL-only, and crosses for LLM-centric designs; diamonds mark seed programs and stars mark the best program for each configuration.

Figure 5: Cross-task transfer of evolved memory programs. Each panel fixes one target benchmark and compares memory programs evolved on different source benchmarks. Highlighted bars mark native-task programs, and the dashed line marks the universal seed baseline.

Benchmark	Mean $\pm$ Std	CV	Best Baseline	Win
LoCoMo	$0.421_{\pm 0.042}$	10.0	0.373 (Mem0)	4/5
HB Data	$0.391_{\pm 0.023}$	5.9	0.327 (GEPA+VS)	5/5
PR Finance	$0.561_{\pm 0.032}$	5.7	0.449 (GEPA)	5/5

Table 3: Stability across evolution seeds. Mean and std. are computed over five independent stability runs with random seeds; metrics match the primary columns in Table 1 . CV is the coefficient of variation in percent, Best Baseline is the strongest fixed baseline, and Win is the number of seeds above that baseline.

	(a) ALFWorld Unseen						(b) LoCoMo
	Simple	Clean	Cool	Heat	2-Obj		Single	Temp.	Causal	Open
	$(n{=}8)$	$(n{=}11)$	$(n{=}7)$	$(n{=}6)$	$(n{=}8)$	Min	$(n{=}18)$	$(n{=}18)$	$(n{=}9)$	$(n{=}55)$	Min
No Memory	0.88	0.82	0.57	0.83	0.62	0.57	0.02	0.00	0.11	0.04	0.00
Mem0	0.88	0.64	0.57	1.00	0.75	0.57	0.33	0.26	0.37	0.43	0.26
Trajectory Retrieval	0.75	0.73	0.71	0.67	0.75	0.67	0.18	0.33	0.34	0.28	0.18
GEPA + Vector Search	0.88	1.00	0.57	1.00	0.62	0.57	0.15	0.38	0.39	0.31	0.15

Table 4: Per-category performance breakdown. (a) ALFWorld Unseen success rate by task type. (b) LoCoMo token F1 by query type.

n

is the number of test episodes or test queries per category. Min reports the minimum score across shown categories. Best per column in bold .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Split	Train	Val	Test
LoCoMo	—	272 sessions	1,440 QA	100 QA
ALFWorld	Unseen	4,652 trajectories	145 ep.	50 ep.
	Seen	4,652 trajectories	150 ep.	50 ep.
HealthBench	Data Tasks	257	120	100
	Emergency	260	122	100
PRBench	Legal	100	100	50

Appendix

Table 5: Dataset split sizes. Rows report train, validation, and test sizes for every benchmark split used by the evaluation pipeline.

Benchmark	Config	Iter.	Task	Reflect.	Judge	Toolkit	Input	Output	Cost	Time
LoCoMo	Mstar	20	5802	48	—	135	4.9M	4.2M	$25.43	5.1h
	Vector Search	0	200	0	—	272	294K	71K	$0.54	2m
	No Memory	0	200	0	—	272	219K	64K	$0.45	1m
ALFWorld	Mstar	20	3355	25	—	443	7.5M	1.4M	$13.92	100h
(Unseen)	Vector Search	0	80	0	—	497	711K	50K	$0.76	14m
	No Memory	0	80	0	—	497	708K	50K	$0.76	11m

Appendix

Table 6: Computational cost of evolution runs and representative baselines. Rows report iteration count, API calls by role, GPT-family token counts, estimated dollar cost, and wall-clock time. Wall-clock time includes environment interaction overhead for ALFWorld runs.

Parameter	Value
Evolution iterations	20
Population seed programs	3
Selection strategy	Softmax ( $\tau=0.15$ )
Static validation subset	32–60 items ( $k$ -means clustering)
Rotating validation subset	5 items per iteration
Train-to-validation ratio	2 (train episodes = $2\times$ validation items)

Appendix

Table 7: Hyperparameters for Mstar evolution. Rows list shared settings and benchmark-specific ranges used in the main experiments.

Seed	Storage	Retrieval	LLM in read
Vector Search	ChromaDB chunks	Top- $k$ cosine similarity	No
LLM Summarizer	In-memory list	Concatenate + LLM summarize	Yes (1 call)
Experience Learner	Separate lesson/fact lists	Return all	No

Appendix

Table 8: Seed program comparison. Rows summarize each seed program’s storage strategy, retrieval strategy, and read-time LLM use.

Benchmark	Lines	SQLite	ChromaDB	LLM in read	Schema fields	Test
LoCoMo	425	Yes	Yes	No	7	0.459
ALFWorld (Unseen)	276	Yes	No	Yes	8	0.881
ALFWorld (Seen)	441	Yes	No	No	8	0.700
HB-Emergency	393	Yes	No	Yes	7	0.493
HB-Data	769	Yes	Yes	Yes	12	0.390
PR-Legal	1184	Yes	Yes	Yes	22	0.660

Appendix

Table 9: Structural comparison of best evolved programs across benchmark configurations. Rows report program size, storage backends, read-time LLM use, schema size, and test score.

	Simple	Movable	Clean	Light	Cool	Heat	2-Obj
	$(n{=}8)$	$(n{=}1)$	$(n{=}11)$	$(n{=}1)$	$(n{=}7)$	$(n{=}6)$	$(n{=}8)$
No Memory	0.88	1.00	0.82	0.00	0.57	0.83	0.62
Vector Search	0.62	1.00	0.64	0.00	0.71	0.67	0.62
G-Memory	1.00	1.00	0.55	1.00	0.29	0.83	0.75
Mem0	0.88	1.00	0.64	0.00	0.57	1.00	0.75
Trajectory Retrieval	0.75	1.00	0.73	0.00	0.71	0.67	0.75

Appendix

Table 10: Complete per-category results for ALFWorld Unseen. Columns report success rate by task type. Best per column appears in bold for each reported task category.

	Simple	Clean	Light	Cool	Heat	2-Obj	Overall
	$(n{=}2)$	$(n{=}12)$	$(n{=}2)$	$(n{=}8)$	$(n{=}7)$	$(n{=}15)$	$(n{=}50)$
No Memory	1.00	0.33	1.00	0.50	0.43	0.87	0.640
Vector Search	1.00	0.42	1.00	1.00	0.29	0.87	0.720
G-Memory	1.00	0.33	1.00	0.12	0.29	0.87	0.560
Mem0	1.00	0.25	1.00	0.62	0.43	0.87	0.640
Trajectory Retrieval	1.00	0.58	1.00	0.88	0.43	0.93	0.780

Appendix

Table 11: Complete per-category results for ALFWorld Seen. Columns report success rate by task type. Best per column appears in bold for each reported task category.

	Single-hop	Temporal	Causal	Open-domain	Overall
	$(n{=}18)$	$(n{=}18)$	$(n{=}9)$	$(n{=}55)$
No Memory	0.02	0.00	0.11	0.04	0.036
Vector Search	0.19	0.29	0.38	0.25	0.256
G-Memory	0.23	0.12	0.31	0.24	0.224
Mem0	0.33	0.26	0.37	0.43	0.373
Trajectory Retrieval	0.18	0.33	0.34	0.28	0.276

Appendix

Table 12: Complete per-category results for LoCoMo. Columns report token F1 by question category: single-hop recall, temporal reasoning, causal reasoning, and open-domain. Best per column appears in bold for each reported question category.

Benchmark	0	1	2	3	4	Mean	$\pm$ Std	CV	Best Baseline
	Test Score by Seed
LoCoMo	0.459	0.463	0.364	0.394	0.423	0.421	0.042	10.0%	0.373
HB-Data	0.390	0.382	0.357	0.411	0.413	0.391	0.023	5.9%	0.327
PR-Finance	0.586	0.544	0.582	0.582	0.513	0.561	0.032	5.7%	0.449

Appendix

Table 13: Per-seed test scores across stability experiments. Best Baseline denotes the strongest fixed baseline.

Benchmark	0	1	2	3	4	Mean	$\pm$ Std
	Best Validation Score
LoCoMo	0.333	0.273	0.289	0.312	0.333	0.308	0.027
HB-Data	0.424	0.381	0.378	0.397	0.380	0.392	0.020
PR-Finance	0.521	0.522	0.559	0.554	0.515	0.534	0.020

Appendix

Table 14: Best validation score during evolution by seed. Scores are computed on the static validation subset used to select the program for test evaluation.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 191 papers

AdMem: Advanced Memory for Task-solving Agents
Jun 5, 2026Runzhe Wang, Huilin Lu, Shengjie Liu +2Continual Learning for LLM Agents LLM Agent Memory
Princeton University · Amazon · Arm
Are We Ready For An Agent-Native Memory System?
Jun 23, 2026Wei Zhou, Xuanhe Zhou, Shaokun Han +5Agent Memory LLM Agent Memory
Shanghai Jiao Tong University · Tsinghua University · MemTensor (Shanghai) Technology Co., Ltd
Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University

Jun 5, 2026J/K move · Enter open · S save

AdMem: Advanced Memory for Task-solving Agents

Runzhe Wang, Huilin Lu, Shengjie Liu, Li Dong+1

Princeton University · Amazon · Arm

Continual Learning for LLM Agents LLM Agent Memory Persistent Memory for Language Models Procedural Memory for LLM Agents Long-Horizon Agent Tasks Multi-Agent LLMs+2

Open paper AskPDF arXiv

Large Language Models (LLMs) show promise as tool-using agents but remain limited in long-horizon tasks that require remembering, organizing, and reusing knowledge. Prior memory approaches aim to resolve the situation, but mainly focus on storing factual information. Recent work on procedural memory improves task reuse, yet often reduces to replaying past successes without addressing failure cases or online scalability. We introduce a unified and automatic memory framework that integrates semantic, episodic, and procedural memory in a bi-level design combining short-term and long-term stores. A multi-agent architecture with actor, memory, and critic agents enables automatic memory generation, reward annotation, and adaptive retrieval. Long-term memory is managed through reward-based evaluation, merging, and pruning, ensuring scalability and continual improvement. Experiments across various environments show that our approach improves robustness and success on long multi-turn tasks compared to existing baselines. This work highlights the importance of comprehensive, adaptive memory for advancing LLM-based agents.

Abstract

Agent Memory

Explore similar work

AdMem: Advanced Memory for Task-solving Agents

Are We Ready For An Agent-Native Memory System?

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents