MemPO: Self-Memory Policy Optimization for Long-Horizon Agents

Ruoran Li; Xinghua Zhang; Haiyang Yu; Shitong Duan; Xiang Li; Wenxin Xiang; Chonghua Liao; Xudong Guo

cs.AIFeb 28, 2026

MemPO: Self-Memory Policy Optimization for Long-Horizon Agents

Authors: Ruoran Li, Xinghua Zhang, Haiyang Yu, Shitong Duan, Xiang Li, Wenxin Xiang, Chonghua Liao, Xudong Guo,

Organizations: Tsinghua University · Tongyi Lab, Alibaba Group

Abstract

Long-horizon agents face the challenge of growing context size during interaction with environment, which degrades the performance and stability. Existing methods typically introduce the external memory module and look up the relevant information from the stored memory, which prevents the model itself from proactively managing its memory content and aligning with the agent's overarching task objectives. To address these limitations, we propose the self-memory policy optimization algorithm (MemPO), which enables the agent (policy model) to autonomously summarize and manage their memory during interaction with environment. By improving the credit assignment mechanism based on memory effectiveness, the policy model can selectively retain crucial information, significantly reducing token consumption while preserving task performance. Extensive experiments and analyses confirm that MemPO achieves absolute F1 score gains of 25.98 over the base model and 7.1 over the previous SOTA baseline, while reducing token usage by 67.58% and 73.12%. The code is released at https://github.com/TheNewBeeKing/MemPO.

Figures & tables

Figure 1: The self-memory inference process of our method, which only uses the previous step interaction for next step input with <mem> action.

Figure 2: Overview of MemPO. At step $t$ of any trajectory $\tau_{i}$ , the context is represented as $\{s_{t}^{mem},s_{t}^{think},s_{t}^{call},s_{t}^{resp}\}$ . The memory reward $R^{M}$ is calculated using conditional probabilities and contributes to the advantage $A^{M}$ . The final advantage is the sum of $A^{M}$ and the trajectory-level advantage $A^{T}$ . During inference, only the previous step’s content is used as context, discarding earlier information.

Local Wiki Search
Model	2-objective		4-objective		6-objective		8-objective		10-objective		Avg
Model	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1 $\uparrow$	EM $\uparrow$	TT $\downarrow$	PT $\downarrow$
Qwen2.5 (ReAct)	33.73	25.60	10.59	7.00	5.37	4.00	5.92	4.25	2.63	1.96	11.65	8.56	3.64	0.61
ReSearch	47.40	36.00	24.13	16.70	20.84	15.93	10.80	7.70	5.08	3.56	21.65	15.98	3.29	0.71
DeepResearcher	30.94	24.70	24.52	18.10	13.88	10.73	9.12	6.85	5.07	3.52	16.71	12.78	4.29	0.77
A-MEM	33.24	25.20	13.71	10.10	9.80	7.07	6.86	5.10	5.56	3.60	13.83	10.21	2.62	0.38

Table 1: The accuracy and token consumption for multi-objective tasks of baselines. Text with bold means SOTA.

Figure 3: The results of ablation study.

Figure 4: The result of step conditional probability analysis.

Figure 5: The result of grouped conditional probability analysis.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Local Wiki Search
Model	2-objective		4-objective		6-objective		8-objective		10-objective		Avg
Model	TT	PT	TT	PT	TT	PT	TT	PT	TT	PT	TT	PT
Qwen2.5 (ReAct)	1.90	0.38	2.80	0.51	3.97	0.63	4.89	0.76	4.66	0.77	3.64	0.61
Research	0.45	0.26	1.58	0.52	3.18	0.76	4.63	0.91	6.62	1.11	3.29	0.71
DeepResearcher	0.96	0.31	2.60	0.61	4.13	0.79	5.48	0.93	8.26	1.19	4.29	0.77
A-MEM	1.14	0.34	2.16	0.38	2.61	0.38	3.49	0.41	3.69	0.40	2.62	0.38

Appendix

Table 2: The token consumption for multi-objective tasks of baselines. Text with bold means SOTA.

Multi-hop QA
Model	2WikiMultiHopQA				Bamboogle				HotpotQA				Musique				Avg
Model	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT
qwen2.5 (ReAct)	45.21	35.90	1.12	0.29	43.70	32.80	0.95	0.24	47.12	34.30	1.07	0.26	23.86	14.40	2.06	0.37	39.97	29.35	1.30	0.29
ReSearch	50.07	41.90	0.55	0.28	53.61	40.80	0.41	0.24	50.26	33.90	0.40	0.23	29.63	17.80	0.58	0.29	45.89	33.60	0.48	0.26
DeepResearcher	51.44	43.90	1.24	0.34	48.48	37.60	1.22	0.30	51.96	38.40	1.01	0.29	26.55	17.00	2.54	0.44	44.61	34.23	1.50	0.34
GRPO (no mem)	62.35	53.80	0.93	0.32	53.18	40.80	0.79	0.28	57.29	42.30	0.62	0.27	33.78	21.50	0.86	0.33	51.65	39.60	0.80	0.30

Appendix

Table 3: The results of multi-hop QA datasets.They are the first part of single-objective QA datasets.

Single-hop QA
Model	NQ				PopQA				TriviaQA				Avg
Model	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT
qwen2.5 (ReAct)	49.18	36.30	1.87	0.30	47.77	40.50	1.43	0.26	62.73	52.60	1.02	0.22	53.23	43.13	1.44	0.26
ReSearch	52.49	38.00	0.23	0.16	52.41	43.80	0.23	0.16	62.04	50.10	0.24	0.17	55.65	43.97	0.23	0.16
DeepResearcher	50.15	39.50	1.19	0.25	47.90	40.80	0.60	0.22	61.89	52.30	0.95	0.23	53.32	44.20	0.91	0.23
GRPO (no mem)	56.04	43.90	0.63	0.23	53.77	46.50	1.00	0.26	64.57	54.50	0.52	0.23	58.13	48.30	0.72	0.24

Appendix

Table 4: The results of single-hop QA datasets.They are the second part of single-objective QA datasets.

Deep Research
Model	GAIA				Frames				WebWalker				Avg
Model	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT	F1	EM	TT	PT
qwen2.5 (ReAct)	14.28	8.74	1.54	0.22	29.37	20.00	0.61	0.15	29.93	8.50	0.42	0.14	24.52	12.41	0.86	0.17
ReSearch	16.00	8.74	0.41	0.17	35.04	22.40	0.37	0.16	33.02	7.29	0.25	0.13	28.02	12.81	0.34	0.15
DeepResearcher	22.44	16.50	4.94	0.33	37.69	24.40	0.44	0.17	32.10	9.31	1.16	0.19	30.75	16.74	2.18	0.23
Ours	25.07	17.48	0.62	0.12	37.56	24.00	0.33	0.10	31.85	8.91	0.29	0.09	31.49	16.79	0.41	0.10

Appendix

Table 5: The results of deep research datasets.

Prompt: You will answer multiple complex questions using iterative reasoning, summarization, and web search. At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to: 1. Update a cumulative, concise summary and perform reasoning within <mem>\n…\n</mem>\n <think>\n…\n</think>, respectively. <mem>\n…\n</mem> must include all essential information from previous <think> and <tool_response> tags. 2. Then choose one of the following actions: - If any question remains unanswered, issue a single query for one question inside <search>\n…\n</search>. - If all questions are answered, provide the final answers separated by semicolons within <answer>\n answer1; answer2; … \n</answer>. The answers must be concise, usually short phrases or words, and avoid any explanations. Important: - The summaries only contain concise and essential information related to the user questions from previous trajectories, but do not include any your own internal knowledge or reasoning content. - Must strictly follow one of these two structures: <mem>\n…\n</mem>\n<think>\n…\n</think>\n <search>\n…\n</search> or <mem>\n…\n</mem>\n<think>\n…\n</think>\n<answer>\n…\n</answer>. - Do not search multiple queries or questions simultaneously. Don’t give up searching for information until you find clear information that provides the answer.

Appendix

Table 6: Training prompt text with cumulative summary memory.

Query

As of Aug 3, 2024, the artist who released the album "Father of Asahd" went to the same high school as an Olympic diver. How many Olympic teams did this diver participate on?

Answer

Step 1

Assistant:

<mem>Need to identify the artist, the diver, and the number of Olympic teams.</mem>

Appendix

Table 7: Case study of our method.

Query

As of Aug 3, 2024, the artist who released the album "Father of Asahd" went to the same high school as an Olympic diver. How many Olympic teams did this diver participate on?

Answer

Step 1

Assistant:

<think>First, identify the artist who released the album "Father of Asahd" …</think>

Appendix

Table 8: Case study of ReAct. It was failed because of the Hallucination in Step 3. Greg Louganis is also a diver, but he is not mentioned anywhere in the context.

Agent Memory Management

Agent memory management studies how AI agents store, update, retrieve, compress, and forget information across long-running tasks.

See all 446 papers

SelfMem: Self-Optimizing Memory for AI Agents
Jul 4, 2026Shu Yang, Junchao Wu, Derek F. Wong +1LLM Agent Memory Long-Term Conversational Memory
1PRADA Lab, King Abdullah University of Science and Technology · NLP2CT Lab, University of Macau
MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents
Aug 7, 2026Zhiyuan Liu, Tinghong Ye, Chenghao Liu +2On-Policy Distillation Memory-Augmented Agents
School of Advanced Manufacturing and Robotics, Peking University · Zhejiang University · Harbin Institute of Technology
HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents
Jun 15, 2026Jiangze Yan, Yi Shen, Wenjing Zhang +6Credit Assignment in RL Long-Horizon Agent Tasks
Unicom Data Intelligence, China Unicom · Data Science & Artificial Intelligence Research Institute, China Unicom

Jul 4, 2026J/K move · Enter open · S save

SelfMem: Self-Optimizing Memory for AI Agents

Shu Yang, Junchao Wu, Derek F. Wong, Di Wang

1PRADA Lab, King Abdullah University of Science and Technology · NLP2CT Lab, University of Macau

LLM Agent Memory Long-Term Conversational Memory Self-Improving Agents Agent Memory Management

Open paper AskPDF arXiv

While current AI agents support increasingly long context windows, tool use, and skill execution for long-horizon tasks, they still require memory systems to effectively leverage historical experience. Existing memory frameworks typically rely on fixed storage, retrieval, and summarization mechanisms, which can be rigid across different tasks and often require manual tuning. To address this limitation, we propose SelfMem, a self-optimizing memory framework. Inspired by prior work on self-improving AI, we follow the principle of "teaching an agent to fish rather than giving it a fish." Instead of forcing the model to follow a predefined memory strategy or format, SelfMem provides an environment with memory tools and feedback signals that allow the agent to explore, evaluate, and refine its own memory strategy. Our results show that SelfMem consistently outperforms retrieval, compression, and agent-memory baselines on BEAM across conversation scales from 100K to 1M tokens. Compared with the strongest baseline, SelfMem improves the official score by 48.7%, 40.8%, and 41.9% at 100K, 500K, and 1M, respectively. Further question-type analysis shows broad robustness across diverse memory demands, and our optimization study shows that model-guided strategy refinement further improves performance.

Abstract

Agent Memory Management

Explore similar work

SelfMem: Self-Optimizing Memory for AI Agents

MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents

HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents