Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

Zeyuan Liu; Jeonghye Kim; Xufang Luo; Dongsheng Li; Yuqing Yang

cs.LGFeb 26, 2026

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

Authors: Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, Yuqing Yang

Organizations: Microsoft Research · KAIST

Abstract

Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO $^2$ ), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO $^2$ achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO $^2$ demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO $^2$ as a promising framework for building more exploratory and generalizable LLM-based agents.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Comparison of the learning curves of GRPO and EMPO 2 (ours) on the ScienceWorld power-component task. While GRPO converges to suboptimal performance, EMPO 2 continues to improve and accomplish the task. (b) Comparison of EMPO 2 and other baselines in in-distribution (ID) and out-of-distribution (OOD) settings on and WebShop. In ID experiments, it adapts well to familiar environments, achieving 128.6% on ScienceWorld and 11.3% on Webshop improvements over GRPO. In OOD experiments, it also shows strong performance with few trials and no weight updates, indicating effective use of memory to explore unfamiliar environments. Full results are in Tables 1 , 2 , and Figure 8 .

Figure 2: Non-parametric updates can encourage exploration, bootstrapping parametric updates.

Figure 3: When training LLM with GRPO in ScienceWorld, the agent struggles because of insufficient exploration. For instance, in the task “turn on the red light bulb,” the agent must first find the red light bulb before activating it. However, the agent fails to locate it and, as a result, cannot complete the task. Rather than analyzing the cause of failure and exploring alternative actions, the agent proceeds unchanged, so its score stagnates even as additional training steps are taken.

Figure 4: In EMPO 2 , the current policy parameters $\pi_{\theta}$ are used to review past rollouts, with the resulting insights added to memory. This updated memory conditions subsequent rollouts and promotes exploration.

Figure 5: EMPO 2 mode combinations. By combining the two rollout modes and update modes, three EMPO mode configurations are possible: on-policy learning without memory, on-policy learning with memory and off-policy learning.

Figure 6: Masking tokens stabilizes training.

Figure 7: Policy entropy comparison with vs. without intrinsic rewards.

Qwen2.5-7B-Instruct		Naive	Non-Parametric	Offline RL	Online RL
Topic	Task		Reflexion	Retrospex	GRPO	EMPO 2
Chem istry	chemistry-mix	$-$ 42.0 $\pm$ 38.0	1.2 $\pm$ 0.7	20.8 $\pm$ 10.0	12.4 $\pm$ 3.5	42.7 $\pm$ 12.4
	chemistry-mix-paint-secondary-color	$-$ 33.0 $\pm$ 47.1	0.0 $\pm 0.0$	27.8 $\pm$ 6.3	7.1 $\pm$ 2.8	33.3 $\pm$ 0.6
	chemistry-mix-paint-tertiary-color	$-$ 33.9 $\pm$ 44.3	36.9 $\pm$ 5.7	7.6 $\pm$ 4.2	42.6 $\pm$ 6.2	39.2 $\pm$ 8.7
Classi fication	find-animal	$-$ 58.2 $\pm$ 50.2	39.5 $\pm$ 5.8	25.9 $\pm$ 13.5	72.4 $\pm$ 6.8	100.0 $\pm$ 0.0
	find-living-thing	$-$ 65.1 $\pm$ 48.1	36.6 $\pm$ 6.1	20.6 $\pm$ 4.8	68.7 $\pm$ 7.1	100.0 $\pm$ 0.0

Table 1: Comparison results of ScienceWorld. Each task in ScienceWorld contains multiple variants. We use the first five variants for training and evaluate on the 20 unseen test variants. Bold shows the best performance per task, while red shading marks cases where parametric updates score lower than non-parametric updates. The EMPO 2 performance we evaluate is the performance of the trained model without memory at test time.

Figure 8: Comparison of GRPO and EMPO 2 adapting to new tasks. Step 0 has no memory, while later steps use accumulated memory as in EMPO 2 training.

Qwen2.5-7B-Instruct	Naive	Non-Parametric	Offline RL	Online RL
Qwen2.5-7B-Instruct	Naive	Reflexion	Retrospex	GRPO	GiGPO w/ std	GiGPO w/o std	EMPO 2
Score	26.4	58.1	73.1 $\pm$ 4.1	79.3 $\pm$ 2.8	84.4 $\pm$ 2.9	86.2 $\pm$ 2.6	88.3 $\pm$ 2.6
Succ.	7.8	28.8	60.4 $\pm$ 3.9	66.1 $\pm$ 3.7	72.8 $\pm$ 3.2	75.2 $\pm$ 3.8	76.9 $\pm$ 4.1

Table 2: Comparison results of WebShop. Following Feng et al. (2025b) , we average results over three random seeds and report both the mean score and the mean success rate (%).

\text{GiGPO}_{\text{w/ std}}

denotes the use of the normalization factor

F_{\text{norm}}=\text{std}

, whereas

\text{GiGPO}_{\text{w/o std}}

uses

F_{\text{norm}}=1

, as specified in Feng et al. (2025b) . The EMPO 2 performance we evaluate is the performance of the trained model without memory at test time.

Figure 9: Comparison of training curves between EMPO 2 and variants that exclude either off-policy learning or on-policy learning with memory.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Update Mode	Rollout Condition	Current Log Prob	Old Log Prob
Regular On-Policy	No tips	$\log\pi_{\theta}(a_{t}\mid s_{t},u)$	$\log\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t},u)$
On-Policy w/ Tips	With $\fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t}$	$\log\pi_{\theta}(a_{t}\mid s_{t},u,\fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t})$	$\log\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t},u,\fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t})$
Off-Policy	With $\fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t}(\text{rollout})$	$\log\pi_{\theta}(a_{t}\mid s_{t},u)$	$\log\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t},u,\fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t})$

Appendix

Table 3: Calculation of the importance sampling ratio

\rho_{\theta}

for different policy update modes. The ratio is computed as

\rho_{\theta}=\frac{\pi_{\theta}(a_{t}\mid\cdot)}{\pi_{\theta_{\text{old}}}(a_{t}\mid\cdot)}

, which in practice is often calculated using log-probabilities.

Panel of Figure 10 — Figure 10: (a) EMPO² learning curves with varying $p$ , (b) with varying $q$

Figure 11: EMPO 2 learning curves with different intrinsic reward configurations on ScienceWorld chemistry-mix-paint-secondary-color task. We compare our full method against four variants: scaling the intrinsic reward coefficient by 0.5× and 2×, substituting it with a Random Network Distillation (RND) bonus, and its complete removal (w/o Intrinsic Reward).

Figure 12: A breakdown of the time each component spends during the rollout of each training step.

Figure 13: Time–performance curves for EMPO 2 and GRPO on ScienceWorld power-component task.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 570 papers

Expanding LLM Agent Boundaries with Strategy-Guided Exploration
Mar 2, 2026Andrew Szot, Michael Kirchhof, Omar Attia +1RL Exploration LLM Agents
Apple
Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Mar 20, 2026Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2RL for Language Model Reasoning RL Exploration
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences
Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
Oct 16, 2025Shiqi Chen, Tongyao Zhu, Zian Wang +8RL Exploration Policy Learning
City University of Hong Kong · National University of Singapore · The Hong Kong Polytechnic University +5

Mar 2, 2026J/K move · Enter open · S save

Expanding LLM Agent Boundaries with Strategy-Guided Exploration

Andrew Szot, Michael Kirchhof, Omar Attia, Alexander Toshev

Apple

RL Exploration LLM Agents LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.

Abstract

RL for LLM Agents

Explore similar work

Expanding LLM Agent Boundaries with Strategy-Guided Exploration

Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs

Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective