Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO2 as a promising framework for building more exploratory and generalizable LLM-based agents.
Figures & tables
Figure 1: (a) Comparison of the learning curves of GRPO and EMPO 2 (ours) on the ScienceWorld power-component task. While GRPO converges to suboptimal performance, EMPO 2 continues to improve and accomplish the task. (b) Comparison of EMPO 2 and other baselines in in-distribution (ID) and out-of-distribution (OOD) settings on and WebShop. In ID experiments, it adapts well to familiar environments, achieving 128.6% on ScienceWorld and 11.3% on Webshop improvements over GRPO. In OOD experiments, it also shows strong performance with few trials and no weight updates, indicating effective use of memory to explore unfamiliar environments. Full results are in Tables 1 , 2 , and Figure 8 .
Figure 2: Non-parametric updates can encourage exploration, bootstrapping parametric updates.
Figure 3: When training LLM with GRPO in ScienceWorld, the agent struggles because of insufficient exploration. For instance, in the task “turn on the red light bulb,” the agent must first find the red light bulb before activating it. However, the agent fails to locate it and, as a result, cannot complete the task. Rather than analyzing the cause of failure and exploring alternative actions, the agent proceeds unchanged, so its score stagnates even as additional training steps are taken.
Figure 4: In EMPO 2 , the current policy parameters πθ are used to review past rollouts, with the resulting insights added to memory. This updated memory conditions subsequent rollouts and promotes exploration.
Figure 5: EMPO 2 mode combinations. By combining the two rollout modes and update modes, three EMPO mode configurations are possible: on-policy learning without memory, on-policy learning with memory and off-policy learning.
Figure 6: Masking tokens stabilizes training.
Figure 7: Policy entropy comparison with vs. without intrinsic rewards.
Qwen2.5-7B-Instruct
Naive
Non-Parametric
Offline RL
Online RL
Topic
Task
Reflexion
Retrospex
GRPO
EMPO 2
Chem istry
chemistry-mix
− 42.0 ± 38.0
1.2 ± 0.7
20.8 ± 10.0
12.4 ± 3.5
42.7 ± 12.4
chemistry-mix-paint-secondary-color
− 33.0 ± 47.1
0.0 ±0.0
27.8 ± 6.3
7.1 ± 2.8
33.3 ± 0.6
chemistry-mix-paint-tertiary-color
− 33.9 ± 44.3
36.9 ± 5.7
7.6 ± 4.2
42.6 ± 6.2
39.2 ± 8.7
Classi fication
find-animal
− 58.2 ± 50.2
39.5 ± 5.8
25.9 ± 13.5
72.4 ± 6.8
100.0 ± 0.0
find-living-thing
− 65.1 ± 48.1
36.6 ± 6.1
20.6 ± 4.8
68.7 ± 7.1
100.0 ± 0.0
Table 1: Comparison results of ScienceWorld. Each task in ScienceWorld contains multiple variants. We use the first five variants for training and evaluate on the 20 unseen test variants. Bold shows the best performance per task, while red shading marks cases where parametric updates score lower than non-parametric updates. The EMPO 2 performance we evaluate is the performance of the trained model without memory at test time.
Figure 8: Comparison of GRPO and EMPO 2 adapting to new tasks. Step 0 has no memory, while later steps use accumulated memory as in EMPO 2 training.
Qwen2.5-7B-Instruct
Naive
Non-Parametric
Offline RL
Online RL
Reflexion
Retrospex
GRPO
GiGPO w/ std
GiGPO w/o std
EMPO 2
Score
26.4
58.1
73.1 ± 4.1
79.3 ± 2.8
84.4 ± 2.9
86.2 ± 2.6
88.3 ± 2.6
Succ.
7.8
28.8
60.4 ± 3.9
66.1 ± 3.7
72.8 ± 3.2
75.2 ± 3.8
76.9 ± 4.1
Table 2: Comparison results of WebShop. Following Feng et al. (2025b) , we average results over three random seeds and report both the mean score and the mean success rate (%). GiGPOw/ std denotes the use of the normalization factor Fnorm=std , whereas GiGPOw/o std uses Fnorm=1 , as specified in Feng et al. (2025b) . The EMPO 2 performance we evaluate is the performance of the trained model without memory at test time.
Figure 9: Comparison of training curves between EMPO 2 and variants that exclude either off-policy learning or on-policy learning with memory.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Update Mode
Rollout Condition
Current Log Prob
Old Log Prob
Regular On-Policy
No tips
logπθ(at∣st,u)
logπθold(at∣st,u)
On-Policy w/ Tips
With \fcolorbox{empo2}{white}{\textcolor{empo2}{tips}}_{t}
Table 3: Calculation of the importance sampling ratio ρθ for different policy update modes. The ratio is computed as ρθ=πθold(at∣⋅)πθ(at∣⋅) , which in practice is often calculated using log-probabilities.
Figure 10: (a) EMPO² learning curves with varying p , (b) with varying q
Figure 11: EMPO 2 learning curves with different intrinsic reward configurations on ScienceWorld chemistry-mix-paint-secondary-color task. We compare our full method against four variants: scaling the intrinsic reward coefficient by 0.5× and 2×, substituting it with a Random Network Distillation (RND) bonus, and its complete removal (w/o Intrinsic Reward).
Figure 12: A breakdown of the time each component spends during the rollout of each training step.
Figure 13: Time–performance curves for EMPO 2 and GRPO on ScienceWorld power-component task.
Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.
Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time. Our code is available at https://github.com/sikelifei/HeRL.
Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences
Large Language Models (LLMs) as agents often fail to improve in new environments. We identify and characterize a failure mode we call exploration collapse: under reinforcement learning (RL) in environments whose states are unfamiliar to the policy, Pass@k, the probability that at least one of k sampled trajectories succeeds, drops markedly over training even as Pass@1 edges up, revealing increasingly brittle exploration; environments closer to the pretraining distribution show no such decline. We trace this collapse to weak grounding in environment states and dynamics, and study a simple remedy: explicitly teaching the agent to estimate the current state and predict its transitions before optimizing for reward. We instantiate it as SPA, an explore-then-exploit recipe that cold-starts the policy with a Self-Experience supervised finetuning (SFT) stage, collecting the model's own interaction trajectories and supervising state and next-state prediction, and then runs standard RL. The resulting world model serves as a grounded initialization for RL rather than an inference-time planner. Across unseen environments, SPA consistently and substantially improves over vanilla RL: for example, it raises the Sokoban success rate from 25.6% to 59.8% on Qwen2.5-1.5B-Instruct, letting sub-3B models surpass a 20B baseline on these tasks. Controlled studies indicate that the gains track four factors: grounded state representations, explicit transition modeling, self-experience trajectories from a sufficiently strong exploration policy, and adequate coverage of transition data.
Shiqi Chen, Tongyao Zhu, Zian Wang +8
City University of Hong Kong · National University of Singapore · The Hong Kong Polytechnic University +5