Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.
Figures & tables
Figure 1: Overview of Strategy-Guided Exploration (SGE). SGE improves reinforcement learning (RL) training on hard agentic tasks for which the base model fails, even after many attempts (left of figure). SGE addresses this by having the LLM policy output a language “strategy” and then conditioning the action generation on this strategy. The reasoning capabilities of the LLM enable it to output diverse strategies through the techniques of: (1) mixed-temperature sampling, where strategy tokens are sampled with higher temperature than remaining tokens, and (2) strategy reflection, where strategies are generated to be distinct from other strategies executed earlier in RL training. This enables SGE to explore to solve hard tasks that the base model is not capable of succeeding in (right of figure).
Figure 2: RL training curves for SGE and baselines in each environment. Training curves show the average pass@1 rate versus the number of RL updates for the training tasks across 3 random seeds in training runs, with the shaded area representing the standard deviation of the pass@1 rate across the seeds. The horizontal lines indicate the base model pass@ k rate with lighter shades meaning a higher k . In the Coding environment, SGE and GRPO are trained longer than the other approaches to show better converged performance. RND and RLAD baselines are also only shown for this environment since they underperform the GRPO and EntropyAdv.
Figure 3: Comparing the pass@ k of the final policies trained by SGE and GRPO and the starting base model across the environments on the training tasks. We compute the pass@ k for k in powers of 2, and the x-axis is on a logarithmic scale. The “Base” line shows that the pass@ k of the base model eventually plateaus for large enough k . GRPO raises the “Base” curve, enabling the pass@1 to closely match the pass@ k for the highest k . On the other hand, SGE exceeds this pass@ k ceiling of the base model and enables solving new problems that are unsolvable by the base model. We report averages across all tasks in the training set.
Zero-Shot
GRPO
SGE
Coding
13.5
22.0 ± 0.3
29.2 ± 0.7
AndroidWorld
16.7
21.9 ± 0.3
36.7 ± 1.3
LangR
42.6
46.0 ± 0.4
60.8 ± 0.6
AppWorld
47.8
49.3 ± 5.0
66.6 ± 2.9
Table 1: Test evaluation on unseen tasks of SGE versus baselines after RL training. Numbers are average and standard error pass@1 across the 3 random seeds on unseen tasks not included in RL training. All coding checkpoints are evaluated after 1k updates. Zero-Shot refers to evaluating the base-LLM directly on the test set without any RL training. RL improves generalization, and SGE further boosts it.
Figure 4: Left: we ablate the parallel strategy sampling by comparing to sampling all tokens with the uniformly high temperature used to sample the strategy tokens or the LLM default temperature. Middle we ablate the sequential strategy reflection by removing SGE’s ability to reflect on successful strategies, negative strategies, or any strategy. Right: Effect of scaling Qwen3 base model parameter count on RL training with and without SGE. Ablation results are over 3 random seeds, and scaling results are over 1 random seed.
Figure 5: Left: Comparing the effect of different mixed-temperature sampling settings on the zero-shot pass@16 performance in the Coding environment. The high-values in the top-left indicate that a mixed-temperature sampling produces the best results. Right: Visualizing actions predicted by SGE vs. regular sampling in the MarkorCreateNote task in AndroidWorld. 16 actions are sampled for the same observation screenshot with the tap locations visualized as colored circles overlaid on the screen (non-tap actions are not visualized). To complete the task, the agent must tap the file extension dropdown. Unlike with regular sampling, SGE is able to sample this correct action.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Code
AndroidWorld
LangR
AppWorld
Learning Rate
1e-6
1e-6
1e-6
1e-5
Group Size
16
6
16
8
Global Update Batch Size
128
64
128
128
# Output Tokens / Step
2048
564
520
768
Clip Param
0.2
0.2
0.2
0.2
Trajectories Collected Per Update
1024
192
512
128
Appendix
Table 2: RL hyperparameters across domains. Unless otherwise specified, these values are shared between all approaches. SGE prefix means the value only applies to SGE, and not the baselines. # Output Tokens / Step refers to the total token budget per LLM output for each step in the environment. This multiplies the number of steps in an episode by the total number of tokens produced in an episode. The Global Update Batch Size is in terms of the number of steps, not the number of trajectories, which consist of multiple steps. Group Size is the number of trajectories used to estimate the advantage in GRPO. Trajectories per update is the number of samples collected in the environment for each update. Token Temperature is the temperature used to generate the non-strategy tokens.
Figure 6: We report the impact of SGE on the zero-shot performance of the policy in the Coding environment and compare to the base model and SGE RL trained model. We evaluate in the same setup as Figure 3 .
Figure 7: Impact of prompting on RL training in Coding environment. Regular GRPO training is with 3 random seeds. Training without the strategy prompt is only with a single random seed.
Figure 8: Two strategies generated by SGE for the same task for the first step in the Coding environment. The text in “<…>" is for visualization and not part of the actual response. Where specified, some of the response is omitted for clarity. The strategies are under <Strategy 1> , <Strategy 2> and are independently generated and are only sequentially shown for visualization. The two strategies follow different approaches to solving the same task, with strategy 1 directly computing a frequency histogram via enumeration and strategy 2 using binary search and a sliding window. Each strategy is followed by by additional textual reasoning and Python code actually implementing the strategy.
Figure 9: Qualitative example in the Coding environment of how SGE uses negative strategy reflection to improve the diversity of generated strategies. The text in “<…>" is for visualization and not part of the actual response. Where specified, some of the response is omitted for clarity. The agent reflects on a previously executed strategy that resulted in a failed outcome from earlier in RL training with the negative strategy reflection prompt. The original strategy incorrectly missed a corner case by assuming the meeting point must be strictly after both positions. But this misses the case where the rightmost person’s building is itself a valid meeting point. The agent reflects on this strategy, sees the error, and then addresses the shortcomings of the old strategy with a new strategy.
Figure 10: Comparing the effect of the sampling temperature on the strategy and remaining tokens in terms of pass@1. Green indicates higher performance.
Figure 11: The cumulative unique program outcomes per task in the coding environment during RL training, where each distinct set of tests that passes is considered a distinct outcome. SGE achieves more outcomes faster during RL training compared to standard GRPO, demonstrating its effectiveness in exploration.
Exploration remains the key bottleneck for large language model agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory Memory-Augmented On- and Off-Policy Optimization (EMPO2), a hybrid RL framework that leverages memory for exploration and combines on- and off-policy updates to make LLMs perform well with memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO2 achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO2 demonstrates superior adaptability to new tasks, requiring only a few trials with memory and no parameter updates. These results highlight EMPO2 as a promising framework for building more exploratory and generalizable LLM-based agents.
Large Language Model (LLM) agents have demonstrated remarkable proficiency in learned tasks, yet they often struggle to adapt to non-stationary environments with feedback. While In-Context Learning and external memory offer some flexibility, they fail to internalize the adaptive ability required for long-term improvement. Meta-Reinforcement Learning (meta-RL) provides an alternative by embedding the learning process directly within the model. However, existing meta-RL approaches for LLMs focus primarily on exploration in single-agent settings, neglecting the strategic exploitation necessary for multi-agent environments. We propose MAGE, a meta-RL framework that empowers LLM agents for strategic exploration and exploitation. MAGE utilizes a multi-episode training regime where interaction histories and reflections are integrated into the context window. By using the final episode reward as the objective, MAGE incentivizes the agent to refine its strategy based on past experiences. We further combine population-based training with an agent-specific advantage normalization technique to enrich agent diversity and ensure stable learning. Experiment results show that MAGE outperforms existing baselines in both exploration and exploitation tasks. Furthermore, MAGE exhibits strong generalization to unseen opponents, suggesting it has internalized the ability for strategic exploration and exploitation. Code is available at https://github.com/Lu-Yang666/MAGE.
Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time. Our code is available at https://github.com/sikelifei/HeRL.
Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences