cs.LGMar 2, 2026
SaveExpanding LLM Agent Boundaries with Strategy-Guided Exploration
Organizations: Apple
Abstract
Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.
Figures & tables
Figure 1: Overview of Strategy-Guided Exploration (SGE). SGE improves reinforcement learning (RL) training on hard agentic tasks for which the base model fails, even after many attempts (left of figure). SGE addresses this by having the LLM policy output a language “strategy” and then conditioning the action generation on this strategy. The reasoning capabilities of the LLM enable it to output diverse strategies through the techniques of: (1) mixed-temperature sampling, where strategy tokens are sampled with higher temperature than remaining tokens, and (2) strategy reflection, where strategies are generated to be distinct from other strategies executed earlier in RL training. This enables SGE to explore to solve hard tasks that the base model is not capable of succeeding in (right of figure).
Figure 2: RL training curves for SGE and baselines in each environment. Training curves show the average pass@1 rate versus the number of RL updates for the training tasks across 3 random seeds in training runs, with the shaded area representing the standard deviation of the pass@1 rate across the seeds. The horizontal lines indicate the base model pass@ rate with lighter shades meaning a higher . In the Coding environment, SGE and GRPO are trained longer than the other approaches to show better converged performance. RND and RLAD baselines are also only shown for this environment since they underperform the GRPO and EntropyAdv.
Figure 3: Comparing the pass@ of the final policies trained by SGE and GRPO and the starting base model across the environments on the training tasks. We compute the pass@ for in powers of 2, and the x-axis is on a logarithmic scale. The “Base” line shows that the pass@ of the base model eventually plateaus for large enough . GRPO raises the “Base” curve, enabling the pass@1 to closely match the pass@ for the highest . On the other hand, SGE exceeds this pass@ ceiling of the base model and enables solving new problems that are unsolvable by the base model. We report averages across all tasks in the training set.
| Zero-Shot | GRPO | SGE | |
| Coding | 13.5 | 22.0 0.3 | 29.2 0.7 |
| AndroidWorld | 16.7 | 21.9 0.3 | 36.7 1.3 |
| LangR | 42.6 | 46.0 0.4 | 60.8 0.6 |
| AppWorld | 47.8 | 49.3 5.0 | 66.6 2.9 |
Table 1: Test evaluation on unseen tasks of SGE versus baselines after RL training. Numbers are average and standard error pass@1 across the 3 random seeds on unseen tasks not included in RL training. All coding checkpoints are evaluated after 1k updates. Zero-Shot refers to evaluating the base-LLM directly on the test set without any RL training. RL improves generalization, and SGE further boosts it.
Figure 4: Left: we ablate the parallel strategy sampling by comparing to sampling all tokens with the uniformly high temperature used to sample the strategy tokens or the LLM default temperature. Middle we ablate the sequential strategy reflection by removing SGE’s ability to reflect on successful strategies, negative strategies, or any strategy. Right: Effect of scaling Qwen3 base model parameter count on RL training with and without SGE. Ablation results are over 3 random seeds, and scaling results are over 1 random seed.
Figure 5: Left: Comparing the effect of different mixed-temperature sampling settings on the zero-shot pass@16 performance in the Coding environment. The high-values in the top-left indicate that a mixed-temperature sampling produces the best results. Right: Visualizing actions predicted by SGE vs. regular sampling in the MarkorCreateNote task in AndroidWorld. 16 actions are sampled for the same observation screenshot with the tap locations visualized as colored circles overlaid on the screen (non-tap actions are not visualized). To complete the task, the agent must tap the file extension dropdown. Unlike with regular sampling, SGE is able to sample this correct action.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Code | AndroidWorld | LangR | AppWorld | |
| Learning Rate | 1e-6 | 1e-6 | 1e-6 | 1e-5 |
| Group Size | 16 | 6 | 16 | 8 |
| Global Update Batch Size | 128 | 64 | 128 | 128 |
| # Output Tokens / Step | 2048 | 564 | 520 | 768 |
| Clip Param | 0.2 | 0.2 | 0.2 | 0.2 |
| Trajectories Collected Per Update | 1024 | 192 | 512 | 128 |
Table 2: RL hyperparameters across domains. Unless otherwise specified, these values are shared between all approaches. SGE prefix means the value only applies to SGE, and not the baselines. # Output Tokens / Step refers to the total token budget per LLM output for each step in the environment. This multiplies the number of steps in an episode by the total number of tokens produced in an episode. The Global Update Batch Size is in terms of the number of steps, not the number of trajectories, which consist of multiple steps. Group Size is the number of trajectories used to estimate the advantage in GRPO. Trajectories per update is the number of samples collected in the environment for each update. Token Temperature is the temperature used to generate the non-strategy tokens.
Figure 6: We report the impact of SGE on the zero-shot performance of the policy in the Coding environment and compare to the base model and SGE RL trained model. We evaluate in the same setup as Figure 3 .
Figure 7: Impact of prompting on RL training in Coding environment. Regular GRPO training is with 3 random seeds. Training without the strategy prompt is only with a single random seed.
Figure 8: Two strategies generated by SGE for the same task for the first step in the Coding environment. The text in “<…>" is for visualization and not part of the actual response. Where specified, some of the response is omitted for clarity. The strategies are under <Strategy 1> , <Strategy 2> and are independently generated and are only sequentially shown for visualization. The two strategies follow different approaches to solving the same task, with strategy 1 directly computing a frequency histogram via enumeration and strategy 2 using binary search and a sliding window. Each strategy is followed by by additional textual reasoning and Python code actually implementing the strategy.
Figure 9: Qualitative example in the Coding environment of how SGE uses negative strategy reflection to improve the diversity of generated strategies. The text in “<…>" is for visualization and not part of the actual response. Where specified, some of the response is omitted for clarity. The agent reflects on a previously executed strategy that resulted in a failed outcome from earlier in RL training with the negative strategy reflection prompt. The original strategy incorrectly missed a corner case by assuming the meeting point must be strictly after both positions. But this misses the case where the rightmost person’s building is itself a valid meeting point. The agent reflects on this strategy, sees the error, and then addresses the shortcomings of the old strategy with a new strategy.
Figure 10: Comparing the effect of the sampling temperature on the strategy and remaining tokens in terms of pass@1. Green indicates higher performance.
Figure 11: The cumulative unique program outcomes per task in the coding environment during RL training, where each distinct set of tests that passes is considered a distinct outcome. SGE achieves more outcomes faster during RL training compared to standard GRPO, demonstrating its effectiveness in exploration.
More in this topic
RL Exploration
Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.
See all 213 papers