Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time. Our code is available at https://github.com/sikelifei/HeRL.
Figures & tables
Figure 1: (Left) A conceptual illustration of the efficiency of exploration and the effectiveness of experience guided sampling. (Right) Model performance comparison between baselines and our proposed HeRL across different reasoning domains.
Figure 2: Performance comparison of sampling strategies. The guided sampling by hindsight experience consistently outperforms stochastic sampling and entropy-based sampling.
Figure 3: The overall framework of HeRL. First, we sample candidate trajectories and evaluate them using checklist-style rubrics. Then we revise failed trajectories with highest reward and preserve the best improvements. Both the original attempts and subsequent improvements are optimized using reinforcement learning, supplemented by a bonus reward to incentive responses with higher improvement potential.
Method
Instruction Following
Writing ∗
Medical
IFEval
IFBench
MulDimIF
WritingBench
LLMEval-Med
HealthBench-500
Qwen2.5-7B-Instruct
72.6 ↑ 0.0
26.2 ↑ 0.0
51.4 ↑ 0.0
57.0 ↑ 0.0
56.0 ↑ 0.0
24.4 ↑ 0.0
+ SFT
75.6 ↑ 3.0
27.9 ↑ 1.7
67.8 ↑ 16.4
51.5 ↓ 5.5
34.8 ↓ 21.2
27.2 ↑ 2.8
+ DPO
66.9 ↓ 5.7
25.9 ↓ 0.3
56.5 ↑ 5.1
52.1 ↓ 4.9
35.8 ↓ 20.2
28.0 ↑ 3.6
+ RLVR
77.3 ↑ 4.7
31.6 ↑ 5.4
73.5 ↑ 22.1
54.8 ↓ 2.2
60.5 ↑ 4.5
30.5 ↑ 6.1
+ HeRL (Ours)
82.4 ↑ 9.8
39.7 ↑ 13.5
83.4 ↑ 32.0
59.1 ↑ 2.1
65.0 ↑ 9.0
34.3 ↑ 9.9
Table 1: Experimental results ( % ) across diverse domains using different LLMs trained on the same data. Best results among all methods are marked as bolded , and arrows indicate improvement or degradation over the initial model. Note Writing* tasks are not in training data.
Model
MATH-500
GPQA
MMLU-Pro
Llama-3.2-3B-Instruct
38.6
26.5
33.9
+ HeRL (Ours)
36.6 ↓ 2.0
29.2 ↑ 2.7
34.8 ↑ 0.9
Qwen2.5-7B-Instruct
76.0
34.1
57.3
+ HeRL (Ours)
77.6 ↑ 1.6
33.9 ↓ 0.2
55.5 ↓ 1.8
Qwen3-4B-Instruct-2507
86.0
46.6
61.5
+ HeRL (Ours)
89.8 ↑ 3.8
45.3 ↓ 1.3
62.1 ↑ 0.6
Table 2: Performance of HeRL on out-of-domain benchmarks.
Figure 4: Investigation of HeRL’s sampling efficiency. (a) Pass@k performance of Qwen2.5-7B-Instruct, RLVR, and HeRL on IFBench. (b) Iterative revision with experience guidance further outperforms Pass@k for both RLVR and HeRL on HealthBench-500 with Qwen3-4B-Instruct-2507.
Setting
IFBench
WritingBench
HealthBench-500
Model I: Qwen2.5-7B-Instruct
Baseline (RLVR)
31.6 ↑ 0.0
54.8 ↑ 0.0
30.5 ↑ 0.0
+ NaiveHE
32.3 ↑ 0.7
52.7 ↓ 2.1
28.8 ↓ 1.7
+ HE
36.7 ↑ 5.1
58.9 ↑ 4.1
31.8 ↑ 1.3
+ HE + BR( HeRL )
39.7 ↑ 8.1
59.1 ↑ 4.3
34.3 ↑ 3.8
Model II: Llama-3.2-3B-Instruct
Table 3: Ablation study of HeRL, where NaiveHE denotes naive Hindsight Experience, HE refers to Hindsight Experience, and BR is Bonus Reward. The arrows indicate the performance gains.
Figure 5: Training dynamics of different models. (Top) Entropy, (Mid) Reward and (Bottom) Validation Reward curves over training steps on the RAR-Medicine dataset are reported for three base models, comparing HeRL with RLVR baseline. Model names are abbreviated in the plots: Qwen2.5 denotes Qwen2.5-7B-Instruct, Qwen3 denotes Qwen3-4B-Instruct-2504, and Llama denotes Llama3.2-3B-Instruct.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 4: Training configurations across different methods and model backbones.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Zishang Jiang, Jinyi Han, Tingyun Li +7
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2× improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.