Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

Lei Huang; Xiang Cheng; Chenxiao Zhao; Guobin Shen; Junjie Yang; Xiaocheng Feng; Yuxuan Gu; Xing Yu

cs.CLMar 4, 2026

Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

Authors: Lei Huang, Xiang Cheng, Chenxiao Zhao, Guobin Shen, Junjie Yang, Xiaocheng Feng, Yuxuan Gu, Xing Yu,

Organizations: Harbin Institute of Technology · Xiaohongshu Inc.

Abstract

Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2 $\times$ improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.

Figures & tables

Figure 1: An illustration of RL with natural language feedabck. Compared with scalar-reward RL (top), aggregating intra-group and external feedback turns sparse outcomes into actionable refinement signals, enabling guided exploration (bottom).

Figure 2: An overview of Golf , which consists of three components. The policy first rollouts a group of candidates and receives both scalar rewards and external critiques. Golf then aggregates the critiques with the failed trajectories in the same group to form group-level NL feedback, which conditions a refinement stage to produce improved responses. Finally, high-quality refinements are adaptively injected back into the rollout group as off-policy guidance, mitigating low-reward regimes. Both generation and refinement are optimized jointly within a unified RL loop.

Model	AlpacaEval-v2		WildBench	Arena-Hard-v1	ArenaHard-v2	CreativeWriting-v3	Average
Model	Win Rate (%)	LC Win Rate (%)	LLM Judge (%)	Win Rate (%)	Win Rate (%)	LLM Judge (%)	Score (%)
Llama-3.1-8B-Instruct	31.93	31.79	-8.25	30.80	5.57	53.96	24.30
+ Direct-Likert	38.88	34.98	13.48	51.55	11.73	64.10	35.79
+ Pairwise-GRPO	45.47	43.19	25.54	49.20	13.30	62.95	39.94
+ Rubric-as-Reward	42.24	36.12	26.51	52.10	15.57	68.12	40.11
+ Critique-GRPO	47.45	43.31	25.09	50.15	13.73	65.76	40.92

Table 1: Experimental results on non-verifiable tasks. All metrics are the higher the better. Bold and underline numbers indicate the best performance and second performance among all methods, respectively. WildBench scores are in

[-100,100]

, while all other metrics are in

[0,100]

. All scores are judged by GPT-4o.

Panel of Figure 3 — Figure 3: Evaluation performance over training steps. We report the LC win rate on AlpacaEval v2.0 (left), WildBench score (middle), and ArenaHard v2.0 win rate (right). The baseline refers to Pairwise-GRPO, which uses the same generative reward model as Golf .

Panel of Figure 4 — Figure 4: Pass@ $k$ comparison between GRPO and Golf on mathematical reasoning benchmarks using Qwen-3-8B.

Model	Mathmatical Reasoning			Instruction Following
Model	AIME 24	AIME 25	AMC 23	IFBench	IFEval
Qwen-3-4B	22.53	18.55	59.41	23.67	81.52
+ Refinement-FT	31.67	21.25	64.06	30.44	83.73
+ Critique-FT	34.58	24.58	65.94	31.67	82.63
+ GRPO	42.72	35.42	76.85	33.33	84.45
+ Critique-GRPO	45.72	35.89	76.14	35.67	85.21

Table 2: Experimental results on verifiable tasks. All metrics are higher is better. Bold and underline numbers indicate the best performance and second performance among all methods.

Panel of Figure 5 — Figure 5: Ablation on feedback sources. We ablate intra-group attempts or external critiques from the aggregated refinement context. Bars report average performance over the non-verifiable, math reasoning, and instruction following suites and we provide per-benchmark results in Appendix G .

Panel of Figure 6 — Figure 6: Evaluation on LCBv6 with Qwen-3-8B . Left: Avg@4 accuracy curve over training steps. Right: Final performance comparison with SDPO ( Hübotter et al., 2026 ) under the same environment feedback setting.

Panel of Figure 7 — Figure 7: Left: Average performance comparison of different off-policy sample learning strategies on non-verifiable benchmarks; full results are provided in Appendix 10 . Right: Zero-reward ratio during training.

Model	Biology	Chemistry	Law	Average
Self-Refinement
Llama-3.1-8B-Instruct	9.09	8.33	23.94	13.79
+ GRPO	22.73	27.77	22.24	24.25
+ Golf	27.27	33.33	24.64	28.41
Guided-Refinement
Llama-3.1-8B-Instruct	18.18	27.78	49.30	31.75

Table 3: RefineBench performance under two settings: self-refinement , where the model refines its initial response, and guided-refinement , where the refinement is conditioned on the checklists.

Panel of Figure 8 — Figure 8: Policy entropy (seq-mean-token-sum-norm) of Golf and the Pairwise-GRPO baseline over training steps.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Feedback Type	pass@4 (%)	Acc (%)
Simple	0.00	0.00
Intra	18.80	6.65
External	27.60	17.00
Mixed	30.40	17.55

Appendix

Table 4: Test-time refinement results under different feedback conditions.

Category	Hyperparameter	Value
Data	Train file	WildChat-IF
	Max prompt length	8192
	Max response length	4096
	Filter overlong prompts	True
Actor Model	Base model 1	Llama-3.1-8B-Instruct
	Base model 2	Qwen-3-8B

Appendix

Table 5: Key hyperparameters used for RL training for non-verifiable tasks in the verl ( Sheng et al., 2025 ) framework.

Category	Hyperparameter	Value
Data	Train file	OpenR1-filtered
	Max prompt length	8192
	Max response length	6144
	Filter overlong prompts	True
Actor Model	Base model 1	Qwen-3-4B (non-thinking)
	Base model 2	Qwen-3-8B (non-thinking)

Appendix

Table 6: Key hyperparameters used for RL training for the mathematical reasoning task used in the verl ( Sheng et al., 2025 ) framework.

Category	Hyperparameter	Value
Data	Train file	IFTrain-filtered
	Max prompt length	8192
	Max response length	4096
	Filter overlong prompts	True
Actor Model	Base model 1	Qwen-3-4B (non-thinking)
	Base model 2	Qwen-3-8B (non-thinking)

Appendix

Table 7: Key hyperparameters used for RL training for the instruction following task used in the verl ( Sheng et al., 2025 ) framework.

Model	AlpacaEval-v2	WildBench	ArenaHard-v2	CreativeWriting-v3	Average
Model	LC Win Rate (%)	LLM Judge (%)	Win Rate (%)	LLM Judge (%)	Score (%)
Llama-3.1-8B-Instruct
Golf	69.67	34.42	25.03	66.21	50.19
w/o intra-feedback	52.70	37.30	20.60	60.94	42.89
w/o external-feedback	51.00	33.91	17.67	55.91	39.62

Appendix

Table 8: Ablation experiment results for the group-level feedback on non-verifiable tasks. Higher is better. Bold and underline numbers indicate the best and second-best performance among all methods, respectively. WildBench scores are in

[-100,100]

, while all other metrics are in

[0,100]

. All scores are judged by GPT-4o.

Model	Mathmatical Reasoning			Instruction Following
Model	AIME 24	AIME 25	AMC 23	IFBench	IFEval
Qwen-3-8B
Golf	58.49	41.65	80.74	38.33	87.80
w/o intra-feedback	53.02	40.30	82.06	36.67	86.88
w/o external-feedback	55.64	39.37	79.43	35.67	85.95

Appendix

Table 9: Experimental results on verifiable tasks. All metrics are higher is better. Bold and underline numbers indicate the best performance and second performance among all methods.

Model	AlpacaEval-v2	WildBench	ArenaHard-v1	ArenaHard-v2	CreativeWriting-v3	Average
Model	LC Win Rate (%)	LLM Judge (%)	Win Rate (%)	Win Rate (%)	LLM Judge (%)	Score (%)
Llama-3.1-8B-Instruct
+ Golf (w/ mixed-policy RL)	69.67	34.42	52.40	25.03	66.21	49.55
+ Golf (w/ sft, coef = 0.1)	39.03	34.62	46.75	15.87	44.41	36.14

Appendix

Table 10: Ablation experiment results for the mixed-policy optimization. Higher is better. Bold numbers indicates the best performance among all methods. WildBench scores are in

[-100,100]

, while all other metrics are in

[0,100]

. All scores are judged by GPT-4o.

Model	AlpacaEval-v2	WildBench	ArenaHard-v1	ArenaHard-v2	CreativeWriting-v3	Average
Model	LC Win Rate (%)	LLM Judge (%)	Win Rate (%)	Win Rate (%)	LLM Judge (%)	Score (%)
Llama-3.1-8B-Instruct
+ Golf (w/ adaptive)	69.67	34.42	52.40	25.03	66.21	49.55
+ Golf (w/o adaptive)	40.73	23.78	45.80	15.80	53.84	35.99

Appendix

Table 11: Ablation experiment results for the adaptive guidance mechanism. Higher is better. Bold numbers indicate the best performance among all methods. WildBench scores are in

[-100,100]

, while all other metrics are in

[0,100]

. All scores are judged by GPT-4o.

Model	AlpacaEval-v2	WildBench	ArenaHard-v1	ArenaHard-v2	CreativeWriting-v3	Average
Model	LC Win Rate (%)	LLM Judge (%)	Win Rate (%)	Win Rate (%)	LLM Judge (%)	Score (%)
Llama-3.1-8B-Instruct
+ Golf (rollout=8)	69.67	34.42	52.40	25.03	66.21	49.55
+ Pairwise-GRPO (rollout=8)	43.19	25.54	49.20	13.30	62.95	38.84
+ Pairwise-GRPO (rollout=16)	46.66	28.62	51.80	14.10	64.21	41.08

Appendix

Table 12: Ablation experiment results for efficiency. Higher is better. Bold numbers indicate the best performance among all methods. WildBench scores are in

[-100,100]

, while all other metrics are in

[0,100]

. All scores are judged by GPT-4o.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 213 papers

Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
Mar 20, 2026Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi +2RL for Language Model Reasoning RL Exploration
Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences
Efficient Exploration at Scale
Mar 18, 2026Seyed Mohammad Asghari, Chris Chute, Vikranth Dwaracherla +5RL from Human Feedback Data-Efficient Learning
The Efficient Agent Team, Google DeepMind
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
Sep 27, 2026Zihan Lin, Xiaohan Wang, Jie Cao +4RL for Language Models RL for Language Model Reasoning
University of Chinese Academy of Sciences · Meituan · MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences

Mar 20, 2026J/K move · Enter open · S save

Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs

Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi, Baisheng Lai+1

Dalian University of Technology · Zhejiang University · Chinese Academy of Sciences

RL for Language Model Reasoning RL Exploration Rubric-Based RL In-Context RL Reward Design for RL

Open paper AskPDF arXiv

Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to curent policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time. Our code is available at https://github.com/sikelifei/HeRL.

Abstract

RL Exploration

Explore similar work

Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs

Efficient Exploration at Scale

TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs