TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

Zihan Lin; Xiaohan Wang; Jie Cao; Jiajun Chai; Wei Lin; Guojun Yin; Ran He

cs.LGSep 27, 2026

TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Wei Lin, Guojun Yin, Ran He

Organizations: University of Chinese Academy of Sciences · Meituan · MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences

Abstract

Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as token-level credit using Jensen--Shannon (JS) divergence between the corresponding temperature-scaled next-token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11 benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.

Figures & tables

Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
Qwen3-14B	50.4	40.0	79.8	87.9	46.1	60.3	60.8
PPO	55.2	45.8	90.6	93.9	49.1	63.5	66.4
GRPO	57.5	45.4	92.9	94.4	49.7	62.2	67.0
DAPO	61.0	46.0	92.8	94.1	49.8	64.8	68.0
Dr.GRPO	61.2	44.5	91.7	94.2	49.2	64.5	67.5
RLOO	59.8	47.9	91.2	93.9	48.6	62.3	67.3

Table 1: Avg@16 results on mathematical reasoning benchmarks with Qwen3 variants ( Yang et al., 2025a ) trained in think mode. Best results are in bold , second-best results are underlined .

Method	LiveCodeBench		CodeForces		HumanEval+
Method	Avg@16	Pass@16	Rating	Percentile	Pass@16
Qwen3-4B	30.5	40.9	578.8	1.2	89.0
PPO	39.7 (+9.2)	57.7 (+16.8)	1377.6 (+798.8)	71.9 (+70.7)	95.1 (+6.1)
GRPO	39.5 (+9.0)	55.1 (+14.2)	1267.9 (+689.1)	63.1 (+61.9)	95.7 (+6.7)
DAPO	41.0 (+10.5)	52.3 (+11.4)	1112.5 (+533.7)	46.7 (+45.5)	95.7 (+6.7)
Dr.GRPO	41.6 (+11.1)	57.7 (+16.8)	1195.0 (+616.2)	55.3 (+54.1)	95.7 (+6.7)

Table 2: Performance on code-generation benchmarks using Qwen3-4B. We report LiveCodeBench Avg@16 and Pass@16, CodeForces Rating and Percentile, and HumanEval+ Pass@16.

Method	ALFWorld							WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Task Score	Succ.
GPT-4o ( Hurst et al., 2024 )	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
DeepSeek-V4-Pro	69.0	66.7	41.9	26.3	52.2	60.0	51.6	77.9	33.3
Prompting Backbone	33.4	21.6	19.3	6.9	2.8	3.2	14.8	26.4	7.8
Prompting ReAct	48.5	35.4	34.3	13.2	18.2	17.6	31.2	46.2	19.5
PPO	92.3	64.0	92.5	89.5	80.3	68.8	80.4	81.4	68.7

Table 3: Performance on ALFWorld and WebShop averaged over three seeds on Qwen2.5-7B-Instruct. We report Success Rate for ALFWorld and both Task Score and Success Rate for WebShop.

Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
GRPO	57.5	45.4	92.9	94.4	49.7	62.2	67.0
GRPO+Ent ( $\lambda{=}10^{-4}$ )	63.2	48.4	90.5	93.7	47.9	62.8	67.8
GRPO+Ent ( $\lambda{=}10^{-3}$ )	57.1	47.7	91.9	92.8	47.0	63.8	66.7
TAMPO	63.3	47.1	92.0	95.2	49.2	61.0	68.0
TGRL (ours)	63.4	49.4	92.7	95.3	48.9	66.4	69.4

Table 4: Exploration-oriented baselines using Qwen3-14B. “GRPO+Ent” denotes GRPO with an added entropy bonus of coefficient

\lambda

; TAMPO is a temperature-adaptive baseline.

Scale	Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
14B	GRPO @ $T_{0}$	57.0	44.6	91.9	93.6	48.9	63.0	66.5
	GRPO @ $T_{1}$	57.5	45.4	92.9	94.4	49.7	62.2	67.0
	TGRL (Uniform Token Credit)	61.0	46.9	91.3	93.4	48.8	68.1	68.2
	TGRL (ours)	63.4	49.4	92.7	95.3	48.9	66.4	69.4
32B	GRPO @ $T_{0}$	55.8	44.4	85.6	93.0	48.0	64.6	65.1
	GRPO @ $T_{1}$	60.4	44.8	88.9	92.2	48.5	62.1	66.2

Table 5: Math ablation results. Comparing GRPO @

T_{0}

and GRPO @

T_{1}

isolates single-temperature exploration. Comparing GRPO @

T_{1}

and TGRL (Uniform Token Credit) evaluates the mixed-temperature TGRL structure without JS-based credit allocation. Comparing TGRL (Uniform Token Credit) and TGRL isolates JS-based token credit allocation.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	Count	Flip rate $\uparrow$	Mean score drop $\uparrow$	Mean position ratio	Mean JS
JS	224	12.1%	0.241	0.444	0.0427
Low-Margin	224	11.6%	0.232	0.482	0.0309
Matched-Random	1120	8.7%	0.173	0.458	0.0064
Entropy	224	7.6%	0.152	0.468	0.0385

Appendix

Table 6: Local-cooling probe on the Qwen3-32B model. Each selector selects a contiguous span under a fixed edit budget within successful exploration trajectories; the span is cooled toward the low-temperature preference and generation continues from the edited prefix. Matched-Random is repeated five times per source trajectory, hence its larger count. Higher flip rate (

\uparrow

) and larger score drop (

\uparrow

) indicate greater temperature sensitivity of the selected span.

Protocol	Method	Best AIME-Comb.	Time of Best (h)	Step	Remarks
(i) Post-warmup $12$ -hour fixed-budget comparison (TGRL vs. GRPO)
TGRL vs. GRPO	TGRL (ours)	1.0833	11.01	80	$13.84$ h absolute
TGRL vs. GRPO	GRPO	1.0354	8.86	100	$11.42$ h absolute
(ii) From-start $12$ -hour fixed-budget comparison (TGRL vs. DAPO)
TGRL vs. DAPO	TGRL (ours)	1.0604	10.21	60	from training start
TGRL vs. DAPO	DAPO	0.9521	10.01	30	from training start

Appendix

Table 7: Wall-clock training efficiency on the 14B setup under identical hardware, data, and evaluation settings. “Time of Best (h)” is measured from the protocol origin: the end of TGRL’s

20

-step warmup in (i), and step

0

in (ii)–(iii). “Best AIME-Comb.” is the highest validation score reached within the budget; “Step” is the step index at which that best was attained.

Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
Qwen3-4B	49.4	35.6	78.8	84.6	40.3	58.2	57.8
GRPO	54.2	39.2	83.9	90.8	44.4	64.2	62.8
Dr.GRPO	49.0	40.0	85.6	92.7	45.1	66.4	63.1
RLOO	53.1	39.0	85.8	81.8	40.4	57.6	59.6
TAMPO	54.0	39.4	85.9	92.2	44.2	64.2	63.3
TGRL (ours)	56.2	42.7	81.4	92.5	44.9	69.0	64.5

Appendix

Table 8: Avg@16 results on mathematical reasoning benchmarks with Qwen3-4B trained in think mode. Best results are in bold , second-best results are underlined .

$T_{1}$	$\Delta T$	Peak	Peak step	Final@150	AUC@150	Step to 0.94
0.8	0.5	0.808	130	0.758	0.698	–
1.0	0.7	0.942	140	0.900	0.822	140
1.2	0.9	0.963	70	0.933	0.885	70
1.4	1.1	0.923	50	0.900	0.846	–
1.6	1.3	0.975	140	0.948	0.814	110

Appendix

Table 9: Summary of the 4B fixed-temperature sweep over the first 150 training steps.

\Delta T

denotes the gap relative to the fixed low-temperature reference

T_{0}=0.3

. AUC is computed from the validation AIME-combined curve. The results show that

T_{1}=1.2

offers the strongest early learning efficiency, whereas larger temperatures remain competitive but less sample-efficient.

Qwen3-14B
Method	AIME24	AIME25	AMC23
GRPO	$57.3\pm 3.8$	$43.2\pm 1.8$	$92.4\pm 1.2$
TGRL (ours)	$64.2\pm 4.1$	$48.7\pm 3.5$	$91.9\pm 0.4$
Qwen3-32B
GRPO	$59.5\pm 2.5$	$45.3\pm 2.6$	$87.9\pm 1.4$
TGRL (ours)	$65.3\pm 3.5$	$48.3\pm 3.3$	$91.8\pm 1.2$

Appendix

Table 10: Multi-seed Avg@16 evaluation on competition-level math benchmarks. We report mean accuracy (%)

\pm

standard deviation across 5 random seeds.

Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
GRPO ( $n{=}8$ )	60.8	46.7	92.5	94.4	48.7	64.1	67.9
TGRL ( $n{=}8$ , ours)	66.0	49.4	92.2	93.7	48.9	64.3	69.1

Appendix

Table 11: Avg@16 results on mathematical reasoning benchmarks with Qwen3-14B under a rollout budget of

n{=}8

per prompt. TGRL allocates

(|\mathcal{G}_{g,T_{0}}|,|\mathcal{G}_{g,T_{1}}|){=}(2,6)

, preserving the 1:3 reference-to-exploration ratio. Best results are in bold .

Split	AIME24	AIME25	AMC23	MATH500	Minerva	Olympiad	Average
$\|\mathcal{G}_{g,T_{0}}\|{=}1,\ \|\mathcal{G}_{g,T_{1}}\|{=}3$	63.4	49.4	92.7	95.3	48.9	66.4	69.4
$\|\mathcal{G}_{g,T_{0}}\|{=}2,\ \|\mathcal{G}_{g,T_{1}}\|{=}2$	60.4	48.8	87.8	93.0	48.4	64.3	67.1
$\|\mathcal{G}_{g,T_{0}}\|{=}3,\ \|\mathcal{G}_{g,T_{1}}\|{=}1$	61.0	49.8	93.9	92.9	47.3	63.8	68.1

Appendix

Table 12: Sensitivity of TGRL to the low-/high-temperature rollout split under a fixed total rollout budget of

4

per prompt. Results are Avg@16 on mathematical reasoning benchmarks. Best results are in bold , second-best results are underlined .

Hyperparameter	Math (Qwen3-14B/32B)	Code (Qwen3-4B)
Training data	DeepScaleR	DeepCoder
Reasoning mode	Qwen3 think	Qwen3 think
Rollouts per prompt $\|\mathcal{G}_{g,T_{0}+T_{1}}\|$	4	4
Low-/high-temperature split $(\|\mathcal{G}_{g,T_{0}}\|,\|\mathcal{G}_{g,T_{1}}\|)$	$(1,3)$	$(1,3)$
Low-temperature-group actor update	disabled	disabled
Warmup steps	20	20

Appendix

Table 13: Reproducibility-critical hyperparameters for the main TGRL experiments.

Hyperparameter	ALFWorld	WebShop
Policy model	Qwen2.5-7B-Instruct	Qwen2.5-7B-Instruct
Engine	vLLM	vLLM
Training data size	16	16
Validation data size	128	64
Rollouts per prompt $\|\mathcal{G}_{g,T_{0}+T_{1}}\|$	8	8
Warmup steps	20	25

Appendix

Table 14: Reproducibility-critical hyperparameters for the TGRL agent experiments. Both runs use Qwen2.5-7B-Instruct as the policy model and share the same TGRL objective defined in the main text and Table 13 ; this table focuses on the agent-specific environment and systems settings.

Control item	Protocol
Backbone and initialization	For each domain and scale, all methods start from the same policy model checkpoint: Qwen3 variants for math/code and Qwen2.5-7B-Instruct for agent experiments. No baseline is initialized from a stronger checkpoint than TGRL.
Training data and verifier	Within each domain, all methods use the same training data source, prompt pipeline, verifier/reward function, and environment interface. Math uses DeepScaleR, code uses DeepCoder, and agent experiments use the same ALFWorld and WebShop environments described in Table 14 .
Rollout budget	All math and code methods use the same total rollout budget $\|\mathcal{G}_{g,T_{0}+T_{1}}\|=4$ responses per prompt. TGRL allocates this fixed budget into $(\|\mathcal{G}_{g,T_{0}}\|,\|\mathcal{G}_{g,T_{1}}\|)=(1,3)$ , while single-temperature baselines spend all four rollouts at their rollout temperature. Agent experiments use the matched budget reported in Table 14 .
Training rollout decoding	Single-temperature RLVR baselines use the same rollout decoding controls as the high-temperature branch unless the method defines an adaptive policy over temperature. In particular, top- $p$ , top- $k$ , maximum response length, and the rollout budget are matched. TAMPO is allowed to adapt temperature because this is its method-specific mechanism, but it receives no additional rollouts.
Evaluation decoding	All math and code checkpoints are evaluated with the same decoding configuration: $T=0.6$ , top- $p=0.95$ , top- $k$ off, and maximum response length $8192$ . Agent evaluation uses the same environment limits and success metrics across methods.
Shared optimization settings	Optimizer family, learning-rate scale, batch size, prompt/response length limits, PPO mini-batch size, number of training epochs, and warmup convention are matched whenever they are not part of a baseline-specific algorithmic definition. Baseline-specific terms, such as entropy bonuses or decoupled clipping rules, are added on top of the shared training recipe rather than changing the data or evaluation protocol.

Appendix

Table 15: Baseline tuning and checkpoint-selection fairness protocol. The same protocol is applied across the reported math, code, and agent experiments unless a method-specific mechanism is explicitly part of the baseline itself.

Explore similar work

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
Jun 23, 2026Wenyang Hu, Junxiang Jia, Zhen Shu +3Verifiable Rewards Trajectory-Level Credit
1National University of Singapore · 2SAP
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Jul 30, 2026Shuang Liang, Haoyang Zhou, Yifan Gong +2Reinforcement Learning With Verifiable Reward LLM Reasoning Strategies
Renmin University of China · AntGroup
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7Reinforcement Learning With Verifiable Reward Verifiable Rewards
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1

Jun 23, 2026, cs.LGJ/K move · Enter open · S save

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier+2

1National University of Singapore · 2SAP

Verifiable Rewards Trajectory-Level Credit Robust Trajectory

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.

Abstract

Explore similar work

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs