Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

Jiashu Yao; Heyan Huang; Daiqing Wu; Zeming Liu; Yuhang Guo

cs.CLApr 13, 2026

Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

Authors: Jiashu Yao, Heyan Huang, Daiqing Wu, Zeming Liu, Yuhang Guo

Organizations: Beijing Institute of Technology · Tsinghua University · Beihang University

Abstract

To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.

Figures & tables

Figure 1: (a) Entropy-guided RL can leads to accuracy drop when prefering high entropy. (b) Unlike previous work, Policy Split innovatively bifurcates the policy into normal and high-entropy modes. (c) Through dual-mode entropy regularization in Policy Split, two modes obtain a larger KL divergence and exhibit distinct entropy behaviors.

Figure 2: The training framework of Policy Split.

Models	MATH-500		AIME-24/25		GPQA-Diam		MMLU-Pro		Average
Models	Acc	Ent	Acc	Ent	Acc	Ent	Acc	Ent	Acc	Ent
Qwen3-1.7B
Original	89.08	0.34	40.54	0.36	38.19	0.51	56.52	0.48	56.08	0.42
GRPO	89.12	0.31	41.04	0.35	37.88	0.51	55.71	0.46	55.94	0.41
w/ Forking Token Only	89.15	0.31	37.92	0.36	38.95	0.51	56.00	0.46	55.51	0.41
w/ Clip-Cov	89.03	0.31	38.33	0.36	37.63	0.50	56.66	0.45	55.41	0.41

Table 1: Main results. The highest average results for each model are shown in bold. All improvement of Policy Split over original models have passed the significance test with

p<0.01

Models	Div-N-Gram	Div-BLEU	Creativity	Originality	Narrative	Impact	Imagery
Original	49.62	60.34	57.69	48.56	69.81	55.19	62.21
GRPO	49.82	60.26	55.29	47.31	68.56	53.94	59.71
w/ Ent Advantage	49.67	60.74	57.02	47.50	69.42	53.56	60.67
Policy Split	50.04	61.67	57.88	48.46	68.08	54.18	60.19
+ HE prompt	52.08	65.34	62.50	55.29	70.48	56.06	65.67

Table 2: Experiments of creative writing on Qwen3-8B-based models. Metrics include rule-based ones (N-gram diversity and self-BLEU diversity) and LLM-as-a-Judge ones (Creativity, Originality, Narrative Flow, Emotional Impact, and Imagery). All scores are scaled at 100. The best results are shown in bold.

$\eta$	$\eta^{\prime}$	$\alpha$	Acc	Ent
Policy Split
$0.03$	$0$	$0.5$	56.78	0.82
Ablations of $\eta$ and $\eta^{\prime}$
$0.01$	$0$	$0.5$	56.06	0.50
$0.10$	$0$	$0.5$	49.21	1.54
$0.03$	$0.01$	$0.5$	56.05	0.95

Table 3: Ablation experiments of entropy coefficient for high entropy mode

\eta

, entropy coefficient for normal mode

\eta^{\prime}

, and the entropy computation weight

\alpha

. All experiments are based on Qwen3-1.7B, and the reported scores of accuracy and entropy are the average scores of normal mode across eight runs on all four tasks.

Models	Rollout	Modeling	Acc
GRPO	$1\times$	$1\times$	55.94
GRPO w/ Sharing	$1\times$	$2\times$	55.86
Policy Split	$1\times$	$2\times$	56.78

Table 4: Ablation of rollout sharing. Rollout refers to the auto-regressive sampling cost, and Modeling refers to the probability re-computation and updating cost.

Models	\| $\bm{\Delta}$ Acc \|	\| $\bm{\Delta}$ Ent \|	\| $\bm{\Delta}$ Len \|	$\mathbb{D}_{KL}$
Qwen3-1.7B
Original	0.56	0.02	600	0.011
GRPO	0.74	0.01	885	0.048
Policy Split	3.60	0.07	1,356	0.293
Qwen3-4B
Original	0.43	0.02	457	0.015

Table 5: The absolute difference between the high entropy mode and the original mode in average accuracy, average entropy, average response length, and the forward KL divergence between the two modes. The highest results for each model are shown in bold. The training process of Policy Split effectively enlarges the distance between the policies of the two modes.

Models	Mode	Avg	Best
Policy Split 1.7B	Normal	56.78	74.81
Policy Split 1.7B	High-Ent	53.18	76.92
Policy Split 4B	Normal	71.75	81.44
Policy Split 4B	High-Ent	70.47	83.80
Policy Split 8B	Normal	75.15	84.91
Policy Split 8B	High-Ent	71.49	86.53

Table 6: The average and best-of-8 accuracy comparison between the normal and the high-entropy modes across Policy Split trained on Qwen3 series 1.7B, 4B, and 8B.

Models	Acc	Ent	$\bm{\Delta}$ Ent
Original	73.94	0.45	-
+ HE prompt	73.38	0.43	-0.02
+ rewritten HE prompt	73.25	0.44	-0.01
+ LE prompt	73.82	0.41	-0.04
GRPO	74.27	0.44	-
+ HE prompt	73.09	0.42	-0.02

Table 7: Experiments of applying different mode-switching system prompts on Qwen3-8B-based models.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameter	Value
parameters	full
epoch	$1$
batch size	$256$
learning rate	$3e-6$
learning rate scheduler	constant
gradient clip norm	$1.0$

Appendix

Table 8: Hyper-parameters for training.

Hyper-parameter	Train	Test
max new tokens	$12,288$	$32,768$
temperature	$1.0$	$0.6$
top k	$-1$	$20$
top p	$1.0$	$0.95$

Appendix

Table 9: Hyper-parameters for inference.

Models	Creativity	Originality	Narrative	Impact	Imagery
GPT-5
Original	57.69	48.56	69.81	55.19	62.21
GRPO	55.29	47.31	68.56	53.94	59.71
w/ Ent Adv	57.02	47.50	69.42	53.56	60.67
Policy Split	57.88	48.46	68.08	54.18	60.19
+ HE prompt	62.50	55.29	70.48	56.06	65.67

Appendix

Table 10: Experiments of creative writing on Qwen3-8B-based models. Metrics include rule-based ones (N-gram and Self-BLEU) and LLM-as-a-Judge ones (Creativity, Originality, Narrative Flow, Emotional Impact, and Imagery).

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

Targeted Exploration via Unified Entropy Control for Reinforcement Learning
Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Reinforcement Learning RL for Language Model Reasoning
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3
DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
Feb 23, 2026Zhongwei Wan, Yun Shen, Zhihao Dou +9Reinforcement Learning RL for Language Model Reasoning
The Ohio State University, USA · Case Western Reserve University, USA · The Chinese University of Hong Kong, Hong Kong +4
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

Apr 16, 2026J/K move · Enter open · S save

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao+4

College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University+3

Reinforcement Learning RL for Language Model Reasoning RL Exploration Maximum Entropy RL RL for VLMs

Open paper AskPDF arXiv

Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@

k

. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.

Abstract

RL for Language Models

Explore similar work

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

Boosting LLM Exploration via Weak-Model Guidance in RLVR