DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Xiaofan Li; Ming Yang; Zhiyuan Ma; Shichao Ma; Jintao Du; Yu Cheng; Weiqiang Wang; Zhizhong Zhang

cs.LGApr 15, 2026

DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Authors: Xiaofan Li, Ming Yang, Zhiyuan Ma, Shichao Ma, Jintao Du, Yu Cheng, Weiqiang Wang, Zhizhong Zhang,

Organizations: East China Normal University · Ant Group · Xiamen University · Shanghai Innovation Institute

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.

Figures & tables

Figure 1: (a) The proportion of Easy/Normal/Hard groups in each step during the DAPO training. (b) The PPL distribution of correct and error samples in the validation set at 300th steps of DAPO training. (c) Illustration of four samples after PSD fine-grained partitioning.

Figure 2: Illustration of DiPO, consisting of three modules: PPL Queue, Perplexity Space Disentangling (PSD), and Bidirectional Reward Reallocation (BRR). Specifically, the PPL Queue caches PPL items; PSD is used for fine-grained sample partition; and then BRR performs reward allocation.

Method	AIME24	AIME25	MATH	AMC	OLY	MIN	AVG
Qwen3-4B-Base
Base model	9.58	3.75	48.48	31.48	25.52	24.82	23.94
GRPO	26.67	23.33	85.83	60.24	53.06	44.39	48.92
DAPO	26.25	23.75	86.43	61.90	53.88	44.34	49.43
DAPO w/ EL	26.67	24.58	86.78	62.95	54.53	44.53	50.01
CDE	26.67	24.17	85.93	62.35	52.25	43.11	49.08

Table 1: Comparison of mathematical reasoning in ACC/mean@8 on 6 mathematics benchmarks. The best and second-best results are respectively marked in bold and underlined.

Method	Non-Live Acc	Live Acc	Multi Turn Acc	Relevance Detection	Irrelevance Detection	Overall
Qwen2.5-3B-Instruct
Base model	42.52	53.96	1.00	44.44	82.49	33.04
SFT400	69.29	41.40	0.00	94.44	60.14	34.08
SFT400+PPO	78.29	58.76	5.12	100.00	48.40	45.80
SFT400+GRPO	76.21	64.15	1.75	94.44	58.63	46.42
PPO, Cold Start	82.42	67.78	4.88	100.00	18.09	51.15

Table 2: Comparison of function calling in acc on BFCLv3 benchmark. The best and second-best results are respectively marked in bold and underlined.

$\alpha$	AIME24	AIME25	MATH	AMC	OLY	MIN	AVG
Qwen3-4B-Base
$\alpha=0.0$	26.25	23.75	86.43	61.90	53.88	44.34	49.43
$\alpha=0.1$	29.17	24.58	86.93	63.70	54.09	44.76	50.55
$\alpha=1.0$	25.83	24.17	86.38	62.05	54.01	45.50	49.66
Qwen3-8B-Base
$\alpha=0.0$	30.08	25.83	89.43	69.12	56.90	48.02	53.23

Table 3: The impact of hyperparameter

\alpha

on performance. The best results are marked in bold .

PSD	BRR	PPL reward	AIME24	AIME25	MATH	AMC	OLY	MIN	AVG
Qwen3-4B-Base
✗	✗	✗	26.25	23.75	86.43	61.90	53.88	44.34	49.43
✗	✓	✗	25.83	20.83	85.10	59.19	49.37	43.06	47.23
✗	✗	✓	24.58	23.75	84.60	61.45	49.09	41.82	47.55
✓	✗	✓	27.50	23.33	86.35	62.35	54.00	44.21	49.62
✓	✓	✗	29.17	24.58	86.93	63.70	54.09	44.76	50.55

Table 4: Ablation study results on Qwen3-4B-Base and Qwen-8B-Base, showing ACC/mean@8 on six mathematical reasoning benchmarks. The best results are marked in bold .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Entropy curves of maximum-PPL reward and maximum-PPL penalty trained on Qwen3-0.6B model with using DAPO-17K.

data	train_batch_size	128	actor	ppo_mini_batch	128	rollout	temperature	1.2
	max_prompt_length	1K		clip_ratio_low	0.2		n	8
	max_response_length	4K		clip_ratio_high	0.28		val_kwargs.n	8
	gen_batch_size	256		kl_loss_coef	0		val_kwargs.temperature	0.6

Appendix

Table 5: Training configurations of mathematical reasoning.

data	train_batch_size	512	actor	ppo_mini_batch	128	rollout	temperature	1.2
	max_prompt_length	2K		clip_ratio_low	0.2		n	4
	max_response_length	1K		clip_ratio_high	0.28		val_kwargs.n	4
	gen_batch_size	1024		kl_loss_coef	0		val_kwargs.temperature	0.6

Appendix

Table 6: Training configurations of function calling.

Method	GSM8K	MATH	AVG
Llama3.1-8B-Instruct	79.80	47.83	63.82
DAPO	89.89	55.75	72.82
DAPO w/ EL	89.66	52.68	71.17
CDE	89.01	53.45	71.23
DiPO (ours)	90.03	56.75	73.39

Appendix

Table 7: Comparison of mathematical reasoning in acc/mean@8 using Llama3.1-8B-Instruct as base model. The best results is marked in bold .

Model	AIME24	AIME25	MATH	AMC	OLY	MIN	AVG
Qwen3-4B-Base
Base model	13.30	6.67	64.00	46.99	37.59	30.15	33.12
GRPO	36.67	26.67	89.60	68.67	58.10	50.37	55.01
DAPO	33.33	30.00	89.60	69.88	59.29	49.26	55.23
DAPO w/ EL	33.33	30.00	90.40	68.67	60.03	50.40	55.47
CDE	36.67	30.00	89.80	69.88	57.06	49.63	55.51

Appendix

Table 8: Comparison of mathematical reasoning in ACC/maj@8 on 6 mathematics benchmarks. The best and second-best results are respectively marked in bold and underlined.

Mathod	coeff	AIME24	AIME25	MATH	AMC	OLY	MIN	AVG
Qwen3-4B-Base
Baseline	0.00	26.25	23.75	86.43	61.90	53.88	44.34	49.43
	0.10	29.17	24.58	87.00	64.91	54.09	44.76	50.75
DiPO	1.00	25.83	24.17	86.38	62.05	54.01	45.50	49.66
	0.001	26.67	24.58	86.78	62.95	54.53	44.53	50.01
DAPO /w EL	0.01	18.33	20.00	81.12	50.00	45.99	39.38	42.47

Appendix

Table 9: The impact of the coefficients for DiPO and entropy loss on the results.

Method	ACC/mean@8	Precision/mean@8	Recall/mean@8	F1/mean@8
Qwen3-8B	52.06	87.18	54.64	67.18
DAPO	76.94	95.96	77.58	85.80
DiPO	78.37	95.69	79.49	86.84

Appendix

Table 10: The results of risk prediction, including accuracy, precision, recall, and F1 score, and all results are the mean of 8 independent inferences.

Figure 7: PPL distribution of correct and error samples for Qwen3-8B-Base trained on DAPO-17K Dataset via DiPO (TOP) and DAPO (Bottom).

Figure 8: Correct case of DAPO.

Figure 9: Error case of DAPO.

Figure 10: Correct case of DiPO.

Figure 11: Error case of DiPO.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Parameter Exploration for RLVR via Variational Learning
Aug 10, 2026Vatsal Venkatkrishna, Nico Daheim, Iryna GurevychRL for Language Model Reasoning RL Exploration
INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

Oct 5, 2025J/K move · Enter open · S save

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Zishang Jiang, Jinyi Han, Tingyun Li, Xinyi Wang+6

School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University+1

RL for Language Model Reasoning LLM-Guided RL RL Exploration Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Abstract

Reinforcement Learning

Explore similar work

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Parameter Exploration for RLVR via Variational Learning

Boosting LLM Exploration via Weak-Model Guidance in RLVR