Difficulty-Estimated Policy Optimization

Yu Zhao; Fan Jiang; Tianle Liu; Bo Zeng; Xintong Wang; Jiaxuan Teng; Yu Liu; Yichao Du

cs.AIFeb 6, 2026

Difficulty-Estimated Policy Optimization

Authors: Yu Zhao, Fan Jiang, Tianle Liu, Bo Zeng, Xintong Wang, Jiaxuan Teng, Yu Liu, Yichao Du

Organizations: Alibaba International Digital Commerce · School of Software Technology, Dalian University of Technology Dalian, China

Abstract

Recent advancements in Large Reasoning Models (LRMs), exemplified by DeepSeek-R1, have underscored the potential of scaling inference-time compute through Group Relative Policy Optimization (GRPO). However, GRPO frequently suffers from gradient signal attenuation when encountering problems that are either too trivial or overly complex. In these scenarios, the disappearance of inter-group advantages makes the gradient signal susceptible to noise, thereby jeopardizing convergence stability. While variants like DAPO attempt to rectify gradient vanishing, they do not alleviate the substantial computational overhead incurred by exhaustive rollouts on low-utility samples. In this paper, we propose Difficulty-Estimated Policy Optimization (DEPO), a novel framework designed to optimize the efficiency and robustness of reasoning alignment. DEPO integrates an online Difficulty Estimator that dynamically assesses and filters training data before the rollout phase. This mechanism ensures that computational resources are prioritized for samples with high learning potential. Empirical results show that DEPO achieves approximately 1.86x faster rollouts than DAPO in the profiled configuration, while improving benchmark performance over GRPO at comparable training cost. Our approach significantly lowers the computational barrier for training high-performance reasoning models, offering a more sustainable path for reasoning scaling.

Figures & tables

Figure 1: Top: the overview of our proposed DEPO framework. Bottom: Training dynamics of downstream accuracy of GRPO and DEPO .

Figure 2: Architectural overview of the proposed DEPO algorithm. DEPO utilizes a Difficulty Estimator to predict an advantage-related filtering score $\hat{A}_{i}$ for each sampled question. In implementation, $\sigma(\hat{A}_{i})$ estimates the prompt-level Avg@ $G$ score, and prompts whose predicted Avg@ $G$ is close to $0$ or $1$ are filtered before rollout. Samples with intermediate predicted scores are employed for updating the Actor Model using the standard GRPO algorithm. The Difficulty Estimator is simultaneously updated using the Avg@ $G$ targets obtained from the GRPO rollouts.

Figure 3: The online difficulty estimator architecture.

Figure 4: The comparison between the predicted rewards from the estimator and the ground-truth target rewards derived from the actor model. The estimator effectively converges, demonstrating a high degree of fidelity in tracking the target reward trajectory throughout the training process.

DAPO-MATH-17K	Qwen2.5-1.5B-Instruct
Dataset	Method	GSM8K	MATH	AMC23	Olympiad	Minerva	Avg.	GPU Hours $\downarrow$
	GRPO	75.6	48.1	38.4	15.8	11.4	37.9	528 (1.0 $\times$ )
	DAPO	78.5	50.1	39.3	17.8	13.1	39.8	905 (1.7 $\times$ )
	Polaris	77.1	47.3	40.8	16.4	11.8	38.7	584 (1.1 $\times$ )
	DEPO	77.0	48.9	42.3	16.7	12.2	39.4	530 (1.0 $\times$ )
	– ranking loss	76.6	48.0	40.9	16.3	12.1	38.8	530 (1.0 $\times$ )

Table 1: Performance comparison (Avg@32) across five math reasoning benchmarks. DEPO achieves performance comparable to DAPO while significantly reducing training overheads.

Figure 5: Dynamics of filtering ratios when training Qwen2.5-7B-Instruct on datasets of varying difficulty.

Method	Sample	Rollout	Adv. Comput.	Reward	Total $\downarrow$
GRPO	74.63	103.75	0.021	0.406	121.85
DAPO	76.98	192.16	0.067	2.064	211.69
DEPO	74.99	103.11	0.186	0.403	125.65

Table 2: Runtime breakdown and efficiency comparison per training step (seconds). DEPO achieves a

1.86\times

speedup in rollout efficiency compared to DAPO , maintaining a total step latency comparable to the GRPO baseline.

Figure 6: Training dynamics of mean rewards and prompt filtering ratios across the training trajectory. We observe that half of the non-informative prompts are filtered out on average, significantly improving the training efficiency.

Figure 7: Impact of ranking loss weight on downstream accuracy and prompt filtering efficiency. A dual effect of increasing ranking loss weight on model performance and the prompt filtering ratio is observed, where the accuracy reaches a peak while the filter ratio increases monotonically with higher weights.

Figure 8: The comparison of rewards attained by the actor model by employing GRPO and DEPO . We observe the model trained using DEPO consistently obtains higher rewards than that of using GRPO .

Model	$\tau$	GSM8K	MATH	AMC23	Olympiad	Minerva	Avg.	$\Delta$
1.5B	-	77.0	48.9	42.3	16.7	12.2	39.4	-
7B	-	92.3	63.9	63.5	28.7	25.5	54.8	-
1.5B + 7B	0.75	87.1	62.4	65.0	28.0	25.1	53.5	+14.1/-1.3
1.5B + 7B	0.75	557/762	129/371	2/38	27/647	35/237	750/2805	26.7%
1.5B + 7B	0.5	82.0	57.5	61.3	25.3	20.6	49.4	+10.0/-5.4
1.5B + 7B	0.5	997/322	303/197	7/33	117/557	125/147	1549/2805	55.2%

Table 3: The results when employing the difficulty estimator to dynamically route incoming queries to different models.

\Delta

indicates the performance difference when comparing to solely using 1.5B/7B models. The number of queries processed by 1.5B and 7B models are presented respectively.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization
Mar 4, 2026Haodong Zhu, Yangyang Ren, Yanjing Li +6Reinforcement Learning RL for Language Model Reasoning
Beihang University · Zhongguancun Academy · Rakuten Singapore +3
ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
Mar 30, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning RL for Language Models
School of Computer and Information Science, Southwest University, Chongqing, 400715, China
$\nabla$ -Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space
Mar 5, 2026Peihao Wang, Ruisi Cai, Zhen Wang +4LLM Reasoning Test-Time Optimization
The University of Texas at Austin · UC San Diego · TTIC +1

Mar 4, 2026J/K move · Enter open · S save

Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization

Haodong Zhu, Yangyang Ren, Yanjing Li, Mingbao Lin+5

Beihang University · Zhongguancun Academy · Rakuten Singapore+3

Reinforcement Learning RL for Language Model Reasoning Group Relative Policy Optimization

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs due to its extensive group-based sampling requirement. While recent selective data utilization methods can mitigate this overhead, they could induce estimation bias by altering the underlying sampling distribution, compromising theoretical rigor and convergence behavior. To address this limitation, we propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation through importance sampling-based correction. By incorporating mathematically derived rescaling factors, DPPO significantly accelerates GRPO training without altering the optimization objective of the full-batch baseline. Furthermore, to mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy that maximizes valid token density and hardware utilization. Extensive experiments demonstrate that DPPO consistently accelerates training across diverse models and benchmarks. For instance, on Qwen3-4B trained on MATH, DPPO achieves 2.37

\times

training speedup and outperforms GRPO by 3.36% in average accuracy across six mathematical reasoning benchmarks.

Abstract

RL for Language Model Reasoning

Explore similar work

Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization

ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models

∇\nabla∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

$\nabla$ -Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space