Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling
Authors: Siyuan Gan, Yuhan Li, Xiran Wang, Linjian Meng, Boyan Wang, Zhen Zhao, Jing Huo, Lei Bai, Yang Gao
Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) is a prominent variant of Group Relative Policy Optimization (GRPO). DAPO introduces several improvements over GRPO. Among these, Dynamic Sampling contributes the most to DAPO's accuracy gains relative to GRPO. To improve accuracy, Dynamic Sampling enhances training stability by eliminating zero policy gradients from zero advantages. Specifically, it avoids such zero gradients by filtering out prompts where sampled responses are either entirely correct or incorrect. However, our theoretical analysis shows that Dynamic Sampling decrease training efficiency as it cannot effectively utilize hard-to-sample correct responses on hard prompts. Formally, it asymmetrically amplifies the advantages of distinct responses to the same prompts. On hard prompts, incorrect responses undergo greater amplification than correct ones. This leads the model to avoid generating the observed incorrect responses rather than capitalizing on the hard-to-sample correct ones on hard prompts, resulting in low training efficiency. To improve training efficiency, we propose Direct Advantage Amplification (DAA), which amplifies the advantages of hard-to-sample correct responses on hard prompts, as obtained by Dynamic Sampling. This ensures that, when Dynamic Sampling is used, these hard-to-sample responses can be effectively capitalized on, implying higher training efficiency. By integrating DAA into DAPO, we obtain Difficulty-aware Advantage Amplification Policy Optimization (DA3PO), which is implemented with fewer than 30 lines of code from DAPO. Experiments show that DA3PO significantly outperforms GRPO and other classical GRPO variants.
Figures & tables
Figure 1: Asymmetric amplification factors α+ and α− under Dynamic Sampling (Theorem 3.1 ).
Figure 2: Ratio γ=α+/α− satisfies γ<1 across the entire shaded region ( p<0.5 ), regardless of group size G (Corollary 3.2 ).
Model
Algorithm
AIME24
AIME25
AIME26
AMC
Minerva
Olympiad
MATH
AVG
avg@32
avg@32
avg@32
avg@32
avg@8
avg@8
avg@8
Qwen3-4B-Base
Base Model
5.21
3.23
3.65
20.18
12.22
19.56
41.23
15.04
GRPO
15.31
12.81
11.56
55.50
23.99
39.35
77.22
33.68
DAPO
18.96
15.10
12.71
52.07
24.17
40.65
78.20
34.55
GSPO
18.85
13.17
14.27
51.51
22.61
41.96
79.87
34.61
DA3PO
23.96
19.17
17.19
60.50
26.84
44.52
85.20
39.63
Table 1: Avg@k (%) of all algorithms across seven mathematical reasoning benchmarks, with absolute improvement of DA3PO over each baseline. We report avg@32 for AIME and AMC, and avg@8 for Minerva, Olympiad and MATH. Best results within each benchmark are bolded .
Model
Algorithm
AIME24
AIME25
AIME26
AMC
Minerva
Olympiad
MATH
AVG
pass@32
pass@32
pass@32
pass@32
pass@8
pass@8
pass@8
Qwen3-4B-Base
Base Model
30.00
23.33
16.67
78.31
28.31
42.67
77.60
42.41
GRPO
36.67
46.67
30.00
86.75
36.03
56.00
90.20
54.62
DAPO
43.33
43.33
30.00
84.34
39.71
59.56
91.00
55.90
GSPO
50.00
40.00
30.00
91.57
35.66
57.33
89.20
56.25
DA3PO
60.00
46.67
40.00
91.57
41.18
61.19
93.80
62.06
Table 2: Pass@k (%) of all algorithms across seven mathematical reasoning benchmarks, with absolute improvement of DA3PO over each baseline. We report pass@32 for AIME and AMC, and pass@8 for Minerva, Olympiad and MATH. Best results within each benchmark are bolded .
Figure 3: Avg@32 of AIME24 over the course of RL training for all algorithms. Due to constraints on computational resources, we only use AIME24 as the validation benchmark.
Model
Algorithm
AIME24
AIME25
AIME26
AMC
Minerva
Olympiad
MATH
AVG
avg@32
avg@32
avg@32
avg@32
avg@8
avg@8
avg@8
Qwen3-4B-Base
λ=1.0
18.96
15.10
12.71
52.07
24.17
40.65
78.20
34.55
λ=1.5
22.08
19.17
15.94
60.35
26.98
44.50
83.80
38.97
λ=2.0
23.96
19.17
17.19
60.50
26.84
44.52
85.20
39.63
λ=2.5
23.02
15.62
13.33
58.32
25.64
39.91
78.33
36.31
Qwen3-8B-Base
λ=1.0
24.58
20.52
17.19
64.83
29.50
48.30
85.52
41.49
Table 3: Ablation on the amplification factor λ . We report avg@32 for AIME and AMC, and avg@8 for Minerva, Olympiad and MATH. λ=1.0 is DAPO as a reference. Best results within each benchmark are bolded .
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs due to its extensive group-based sampling requirement. While recent selective data utilization methods can mitigate this overhead, they could induce estimation bias by altering the underlying sampling distribution, compromising theoretical rigor and convergence behavior. To address this limitation, we propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation through importance sampling-based correction. By incorporating mathematically derived rescaling factors, DPPO significantly accelerates GRPO training without altering the optimization objective of the full-batch baseline. Furthermore, to mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy that maximizes valid token density and hardware utilization. Extensive experiments demonstrate that DPPO consistently accelerates training across diverse models and benchmarks. For instance, on Qwen3-4B trained on MATH, DPPO achieves 2.37× training speedup and outperforms GRPO by 3.36% in average accuracy across six mathematical reasoning benchmarks.
Haodong Zhu, Yangyang Ren, Yanjing Li +6
Beihang University · Zhongguancun Academy · Rakuten Singapore +3
Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07× training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at https://github.com/Nicozwy/FastRL.
Jiahua Yang, Zhiwei Yang, Xianpeng Zhang +7
Guangdong Institute of Smart Education, Jinan University, Guangzhou, China · OPPO AI Center, Shenzhen, China · Ragentile Intelligence Inc, Edmonton, Canada
Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.