Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization

Haodong Zhu; Yangyang Ren; Yanjing Li; Mingbao Lin; Linlin Yang; Xuhui Liu; Xiantong Zhen; Haiguang Liu

cs.LGMar 4, 2026

Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization

Authors: Haodong Zhu, Yangyang Ren, Yanjing Li, Mingbao Lin, Linlin Yang, Xuhui Liu, Xiantong Zhen, Haiguang Liu, +1 more

Organizations: Beihang University · Zhongguancun Academy · Rakuten Singapore · Communication University of China · KAUST · United Imaging

Abstract

Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs due to its extensive group-based sampling requirement. While recent selective data utilization methods can mitigate this overhead, they could induce estimation bias by altering the underlying sampling distribution, compromising theoretical rigor and convergence behavior. To address this limitation, we propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation through importance sampling-based correction. By incorporating mathematically derived rescaling factors, DPPO significantly accelerates GRPO training without altering the optimization objective of the full-batch baseline. Furthermore, to mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy that maximizes valid token density and hardware utilization. Extensive experiments demonstrate that DPPO consistently accelerates training across diverse models and benchmarks. For instance, on Qwen3-4B trained on MATH, DPPO achieves 2.37 $\times$ training speedup and outperforms GRPO by 3.36% in average accuracy across six mathematical reasoning benchmarks.

Figures & tables

Figure 1: Comparison of accuracy and training time on the MATH dataset for Qwen3-4B and Qwen3-8B. Bars indicate accuracy and red lines indicate training time. Our method achieves the highest accuracy while requiring the least training time.

Figure 2: Overview of our DPPO. It employs a hierarchical pruning strategy to accelerate GRPO by reducing redundancy at both the prompt level (via difficulty estimation $H_{t}(q)$ ) and the completion level (via advantage assessment $|A_{t}|$ ). In the left panel, a mathematically grounded rescaling mechanism is applied to retained samples to correct for estimation bias. The right panel details the end-to-end training loop, where dynamic pruning and importance-based rescaling are integrated to ensure efficient yet unbiased policy optimization.

Figure 3: Dense Prompt Packing Strategy. The left panel shows the window-based greedy algorithm for assembling variable-length prompts into compact sequences. The right panel indicates the mitigation of pruning-induced sparsity. Unlike standard batching, our approach maximizes valid token density and hardware saturation, ensuring throughput remains consistent with the full-batch pattern.

Setting	Method	$r_{q}$	$r_{o}$	GSM8K Dataset			Math Dataset
Setting	Method	$r_{q}$	$r_{o}$	GPU-h	Speedup	Acc (%)	GPU-h	Speedup	Acc (%)
Qwen3-4B	GRPO	0%	0%	64.68	1.00 $\times$	93.85	137.04	1.00 $\times$	77.00
	DPPO	50%	50%	39.28	1.65 $\times$	$93.49^{{\color[rgb]{1,0,0}-0.36}}$	88.32	1.55 $\times$	$78.10^{{\color[rgb]{0,0,1}+1.10}}$
	DPPO	0%	50%	49.47	1.31 $\times$	$94.01^{{\color[rgb]{0,0,1}+0.16}}$	110.80	1.24 $\times$	$80.02^{{\color[rgb]{0,0,1}+3.02}}$
	DPPO	50%	0%	45.74	1.41 $\times$	$94.24^{{\color[rgb]{0,0,1}+0.37}}$	102.48	1.34 $\times$	$77.65^{{\color[rgb]{0,0,1}+0.65}}$
[2pt/1pt]	DPPO	30%	30%	45.94	1.41 $\times$	$93.86^{{\color[rgb]{0,0,1}+0.01}}$	101.12	1.36 $\times$	$77.75^{{\color[rgb]{0,0,1}+0.75}}$

Table 1: Comparision between GRPO and DPPO on GSM8K and Math test subset for Qwen3-4B and Qwen3-8B, with DPPO evaluated under varying prompt-level (

r_{q}

) and completion-level (

r_{o}

) pruning ratios. Blue ( red ) values indicate gains (drops) in accuracy relative to the GRPO baseline. Speedup is computed w.r.t. GRPO time (1.00

\times

). GPU-h denotes GPU hours.

Model	Method	Speedup $\uparrow$	MATH500	AIME25	AIME24	AMC	Minerva.	Olympiad.	Avg. $\uparrow$
Qwen3-4B	GRPO	1.00 $\times$	76.20	8.33	7.50	45.78	29.41	34.72	33.44
	CPPO	1.35 $\times$	75.15	6.67	5.83	41.87	28.49	31.49	31.58
	GRESO	2.31 $\times$	76.40	10.83	10.83	47.29	30.15	37.28	35.10
	DPPO	2.37 $\times$	77.90	11.67	17.50	46.99	29.96	38.02	36.80
Qwen3-8B	GRPO	1.00 $\times$	75.80	8.33	7.50	46.08	29.32	34.27	33.55
	CPPO	1.32 $\times$	75.40	6.67	5.83	41.57	29.23	31.53	31.71

Table 2: Comparison of DPPO (

r_{q}=0.9

r_{o}=0.9

) and baseline RL methods on different mathematical benchmarks, evaluated on Qwen3-4B and Qwen3-8B in terms of accuracy and speedup (GRPO=1.00

\times

). Bold indicates the best result in each column.

Setting	Method	GPU-h	Speedup	Acc (%)
Qwen3-4B	DAPO	48.53	1.00 $\times$	94.09
	+ DPPO	24.57	1.98 $\times$	94.10
	GSPO	61.40	1.00 $\times$	93.78
	+ DPPO	25.07	2.45 $\times$	94.17
Qwen3-8B	DAPO	55.24	1.00 $\times$	95.14
	+ DPPO	28.72	1.92 $\times$	94.85

Table 3: Generalization of DPPO when integrated with DAPO on GSM8K for Qwen3-4B and Qwen3-8B, with prompt-level and completion-level pruning ratios set to

r_{q}=0.9

and

r_{o}=0.9

Method	Rollout ( $G$ )	GPU-h	Speedup	Acc (%)
GRPO	8	89.76	1.00 $\times$	94.84
DPPO	8	54.80	1.64 $\times$	94.47
GRPO	16	173.36	1.00 $\times$	94.69
DPPO	16	107.76	1.61 $\times$	94.47

Table 4: Ablation study on the impact of rollout numbers (

G

). We report the accuracy and computational cost of Qwen3-4B on GSM8K under different group sizes with prompt-level and completion-level pruning ratios set to

r_{q}=0.5

and

r_{o}=0.5

Module	Pruning Rules	$r_{q}$	$r_{o}$	Acc (%)
Baseline (No Pruning)	–	–	–	93.85
Completion-level Ablation
Ours (Prune Low)	$\|A^{q_{k}}_{i,t}\|\leq\bar{\mathcal{A}}^{q_{k}}_{t}$	0	0.5	94.01
Reverse (Prune High)	$\|A^{q_{k}}_{i,t}\|>\bar{\mathcal{A}}^{q_{k}}_{t}$	0	0.5	93.52
Prompt-level Ablation
Ours (Prune Low)	$q_{k}\in\mathcal{B}_{t}$	0.5	0	94.24

Table 5: Ablation study on pruning criteria, evaluated on GSM8K with Qwen3-4B. ‘Reverse’ retains the samples that our method would prune.

Method	Training Time (GPU hours)	Speedup	Acc (%)
Baseline	78.93	1.00 $\times$	94.85
+ Comp. Pruning	55.10	1.34 $\times$	94.77
+ Prompt Pruning	48.79	1.52 $\times$	94.94
+ Dense Packing	47.00	1.68 $\times$	95.15

Table 6: Ablation study on the incremental contribution of each component (

r_{q}=0.5

r_{o}=0.5

), evaluated on GSM8K with Qwen3-8B. ‘Comp.’ and ‘Prompt’ denote Completion-level and Prompt-level pruning, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Method	$r_{q}$	$r_{o}$	GPU-h	Speedup	Acc (%)
Qwen3-4B	DAPO	0%	0%	48.53	1.00 $\times$	94.09
	+ DPPO	50%	50%	32.43	1.50 $\times$	$94.02^{{\color[rgb]{1,0,0}-0.07}}$
	+ DPPO	0%	50%	42.90	1.13 $\times$	$94.24^{{\color[rgb]{0,0,1}+0.15}}$
	+ DPPO	50%	0%	42.45	1.14 $\times$	$94.24^{{\color[rgb]{0,0,1}+0.15}}$
[2pt/1pt]	+ DPPO	30%	30%	37.27	1.30 $\times$	$94.70^{{\color[rgb]{0,0,1}+0.61}}$
	+ DPPO	70%	70%	31.74	1.53 $\times$	$94.70^{{\color[rgb]{0,0,1}+0.61}}$

Appendix

Table 7: Comparison between DAPO and DPPO on GSM8K for Qwen3-4B and Qwen3-8B, with DPPO evaluated under varying prompt-level (

r_{q}

) and completion-level (

r_{o}

) pruning ratios. Accuracy changes relative to DAPO baseline are shown in superscript ( blue for improvement, red for decline).

Setting	Method	$r_{q}$	$r_{o}$	GPU-h	Speedup	Acc (%)
Qwen3-4B	GSPO	0%	0%	61.40	1.00 $\times$	93.78
	+ DPPO	50%	50%	35.85	1.71 $\times$	$93.49^{{\color[rgb]{1,0,0}-0.29}}$
	+ DPPO	0%	50%	47.09	1.30 $\times$	$94.17^{{\color[rgb]{0,0,1}+0.39}}$
	+ DPPO	50%	0%	44.64	1.38 $\times$	$94.17^{{\color[rgb]{0,0,1}+0.39}}$
[2pt/1pt]	+ DPPO	30%	30%	42.85	1.43 $\times$	$94.17^{{\color[rgb]{0,0,1}+0.39}}$
	+ DPPO	70%	70%	27.84	2.21 $\times$	$93.94^{{\color[rgb]{0,0,1}+0.16}}$

Appendix

Table 8: Comparison between GSPO and DPPO on GSM8K for Qwen3-4B and Qwen3-8B, with DPPO evaluated under varying prompt-level (

r_{q}

) and completion-level (

r_{o}

) pruning ratios. Accuracy changes relative to GSPO baseline are shown in superscript ( blue for improvement, red for decline).

Setting	Method	$r_{q}$	$r_{o}$	GPU-h	Speedup	Acc (%)
Qwen2.5-7B-Instruct	GRPO	0%	0%	58.39	1.00 $\times$	71.84
	+ DPPO	50%	50%	37.31	1.56 $\times$	$71.97^{{\color[rgb]{0,0,1}+0.13}}$
	+ DPPO	0%	50%	41.31	1.41 $\times$	$72.47^{{\color[rgb]{0,0,1}+0.63}}$
	+ DPPO	50%	0%	42.84	1.36 $\times$	$71.76^{{\color[rgb]{1,0,0}-0.08}}$
[2pt/1pt]	+ DPPO	30%	30%	42.60	1.37 $\times$	$72.07^{{\color[rgb]{0,0,1}+0.23}}$
	+ DPPO	70%	70%	32.35	1.80 $\times$	$71.56^{{\color[rgb]{1,0,0}-0.28}}$

Appendix

Table 9: Robustness across RL algorithms. We integrate DPPO into GRPO and DAPO on the MATH dataset using Qwen2.5-7B-Instruct and Llama3.2-3B-Instruct . Accuracy changes relative to each baseline are shown in superscript ( blue for improvement, red for decline).

Setting	Method	$r_{q}$	$r_{o}$	GPU-h	Speedup	Acc (%)
Qwen3-32B	GRPO	0%	0%	204.08	1.00 $\times$	80.43
	+ DPPO	50%	50%	127.92	1.60 $\times$	$82.55^{{\color[rgb]{0,0,1}+2.12}}$
	+ DPPO	0%	50%	166.08	1.23 $\times$	$81.95^{{\color[rgb]{0,0,1}+1.52}}$
	+ DPPO	50%	0%	159.84	1.28 $\times$	$80.80^{{\color[rgb]{0,0,1}+0.37}}$
[2pt/1pt]	+ DPPO	30%	30%	157.28	1.30 $\times$	$81.25^{{\color[rgb]{0,0,1}+0.82}}$
	+ DPPO	70%	70%	101.20	2.02 $\times$	$84.62^{{\color[rgb]{0,0,1}+4.19}}$

Appendix

Table 10: Robustness across larger model on Math dataset. Accuracy changes relative to each baseline are shown in superscript ( blue for improvement, red for decline).

Panel of Figure 4 — Figure 4: Training dynamics of GRPO and DPPO variants on MATH dataset with Qwen3-4B (left) and Qwen3-8B (right).

Question:

Let

x_{1},

x_{2},

\dots,

x_{101}

be positive real numbers such that

x_{1}^{2}+x_{2}^{2}+\dots+x_{101}^{2}=1.

Find the maximum value of

x_{1}x_{2}+x_{1}x_{3}+\dots+x_{1}x_{101}.

Let’s think step by step and output the final answer within box.

Ground Truth: 5

GRPO Response ✗ :

<think> Okay, so I need to find the maximum value of the sum

x_{1}x_{2}+x_{1}x_{3}+\cdots+x_{1}x_{101}

, given that the sum of the squares of all the

x_{i}

’s is

1

. Hmm, let me think. First, the expression we need to maximize is the sum of all products of

x_{1}

with each of the other variables. So that’s like

x_{1}

multiplied by

(x_{2}+x_{3}+\cdots+x_{101})

. Let me write that down:

\text{Sum}=x_{1}(x_{2}+x_{3}+\cdots+x_{101}).

Let me denote

S=x_{2}+x_{3}+\cdots+x_{101}

. Then the sum becomes

x_{1}S

. So maybe I can use some inequality here. Since we have a constraint on the sum of squares, maybe Cauchy–Schwarz or AM–QM or something? Wait, Cauchy–Schwarz might be useful here. Let me recall that for vectors

\mathbf{a}

and

\mathbf{b}

(\mathbf{a}\cdot\mathbf{b})^{2}\leq(\mathbf{a}\cdot\mathbf{a})(\mathbf{b}\cdot\mathbf{b})

. But here, maybe I can think of

x_{1}

and

S

as part of some vectors? Alternatively, maybe I can use the Cauchy–Schwarz inequality on

x_{1}

and the vector

(x_{2},x_{3},\ldots,x_{101})

. Let me try that. The sum

x_{1}S

is the dot product of

(x_{1},0,0,\ldots,0)

and

(x_{2},x_{3},\ldots,x_{101})

. Then by Cauchy–Schwarz:

x_{1}S\leq\sqrt{x_{1}^{2}+0+\cdots+0}\;\sqrt{x_{2}^{2}+x_{3}^{2}+\cdots+x_{101}^{2}}=x_{1}\sqrt{1-x_{1}^{2}}.

So then the sum is

\leq x_{1}\sqrt{1-x_{1}^{2}}

. Now, to maximize this expression with respect to

x_{1}

, let me set

f(x)=x\sqrt{1-x^{2}}

. To find its maximum, take derivative:

f^{\prime}(x)=\sqrt{1-x^{2}}+x\cdot\frac{1}{2\sqrt{1-x^{2}}}\cdot(-2x)=\frac{1-2x^{2}}{\sqrt{1-x^{2}}}.

Set derivative to zero:

1-2x^{2}=0\Rightarrow x^{2}=\frac{1}{2}\Rightarrow x=\frac{1}{\sqrt{2}}

. Therefore, the maximum value is

\frac{1}{\sqrt{2}}\sqrt{1-\frac{1}{2}}=\frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}}=\frac{1}{2}.

So the maximum sum is

\frac{1}{2}

. Therefore, the answer is

\boxed{\tfrac{1}{2}}

. Wait, but let me check if this is correct. Because when we apply Cauchy–Schwarz, we have that the maximum of

x_{1}S

\frac{1}{2}

, given that

x_{1}^{2}+\bigl(\text{sum of squares of others}\bigr)=1

. So yes, that seems right. Therefore, the maximum value is

\frac{1}{2}

. Final Answer

\boxed{\tfrac{1}{2}}

</think>

GRESO Response ✗ :

<think> Okay, so I need to find the maximum value of the sum

x_{1}x_{2}+x_{1}x_{3}+\cdots+x_{1}x_{101}

given that the sum of the squares of all the variables is

1

. Let me think about how to approach this. First, the problem is about maximizing a sum of products of variables, given a constraint on their squares. This seems like a problem where the Cauchy–Schwarz inequality or Lagrange multipliers might be useful. Since all variables are positive, maybe Cauchy–Schwarz is the way to go. Let me write down the expression I need to maximize. The sum is

x_{1}(x_{2}+x_{3}+\cdots+x_{101})

. Let me denote

S=x_{2}+x_{3}+\cdots+x_{101}

. Then the expression becomes

x_{1}S

. So I need to maximize

x_{1}S

, given that

x_{1}^{2}+x_{2}^{2}+\cdots+x_{101}^{2}=1

. Hmm. Maybe I can use the Cauchy–Schwarz inequality on

x_{1}

and

S

. Let me recall that for any vectors

\mathbf{a}

and

\mathbf{b}

(\mathbf{a}\cdot\mathbf{b})\leq\|\mathbf{a}\|\,\|\mathbf{b}\|

. If I take

\mathbf{a}=(x_{1},0,\ldots,0)

and

\mathbf{b}=(0,x_{2},x_{3},\ldots,x_{101})

, then their dot product is

x_{1}S

. The norm of

\mathbf{a}

x_{1}

, and the norm of

\mathbf{b}

\sqrt{x_{2}^{2}+\cdots+x_{101}^{2}}

. Therefore,

x_{1}S\leq x_{1}\sqrt{x_{2}^{2}+\cdots+x_{101}^{2}}

. But then, the right-hand side is

x_{1}\sqrt{1-x_{1}^{2}}

. So we consider the maximum of

x_{1}\sqrt{1-x_{1}^{2}}

. Let

f(x)=x\sqrt{1-x^{2}}

. Taking derivative:

f^{\prime}(x)=\sqrt{1-x^{2}}+x\cdot\frac{1}{2\sqrt{1-x^{2}}}\cdot(-2x)=\sqrt{1-x^{2}}-\frac{x^{2}}{\sqrt{1-x^{2}}}=\frac{1-2x^{2}}{\sqrt{1-x^{2}}}

. Setting to zero gives

1-2x^{2}=0\Rightarrow x=\frac{1}{\sqrt{2}}

. Then maximum value is

\left(\frac{1}{\sqrt{2}}\right)\sqrt{1-\frac{1}{2}}=\left(\frac{1}{\sqrt{2}}\right)\left(\frac{1}{\sqrt{2}}\right)=\frac{1}{2}

. Therefore, the maximum value is

\frac{1}{2}

. But wait, is this tight? Because we have equality in Cauchy–Schwarz when vectors are proportional. So for equality, we need

(x_{1},0,\ldots,0)

and

(0,x_{2},\ldots,x_{101})

to be proportional, which would mean that

x_{2}=x_{3}=\cdots=x_{101}

. Let me check. If all

x_{2}

x_{101}

are equal, denote them as

y

. Then the sum of squares is

x_{1}^{2}+100y^{2}=1

. The expression to maximize is

x_{1}\cdot 100y

. From the earlier result, maximum of

x_{1}\cdot 100y

100\cdot\frac{1}{2}=50

? Wait, no. Earlier we found that

x_{1}\sqrt{1-x_{1}^{2}}

has maximum

\frac{1}{2}

, but here

\sqrt{1-x_{1}^{2}}=\sqrt{100y^{2}}=10y

. Therefore,

x_{1}\cdot 10y\leq\frac{1}{2}

. Therefore, the maximum of

x_{1}\cdot 10y

\frac{1}{2}

. Therefore, the total sum

x_{1}(x_{2}+\cdots+x_{101})

100x_{1}y

, so

100x_{1}y\leq\frac{1}{2}

. Therefore, maximum value is

\frac{1}{2}

. Therefore, the answer is

\boxed{\tfrac{1}{2}}

. Final Answer

\boxed{\tfrac{1}{2}}

</think>

Appendix

Table 11: Comparisons of completions generated by four methods on questions sampled from the test set of MATH dataset. All responses are produced using the MATH-trained Qwen3-8B model. ✓ and ✗ denote correct and incorrect responses, respectively.

CPPO Response ✗ :

<think> Okay, so I need to find the maximum value of the sum

x_{1}x_{2}+x_{1}x_{3}+\cdots+x_{1}x_{101}

, given that the sum of the squares of all the

x_{i}

’s is

1

. Let me think about how to approach this. First, the problem is about maximizing a sum of products involving

x_{1}

with each of the other variables. The constraint is on the sum of squares. This seems like an optimization problem with a constraint, so maybe I can use Lagrange multipliers? Or perhaps there’s a way to apply the Cauchy–Schwarz inequality or some other inequality? Let me write down what I need to maximize:

S=x_{1}x_{2}+x_{1}x_{3}+\cdots+x_{1}x_{101}

. So that’s

x_{1}

multiplied by the sum of

x_{2}

through

x_{101}

. Let me denote that sum as

T=x_{2}+x_{3}+\cdots+x_{101}

. Then

S=x_{1}T

. So the problem becomes maximizing

x_{1}T

, given that

x_{1}^{2}+x_{2}^{2}+\cdots+x_{101}^{2}=1

. Hmm. Maybe I can express

T

in terms of the other variables? Or maybe use Cauchy–Schwarz on

T

? Alternatively, think of

S

x_{1}\left(\sum_{i=2}^{101}x_{i}\right)

. Let me consider variables

x_{2},x_{3},\ldots,x_{101}

. Let me denote them as

y_{1},y_{2},\ldots,y_{100}

, so that

T=y_{1}+y_{2}+\cdots+y_{100}

. Then the constraint becomes

x_{1}^{2}+y_{1}^{2}+y_{2}^{2}+\cdots+y_{100}^{2}=1

. So we need to maximize

x_{1}(y_{1}+\cdots+y_{100})

with the constraint

x_{1}^{2}+\sum y_{i}^{2}=1

. This seems like a standard problem. Maybe use Cauchy–Schwarz on the terms

x_{1}

and the sum of

y_{i}

? Wait, let me think. Let me consider vectors. Suppose I have two vectors: one is

(x_{1},y_{1},y_{2},\ldots,y_{100})

and another… Hmm, maybe not. Alternatively, think of the sum

x_{1}(y_{1}+\cdots+y_{100})

as the dot product of

(x_{1},0,0,\ldots,0)

and

(y_{1},y_{2},\ldots,y_{100})

. But I don’t know if that helps directly. Alternatively, use Cauchy–Schwarz on the terms

x_{1}

and the sum of

y_{i}

. Let me recall that for any vectors

\mathbf{u}

and

\mathbf{v}

|\mathbf{u}\cdot\mathbf{v}|\leq\|\mathbf{u}\|\,\|\mathbf{v}\|

. If I take

\mathbf{u}=(x_{1},0,0,\ldots,0)

and

\mathbf{v}=(y_{1},y_{2},\ldots,y_{100})

, then their dot product is

x_{1}(y_{1}+\cdots+y_{100})=S

. The norm of

\mathbf{u}

\sqrt{x_{1}^{2}}=x_{1}

. The norm of

\mathbf{v}

\sqrt{y_{1}^{2}+\cdots+y_{100}^{2}}

. Therefore, by Cauchy–Schwarz:

S\leq x_{1}\sqrt{y_{1}^{2}+\cdots+y_{100}^{2}}

. But the constraint is

x_{1}^{2}+y_{1}^{2}+\cdots+y_{100}^{2}=1

. Let me denote that sum of squares of

y_{i}

Q

. Then

Q=1-x_{1}^{2}

. Therefore:

S\leq x_{1}\sqrt{1-x_{1}^{2}}

. So now, the problem reduces to maximizing

x_{1}\sqrt{1-x_{1}^{2}}

over

x_{1}>0

. Let me set

f(x)=x\sqrt{1-x^{2}}

. To find its maximum, take derivative:

f^{\prime}(x)=\sqrt{1-x^{2}}+x\left(\frac{1}{2\sqrt{1-x^{2}}}\right)(-2x)=\sqrt{1-x^{2}}-\frac{x^{2}}{\sqrt{1-x^{2}}}=\frac{(1-x^{2})-x^{2}}{\sqrt{1-x^{2}}}=\frac{1-2x^{2}}{\sqrt{1-x^{2}}}

. Set derivative equal to zero:

1-2x^{2}=0\Rightarrow x^{2}=\frac{1}{2}\Rightarrow x=\frac{1}{\sqrt{2}}

. Therefore, maximum value is

f\!\left(\frac{1}{\sqrt{2}}\right)=\left(\frac{1}{\sqrt{2}}\right)\sqrt{1-\frac{1}{2}}=\left(\frac{1}{\sqrt{2}}\right)\left(\frac{1}{\sqrt{2}}\right)=\frac{1}{2}

. Therefore, the answer is

\boxed{\tfrac{1}{2}}

. Final Answer

\boxed{\tfrac{1}{2}}

</think>

DPPO Response (Ours) ✓ :

<think> Okay, so I need to find the maximum value of the sum

x_{1}x_{2}+x_{1}x_{3}+\cdots+x_{1}x_{101}

, given that the sum of the squares of

x_{1}

x_{101}

1

. All the

x_{i}

are positive real numbers. Hmm, let me think about how to approach this. First, the expression we need to maximize is the sum of all products of

x_{1}

with each of the other variables. So that’s

x_{1}

multiplied by

(x_{2}+x_{3}+\cdots+x_{101})

. Let me denote

S=x_{2}+x_{3}+\cdots+x_{101}

. Then the expression becomes

x_{1}\cdot S

. Now, we know from the given condition that

x_{1}^{2}+x_{2}^{2}+\cdots+x_{101}^{2}=1

. So maybe I can relate

S

and

x_{1}

using some inequality. Since all variables are positive, perhaps Cauchy-Schwarz inequality could be useful here. By Cauchy-Schwarz,

(x_{2}^{2}+x_{3}^{2}+\cdots+x_{101}^{2})(1^{2}+1^{2}+\cdots+1^{2})\geq(x_{2}+x_{3}+\cdots+x_{101})^{2}

. There are

100

terms in the sum for

x_{2}^{2}

x_{101}^{2}

, so that’s

100

terms. Therefore,

(\sum_{i=2}^{101}x_{i}^{2})\cdot 100\geq S^{2}

. Hence,

S^{2}\leq 100*(1-x_{1}^{2})

. Therefore,

S\leq 10*\sqrt{1-x_{1}^{2}}

. So our expression

x_{1}*S\leq x_{1}*10*\sqrt{(}1-x_{1}^{2})

. Now, we need to maximize this expression with respect to

x_{1}

. Let me set

f(x)=10x*\sqrt{(}1-x^{2})

. To find its maximum, take derivative:

f^{\prime}(x)=10*\sqrt{(}1-x^{2})+10x*(1/(2*\sqrt{(}1-x^{2})))*(-2x)=10*\sqrt{(}1-x^{2})-10x^{2}/\sqrt{(}1-x^{2})=[10(1-x^{2})-10x^{2}]/\sqrt{(}1-x^{2})=10(1-2x^{2})/\sqrt{(}1-x^{2})

. Set derivative equal to zero:

1-2x^{2}=0\Rightarrow x^{2}=1/2\Rightarrow x=1/\sqrt{2}

. Since

x

is positive. So maximum value is

10*(1/\sqrt{2})*\sqrt{(}1-1/2)=10*(1/\sqrt{2})*(1/\sqrt{2})=10*(1/2)=5

. Therefore, the maximum value is

5

. So the answer is

\boxed{5}

. Final Answer

\boxed{5}

</think>

Appendix

Table 16

Explore similar work

Sep 29, 2026

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07

\times

training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at https://github.com/Nicozwy/FastRL.

Jiahua Yang, Zhiwei Yang, Xianpeng Zhang +7

Guangdong Institute of Smart Education, Jinan University, Guangzhou, China · OPPO AI Center, Shenzhen, China · Ragentile Intelligence Inc, Edmonton, Canada

Group Relative Policy Optimization RL for VLMs

Ask PDF

May 29, 2026

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.

Yiming Ren, Yiran Xu, Zicheng Lin +8

Tsinghua University · Shanghai AI Laboratory · The Chinese University of Hong Kong +1

Reinforcement Learning RL for Language Model Reasoning Group Relative Policy Optimization

Ask PDF

May 6, 2026

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level granularity that ignores heterogeneous informational value, uniform polarity that penalizes correct steps and rewards incorrect ones, and zero-variance collapse that erases outcome-driven gradients. We systematically quantify these failures, revealing highly non-uniform token informativeness, widespread step-level polarity misalignment, and substantial training waste. To address these limitations, we propose Entropy-Progress Aligned GRPO (EP-GRPO), a framework that mines the model's intrinsic information flow for dense, self-supervised guidance. EP-GRPO integrates entropy-gated modulation to prioritize high entropy decision pivots, implicit process signals from policy divergence anchored to outcome advantages for directional token-level feedback without external reward models, and cumulative entropy mapping that enables progress-aligned advantage normalization, naturally maintaining gradient flow under zero reward variance. Extensive experiments on mathematical reasoning benchmarks demonstrate that EP-GRPO achieves superior accuracy and efficiency compared to GRPO and its variants. The code will be available.

Song Yu, Li Li, Wenwen Zhao +1

School of Computer and Information Science, Southwest University, Chongqing 400715, China

RL for Language Model Reasoning Group Relative Policy Optimization Credit Assignment in RL Reinforcement Learning with Verifiable Rewards

Ask PDF