cs.LGMar 15, 2026

From log⁡π\log π to ππ: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight

Authors: Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, 

Organizations: Meituan · Fudan University · Tsinghua University · Peking University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed a leap in Large Language Model (LLM) reasoning, yet its optimization dynamics remain fragile. Standard algorithms like GRPO enforce stability via "hard clipping", which inadvertently stifles exploration by discarding gradients of tokens outside the trust region. While recent "soft clipping" methods attempt to recover these gradients, they suffer from a critical challenge: relying on log-probability gradient (∇θlog⁡πθ\nabla_θ\log π_θ) yields divergent weights as probabilities vanish, destabilizing LLM training. We rethink this convention by establishing probability gradient (∇θπθ\nabla_θπ_θ) as the superior optimization primitive. Accordingly, we propose Decoupled Gradient Policy Optimization (DGPO), which employs a decoupled decay mechanism based on importance sampling ratios. By applying asymmetric, continuous decay to boundary tokens, DGPO resolves the conflict between stability and sustained exploration. Extensive experiments across DeepSeek-R1-Distill-Qwen series models (1.5B/7B/14B) demonstrate that DGPO consistently outperforms strong baselines on various mathematical benchmarks, offering a robust and scalable solution for RLVR. Our code and implementation are available at: https://github.com/FlyTune/DGPO-RL.

Figures & tables

Figure 1: Schematic overview of our DGPO algorithm. While “hard clipping” methods (GRPO/ASPO) discard gradients at boundaries, and prior “soft clipping” approaches (CISPO/GPPO/CE-GPPO) risk divergence on the left boundary and limit exploration on the right boundary, DGPO optimizes the gradient decay mechanism accordingly. By enforcing a controlled “Slow Down” behavior for stability and a “Slow Down Gently” behavior to sustain exploration, DGPO effectively resolves the exploration-stability conflict while maintaining minimal bias against the true policy gradient.
CaseAbbrev.Condition
Left BoundaryLNwi,t(θ)<1−εloww_{i,t}(\theta)<1-\varepsilon_{\text{low}}
(Low ratio, Neg. adv.)∧ A^i<0\land\ \hat{A}_{i}<0
Right BoundaryHPwi,t(θ)>1+εhighw_{i,t}(\theta)>1+\varepsilon_{\text{high}}
(High ratio, Pos. adv.)∧ A^i>0\land\ \hat{A}_{i}>0
Reverse Left BoundaryLPwi,t(θ)<1−εloww_{i,t}(\theta)<1-\varepsilon_{\text{low}}
(Low ratio, Pos. adv.)∧ A^i>0\land\ \hat{A}_{i}>0
Table 1: Definition of clipping regions based on IS ratio wi,t(θ)w_{i,t}(\theta) and advantage A^i\hat{A}_{i} . Abbrev. denotes the abbreviation of five cases under different conditions.
Figure 2: Comparative analysis of gradient dynamics. We systematically contrast DGPO with the standard GRPO, prior “soft clipping” enhancements (CISPO, GPPO, and CE-GPPO), and importance sampling improvements (ASPO). The visualization highlights the theoretical properties regarding the exploration capability of clipped tokens and the alignment with the true policy gradient, demonstrating DGPO’s superior stability and gradient consistency.
Bias ConditionMagnitude Comparison
In-Boundary Bias0={\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{M}}}=\text{Bias}_{\text{GRPO}}^{\text{M}}=\text{Bias}_{\text{CISPO}}^{\text{M}}=\text{Bias}_{\text{GPPO}}^{\text{M}}=\text{Bias}_{\text{CE}}^{\text{M}}\leq\text{Bias}_{\text{ASPO}}^{\text{M}}
Left-Boundary Bias\begin{cases}0<{\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{LN}}}<\text{Bias}_{\text{CISPO}}^{\text{LN}}=\text{Bias}_{\text{GPPO}}^{\text{LN}}<\text{Bias}_{\text{CE}}^{\text{LN}}<\text{Bias}_{\text{GRPO}}^{\text{LN}}=\text{Bias}_{\text{ASPO}}^{\text{LN}},&n=1\\ 0<\text{Bias}_{\text{CISPO}}^{\text{LN}}=\text{Bias}_{\text{GPPO}}^{\text{LN}}<{\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{LN}}}<\text{Bias}_{\text{CE}}^{\text{LN}}<\text{Bias}_{\text{GRPO}}^{\text{LN}}=\text{Bias}_{\text{ASPO}}^{\text{LN}},&n>1\end{cases}
Right-Boundary Bias0<{\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{HP}}}\leq\text{Bias}_{\text{CISPO}}^{\text{HP}}=\text{Bias}_{\text{GPPO}}^{\text{HP}}=\text{Bias}_{\text{CE}}^{\text{HP}}<\text{Bias}_{\text{GRPO}}^{\text{HP}}=\text{Bias}_{\text{ASPO}}^{\text{HP}}
Reverse Left-Boundary Bias0={\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{LP}}}=\text{Bias}_{\text{GRPO}}^{\text{LP}}=\text{Bias}_{\text{GPPO}}^{\text{LP}}=\text{Bias}_{\text{CE}}^{\text{LP}}<\text{Bias}_{\text{CISPO}}^{\text{LP}}=\text{Bias}_{\text{ASPO}}^{\text{LP}}
Reverse Right-Boundary Bias0={\color[rgb]{0.7539,0,0}\text{Bias}_{\text{DGPO}}^{\text{HN}}}=\text{Bias}_{\text{GRPO}}^{\text{HN}}=\text{Bias}_{\text{GPPO}}^{\text{HN}}=\text{Bias}_{\text{CE}}^{\text{HN}}<\text{Bias}_{\text{CISPO}}^{\text{HN}}=\text{Bias}_{\text{ASPO}}^{\text{HN}}
Table 2: Comparative analysis of gradient bias. We evaluate the bias magnitudes of DGPO and other baselines relative to the true policy gradient under different boundary conditions.
MethodAIME24AIME25AMC23MATH500MinervaOlympiadAvg.
A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32
DeepSeek-R1-Distill-Qwen-1.5B
GRPO33.271.827.749.979.594.877.690.826.148.846.364.748.470.1
CISPO34.869.125.853.376.994.976.891.826.554.245.865.847.871.5
GPPO29.660.523.551.973.594.176.389.126.650.043.964.245.668.3
CE-GPPO35.170.227.755.182.595.076.790.227.850.545.663.149.270.7
Table 3: Comparison results of different methods on various benchmarks. Avg@32 (%) and Pass@32 (%) are abbreviated as A@32 and P@32. The best results are bold , and the second-best results are underlined , respectively.
Figure 3: Comprehensive comparison of training dynamics and performance. Top row: DeepSeek-R1-Distill-Qwen-1.5B results. Bottom row: DeepSeek-R1-Distill-Qwen-7B results. Columns (L-R): Pass@K on AIME25, Avg@32 on AIME24/25, policy entropy, followed by hyperparameter analysis for Avg@32 and entropy.
Figure 4: Comparison of gradient weight distributions (Prob: Probability, Grad: Gradient). (a) Overall distribution. (b) Detailed scatter plots of grad weight vs. IS ratio (top row) and prob (bottom row) across three methods: GRPO (left), GPPO (middle), and DGPO (right). Points are colored by advantage. (c) Boundary distribution analysis.
Method1.5B7B14B 1 1 1 The performance fluctuation is attributed to the base model difference: 1.5B/7B models are derived from Qwen-2.5-Math, whereas the 14B model is derived from Qwen-2.5-Base.
(A / P)(A / P)(A / P)
GRPO48.4 / 70.158.9 / 77.253.6 / 67.4
DGPO52.7 / 73.062.0 / 77.956.7 / 70.4
Improvement+4.3 / +2.9+3.1 / +0.7+3.1 / +3.0
Table 4: Scalability analysis of average Avg@32 (A) and Pass@32 (P) across 1.5B, 7B, and 14B models.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Bias TypeMathematical Definition
In-Boundary BiasBiasAlgoM=∥∇θJAlgoM(θ)−∇θJPGM(θ)∥\text{Bias}_{\text{Algo}}^{\text{M}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{Algo}}^{\text{M}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{M}}(\theta)\right\|
Left-Boundary BiasBiasAlgoLN=∥∇θJAlgoLN(θ)−∇θJPGLN(θ)∥\text{Bias}_{\text{Algo}}^{\text{LN}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{Algo}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|
Right-Boundary BiasBiasAlgoHP=∥∇θJAlgoHP(θ)−∇θJPGHP(θ)∥\text{Bias}_{\text{Algo}}^{\text{HP}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{Algo}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|
Reverse Left-Boundary BiasBiasAlgoLP=∥∇θJAlgoLP(θ)−∇θJPGLP(θ)∥\text{Bias}_{\text{Algo}}^{\text{LP}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{Algo}}^{\text{LP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LP}}(\theta)\right\|
Reverse Right-Boundary BiasBiasAlgoHN=∥∇θJAlgoHN(θ)−∇θJPGHN(θ)∥\text{Bias}_{\text{Algo}}^{\text{HN}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{Algo}}^{\text{HN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HN}}(\theta)\right\|
Appendix
Table 5: Definition of bias relative to Policy Gradient
AlgorithmIn-Boundary Bias Magnitude
GRPO, CISPO,BiasXM=∥∇θJXM(θ)−∇θJPGM(θ)∥=0\displaystyle\text{Bias}_{\text{X}}^{\text{M}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{X}}^{\text{M}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{M}}(\theta)\right\|=0
GPPO, CE-GPPO
DGPO
ASPOBiasASPOM=∥Eq∼D, o∼πθold(⋅∣q)[vtM⋅(1rt−rt)⋅At∇θlog⁡πθ(ot∣q,o<t)]∥≈∥Eq∼D, o∼πθold(⋅∣q)[vtM⋅−2(rt−1)⏟λt⋅At∇θlog⁡πθ(ot∣q,o<t)]∥≠0∗\begin{aligned} \text{Bias}_{\text{ASPO}}^{\text{M}}&=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\left[v_{t}^{\text{M}}\cdot\left(\frac{1}{r_{t}}-r_{t}\right)\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\\ &\approx\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\left[v_{t}^{\text{M}}\cdot\underbrace{-2(r_{t}-1)}_{\lambda_{t}}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\neq 0^{*}\end{aligned}
Appendix
Table 6: In-Boundary bias among various policy optimization algorithms
AlgorithmLeft-Boundary Bias Magnitude
GRPOBiasGRPOLN=∥∇θJGRPOLN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold[vtLN⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{GRPO}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{GRPO}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
CISPOBiasCISPOLN=∥∇θJCISPOLN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold[vtLN⋅(r0−rt)⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{CISPO}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot(r_{0}-r_{t})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
GPPOBiasGPPOLN=∥∇θJGPPOLN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold[vtLN⋅(r0−rt)⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{GPPO}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{GPPO}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot(r_{0}-r_{t})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
CE-GPPOBiasCELN=∥∇θJCELN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold{vtLN⋅[β1r0−rt]⋅At∇θlog⁡πθ(ot∣q,o<t)}∥\begin{aligned} \text{Bias}_{\text{CE}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{CE}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left\{v_{t}^{\text{LN}}\cdot[\beta_{1}r_{0}-r_{t}]\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right\}\right\|\end{aligned}
ASPOBiasASPOLN=∥∇θJASPOLN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold[vtLN⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{ASPO}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{ASPO}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
DGPOBiasDGPOLN=∥∇θJDGPOLN(θ)−∇θJPGLN(θ)∥=∥Eq∼D, o∼πθold[vtLN⋅(rt−rtn+1r0n)⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{DGPO}}^{\text{LN}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{DGPO}}^{\text{LN}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot(r_{t}-\frac{r_{t}^{n+1}}{r_{0}^{n}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
Appendix
Table 7: Left-Boundary bias among various policy optimization algorithms
 {∇θJPGLN(θ)=Eq∼D, o∼πθold[vtLN⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGRPOLN(θ)=Eq∼D, o∼πθold[vtLN⋅0⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCISPOLN(θ)=Eq∼D, o∼πθold[vtLN⋅(1−εlow)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGPPOLN(θ)=Eq∼D, o∼πθold[vtLN⋅(1−εlow)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCELN(θ)=Eq∼D, o∼πθold[vtLN⋅β1(1−εlow)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJASPOLN(θ)=Eq∼D, o∼πθold[vtLN⋅0⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJDGPOLN(θ)=Eq∼D, o∼πθold[vtLN⋅rtn+1(1−εlow)n⋅At∇θlog⁡πθ(ot∣q,o<t)]\begin{cases}\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GRPO}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot 0\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot(1-\varepsilon_{\text{low}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GPPO}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot(1-\varepsilon_{\text{low}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CE}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot\beta_{1}(1-\varepsilon_{\text{low}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{ASPO}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot 0\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{DGPO}}^{\text{LN}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LN}}\cdot\frac{r_{t}^{n+1}}{(1-\varepsilon_{\text{low}})^{n}}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\end{cases} (37)
Appendix
Table 12
AlgorithmRight-Boundary Bias Magnitude
GRPOBiasGRPOHP=∥∇θJGRPOHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold[vtHP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{GRPO}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{GRPO}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
CISPOBiasCISPOHP=∥∇θJCISPOHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold{vtHP⋅[rt−(1+εhigh)]⋅At∇θlog⁡πθ(ot∣q,o<t)}∥\begin{aligned} \text{Bias}_{\text{CISPO}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left\{v_{t}^{\text{HP}}\cdot[r_{t}-(1+\varepsilon_{\text{high}})]\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right\}\right\|\end{aligned}
GPPOBiasGPPOHP=∥∇θJGPPOHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold{vtHP⋅[rt−(1+εhigh)]⋅At∇θlog⁡πθ(ot∣q,o<t)}∥\begin{aligned} \text{Bias}_{\text{GPPO}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{GPPO}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left\{v_{t}^{\text{HP}}\cdot[r_{t}-(1+\varepsilon_{\text{high}})]\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right\}\right\|\end{aligned}
CE-GPPOBiasCEHP=∥∇θJCEHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold{vtHP⋅[rt−β2(1+εhigh)]⋅At∇θlog⁡πθ(ot∣q,o<t)}∥\begin{aligned} \text{Bias}_{\text{CE}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{CE}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left\{v_{t}^{\text{HP}}\cdot[r_{t}-\beta_{2}(1+\varepsilon_{\text{high}})]\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right\}\right\|\end{aligned}
ASPOBiasASPOHP=∥∇θJASPOHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold[vtHP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{ASPO}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{ASPO}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
DGPOBiasDGPOHP=∥∇θJDGPOHP(θ)−∇θJPGHP(θ)∥=∥Eq∼D, o∼πθold[vtHP⋅[rt−(1+εhigh)1mrt1−1m]⋅At∇θlog⁡πθ(ot∣q,o<t)]∥\begin{aligned} \text{Bias}_{\text{DGPO}}^{\text{HP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{DGPO}}^{\text{HP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot[r_{t}-(1+\varepsilon_{\text{high}})^{\frac{1}{m}}r_{t}^{1-\frac{1}{m}}]\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\end{aligned}
Appendix
Table 8: Right-Boundary bias among various policy optimization algorithms
 {∇θJPGHP(θ)=Eq∼D, o∼πθold[vtHP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGRPOHP(θ)=Eq∼D, o∼πθold[vtHP⋅0⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCISPOHP(θ)=Eq∼D, o∼πθold[vtHP⋅(1+εhigh)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGPPOHP(θ)=Eq∼D, o∼πθold[vtHP⋅(1+εhigh)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCEHP(θ)=Eq∼D, o∼πθold[vtHP⋅β2(1+εhigh)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJASPOHP(θ)=Eq∼D, o∼πθold[vtHP⋅0⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJDGPOHP(θ)=Eq∼D, o∼πθold[vtHP⋅(1+εhigh)1mrt1−1m⋅At∇θlog⁡πθ(ot∣q,o<t)]\begin{cases}\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GRPO}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot 0\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot(1+\varepsilon_{\text{high}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GPPO}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot(1+\varepsilon_{\text{high}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CE}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot\beta_{2}(1+\varepsilon_{\text{high}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{ASPO}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot 0\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{DGPO}}^{\text{HP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{HP}}\cdot(1+\varepsilon_{\text{high}})^{\frac{1}{m}}r_{t}^{1-\frac{1}{m}}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\end{cases} (38)
Appendix
Table 14
AlgorithmReverse Left-Boundary Bias Magnitude
GRPO, GPPOBiasXLP=∥∇θJXLP(θ)−∇θJPGLP(θ)∥=0\begin{aligned} \text{Bias}_{\text{X}}^{\text{LP}}=\left\|\nabla_{\theta}\mathcal{J}_{\text{X}}^{\text{LP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LP}}(\theta)\right\|=0\end{aligned}
CE-GPPO, DGPO
CISPO, ASPOBiasCISPOLP=∥∇θJCISPOLP(θ)−∇θJPGLP(θ)∥=∥Eq∼D, o∼πθold[vtLP⋅(1−εlow−rt)⋅At∇θlog⁡πθ(ot∣q,o<t)]∥≠0∗\begin{aligned} \text{Bias}_{\text{CISPO}}^{\text{LP}}&=\left\|\nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{LP}}(\theta)-\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LP}}(\theta)\right\|\\ &=\left\|\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot(1-\varepsilon_{\text{low}}-r_{t})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\right\|\neq 0^{*}\end{aligned}
Appendix
Table 9: Reverse Left-Boundary bias among various policy optimization algorithms
 {∇θJPGLP(θ)=Eq∼D, o∼πθold[vtLP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGRPOLP(θ)=Eq∼D, o∼πθold[vtLP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCISPOLP(θ)=Eq∼D, o∼πθold[vtLP⋅(1−εlow)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJGPPOLP(θ)=Eq∼D, o∼πθold[vtLP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJCELP(θ)=Eq∼D, o∼πθold[vtLP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJASPOLP(θ)=Eq∼D, o∼πθold[vtLP⋅(1−εlow)⋅At∇θlog⁡πθ(ot∣q,o<t)]∇θJDGPOLP(θ)=Eq∼D, o∼πθold[vtLP⋅rt⋅At∇θlog⁡πθ(ot∣q,o<t)]\begin{cases}\nabla_{\theta}\mathcal{J}_{\text{PG}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GRPO}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CISPO}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot(1-\varepsilon_{\text{low}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{GPPO}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{CE}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{ASPO}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot(1-\varepsilon_{\text{low}})\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\\ \nabla_{\theta}\mathcal{J}_{\text{DGPO}}^{\text{LP}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta_{\text{old}}}}\left[v_{t}^{\text{LP}}\cdot r_{t}\cdot A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right]\end{cases} (39)
Appendix
Table 16
DatasetCore DescriptionKey Characteristics
AIME 2024The 2024 edition of the American Invitational Mathematics Examination, serving as a bridge between the AMC and the USAMO.• Focuses on arithmetic precision and number theory. • Requires answers in a strict integer format (000–999). • Tests robustness against recent contamination.
AIME 2025The most recent iteration of the AIME competition, representing a strictly “held-out” set for evaluating generalization to unseen problems.• Zero data contamination risk due to recency. • High complexity requiring multi-step logical chains. • Validates the model’s potential for future reasoning.
AMC 2023Selected problems from the 2023 American Mathematics Competitions (AMC 10/12), representing the entry-level olympiad difficulty.• Covers broad topics: Algebra, Geometry, Counting. • Functions as a baseline for competitive math ability. • Requires mapping multiple-choice logic to open-ended generation.
MATH-500A curated subset of 500 representative problems from the widely used MATH dataset, designed by OpenAI for efficient evaluation.• Spans 7 categories including Calculus and Probability. • Reduces evaluation costs while maintaining distribution fidelity. • Heavily relies on LaTeX understanding.
MinervaA collection of technical mathematics problems derived from scientific papers and undergraduate-level coursework.• Involves higher-order symbolic reasoning. • Contains domain-specific vocabulary and notation. • Tests capabilities beyond standard competition math.
OlympiadBenchA comprehensive aggregate of international mathematics competitions (e.g., IMO, CMO) spanning multiple languages and formats.• Represents the upper bound of mathematical reasoning. • Includes theorem proving and fill-in-the-blank types. • Challenges the model’s cross-lingual mathematical logic.
Appendix
Table 11: Detailed comparison of the mathematical reasoning benchmarks used in this study.
Model ScaleAlgorithmLearning RateSpecific Hyperparameters
Common Settings: Mini-batch Size = 32, Rollout Batch Size = 512, Max Length = 8192
DeepSeek-R1-Distill-Qwen-1.5BGRPO1.0×10−61.0\times 10^{-6}εlow=εhigh=0.2\varepsilon_{\text{low}}=\varepsilon_{\text{high}}=0.2
CISPOεlow=εhigh=0.2\varepsilon_{\text{low}}=\varepsilon_{\text{high}}=0.2
GPPOεlow=εhigh=0.2\varepsilon_{\text{low}}=\varepsilon_{\text{high}}=0.2
CE-GPPOεlow=εhigh=0.2,β1=0.75,β2=1\varepsilon_{\text{low}}=\varepsilon_{\text{high}}=0.2,\beta_{1}=0.75,\beta_{2}=1
ASPOεlow=εhigh=0.2,εlow′=0.33,εhigh′=3\varepsilon_{\text{low}}=\varepsilon_{\text{high}}=0.2,\varepsilon^{\prime}_{\text{low}}=0.33,\varepsilon^{\prime}_{\text{high}}=3
Appendix
Table 12: Detailed hyperparameter configurations for all experiments. To ensure a fair comparison, we maintain identical training settings across all algorithms, varying only the learning rate according to the model scale.
Figure 5: Training dynamics of DeepSeek-R1-Distill-Qwen-14B comparing GRPO and DGPO.
MethodAIME24AIME25AMC23MATH500MinervaOlympiadAvg.
A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32A@32P@32
DeepSeek-R1-Distill-Qwen-14B
GRPO56.682.940.567.892.297.466.570.422.635.343.250.753.667.4
DGPO63.386.247.666.493.999.167.075.123.037.945.157.456.770.4
Appendix
Table 13: Full performance comparison on DeepSeek-R1-Distill-Qwen-14B.
More in this topic

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

Explore similar work

CardsList

  1. When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR

    May 19, 2026Yuchun Miao, Sen Zhang, Yuqi Zhang +4RL for Language Model ReasoningPolicy Optimization

    National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University · Meituan Longcat Team · The University of Sydney +1

  2. MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

    Feb 19, 2026Xiaoliang Fu, Jiaye Lin, Yangyi Fang +7RL for Language ModelsRL for Language Model Reasoning

    Meituan · Fudan University · Tsinghua University +2

  3. Rethinking the Divergence Regularization in LLM RL

    Jun 8, 2026Jiarui Yao, Xiangxin Zhou, Penghui Qi +3RL for Language ModelsPolicy Optimization

    1Tencent Hunyuan · 2UIUC · 3NUS