When KL Regularization Misfires in Group Policy Optimization

Fei Ding

cs.LGOct 8, 2026

When KL Regularization Misfires in Group Policy Optimization

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.

Figures & tables

Figure 1: KL placement and seven types of mismatch. The right panel illustrates the coefficient decomposition ( $d_{i}=\langle\kappa\rangle_{i}=\widetilde{D}_{i}$ , $K_{i}^{\kappa}=K_{i}$ ): a uniform shift represents the common KL offset, and the gold arrows indicate the relative calibration contribution. $S$ denotes the base surrogate, $\operatorname{sg}$ stops gradients, $r=\pi_{\theta}/\pi_{\rm old}$ is the probability ratio for a sampled token, and $\beta\geq 0$ is the KL weight; the independent term is illustrated using $\ell^{\rm KL}=e^{-\Delta}+\Delta-1$ ( $k_{3}$ ). $A_{i}=R_{i}-\bar{R}$ , $\hat{A}_{i}=A_{i}/(\sigma_{q}+\varepsilon_{\rm adv})$ , and $s_{q}=\max_{j}|A_{j}|$ ; $R,\sigma_{q},T,h,G,q$ denote the reward, reward standard deviation, length, prefix, group size, and prompt, respectively; overbars indicate group means, and $\varepsilon_{\rm adv}>0$ . Conditions illustrated: F2 assumes the same context/token, all gates open, and equal weights; F3 assumes unfiltered groups; F4 assumes no length averaging and $\ell^{\rm KL}_{i,t}\approx c>0$ ; F5 assumes nonzero denominators; F7 assumes $G\geq 2$ independent single-token responses sampled from the current policy for the same prompt, with distinct, full-support $\pi_{\theta},\pi_{\rm ref}$ . The coefficient objective first skips groups with $G<2$ or $s_{q}=0$ .

Figure 2: ZCPO coefficient construction. $A_{i}=R_{i}-\bar{R}$ , $B_{q}=\mathbf{1}\{G\geq 2,\ s_{q}>0\}$ . All tokens in a response share $\operatorname{sg}(C_{i})$ . Zero-sum coefficients do not imply a zero total gradient; conditional KL eliminates only token sampling noise conditional on a given prefix.

Model/method	AIME24 (%)	AIME25 (%)
DAPO without KL ( $\beta=0$ )	$47.3$	$37.1$
DAPO + standard KL, $\beta=0.04$	$31.2$	$18.9$
DAPO + standard KL, $\beta=0.001$	$44.5$	$34.7$
ZCPO	$\mathbf{56.8}$	$\mathbf{48.5}$
No KL centering ( $K_{i}\leftarrow D_{i}/T_{i}$ )	$45.6$	$36.7$
No KL length norm. ( $K_{i}\leftarrow D_{i}-\frac{1}{G}\sum_{j=1}^{G}D_{j}$ )	$53.9$	$43.5$

Table 1: Method comparisons, ablations, and F1 controls. Both AIME24 and AIME25 report avg@32 (%), with means over 3 random seeds. Each ZCPO ablation independently substitutes the definition shown in parentheses; all other training settings match full ZCPO.

Figure 3: KL failure-mode diagnostics. (a) F1: KL gradient share at clipped tokens. (b) F2: KL/total gradient norm ratios for groups in the top/bottom 10% by reward-gradient cancellation. (c) F4: at step 100, vanilla has mean lengths of 774 and 611 tokens at $\beta=0.04$ and $4$ , versus 1695 for ZCPO at $\beta=4$ . (d) F5: KL gradient shares in the shortest/longest 10% of groups. (e,f) F6: KL signal concentration and first-token drift. (g,h) F7: within-group dispersion of mean response KL and negative-estimate fractions, both before centering. Dashed/dotted curves in (g) give small-KL sampling-noise reference scales; the dashed curve in (h) gives the normal approximation for the k1 control. Vanilla denotes DAPO with independent KL loss.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration item	Setting
Training framework	verl
Training data	DAPO-Math-17K: approximately 17K prompts with integer answers
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Learning-rate warmup	Linear warmup over the first 20 rollout steps
Learning-rate schedule after warmup	Constant

Appendix

Table 2: Base training and evaluation settings and KL coefficients for the main experiments.

$\bm{\beta}$	AIME24 (%)
$0$	$48.1$
$4$	$56.8$
$20$	$53.7$

Appendix

Table 3: ZCPO sensitivity to

\beta

: AIME24 avg@32 (%) on Qwen2.5-32B, reporting means over 3 random seeds. The result at

\beta=4

is reproduced from Table 1 .

Reward normalization	KL calibration off ( $\beta=0$ )	KL calibration on ( $\beta=4$ )
Standard deviation	$47.3$	$55.4$
Maximum absolute advantage	$48.1$	$56.8$

Appendix

Table 4: Joint comparison of reward normalization and KL-coefficient calibration. Entries are collected from Tables 1 and 3 , reporting Qwen2.5-32B AIME24 avg@32 (%) means over 3 random seeds.

Method	$\bm{\beta}$	KL/total gradient norm ratio
vanilla	0.001	0.4%
vanilla	0.04	5.8%
ZCPO	4	7.0%
ZCPO	20	30.2%

Appendix

Table 5: Median ratio of the KL gradient norm to the total gradient norm in the diagnostic runs (steps 11–100).

Figure 4: Conditional sampling-noise comparisons. (a) Using the first mini-batch at each of the first 20 steps of ZCPO, the orange curve shows the residual RMSE of the k1 coefficient relative to ZCPO, and the green curve shows ZCPO’s own zero residual. ZCPO’s zero conditional sampling error refers only to the local estimation term obtained by replacing sampled $\Delta_{i,t}$ with $\kappa_{i,t}$ for a given prefix; variation in sampled prefixes remains. (b) GVPO training curves. “Noise” denotes the sampling residual of the log-ratio relative to its per-prefix conditional mean. Its magnitude is the measured RMS of $D_{i}^{\rm k1}-D_{i}=\sum_{t}(\Delta_{i,t}-\kappa_{i,t})$ after within-group centering, without multiplication by the KL weight. The green dashed line is the zero reference for conditional sampling error.

Model	DocMath	LBV2	Frames	MRCR	CorpusQA	LBV1-QA
Qwen3-4B-Thinking-2507	60.7	39.4	64.9	37.5	49.2	64.2
+ (GRPO + standard KL)	61.6	41.7	64.8	49.3	53.8	64.3
+ (GRPO without KL)	62.3	45.2	66.4	65.7	61.9	65.6
+ (GRPO-ZCPO)	63.8	48.2	68.2	66.0	69.6	66.4
+ (GRPO-ZCPO, $B_{q}=1$ )	63.3	48.0	67.6	66.3	68.5	66.1

Appendix

Table 6: Long-context evaluation results and group-gating ablation for Qwen3-4B. All methods trained in this work use the same data subset.

B_{q}=1

denotes the ablation that removes gating for groups with identical rewards, with all other ZCPO settings unchanged.

Data
Hyperparameter	Value
Max prompt length	160K
Max response length	16K
Responses per prompt	16
Optimization
Learning rate	2e-6

Appendix

Table 7: Training hyperparameters for the long-context experiments.

Parameter	Setting
Model, Data, and Method
Starting model	Qwen3-4B-Thinking-2507
Training data	A fixed subset of 8,000 GoLongRL examples; the same subset for all methods
Training framework and entry point	verl; recipe.dapo.main_dapo
Trainer configuration	dapo_megatron_trainer
Update coefficient	GRPO uses group-relative advantages; ZCPO uses the joint coefficient $C_{i}$ defined in the main text

Appendix

Table 8: Additional configuration for the Qwen3-4B long-context experiments. KL and group-gating settings that differ across methods are listed separately.

Model/method	AIME24 avg@32 (%)
GLM-4-9B-0414 + DAPO without KL	$24.5$
GLM-4-9B-0414 + ZCPO	$\mathbf{36.2}$

Appendix

Table 9: Cross-model math results on GLM-4-9B-0414 (AIME24 avg@32, %).

Explore similar work

Group Adaptive Clipping Policy Optimization
Aug 31, 2026Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan +1Group Relative Policy Optimization Policy Optimization
University of Toronto · Amazon
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
Constrained Group Relative Policy Optimization
Feb 5, 2026Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3Group Relative Policy Optimization Constrained RL
Mila – Quebec AI Institute Université de Montréal

Aug 31, 2026, cs.LGJ/K move · Enter open · S save

Group Adaptive Clipping Policy Optimization

Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft

University of Toronto · Amazon

Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards Trust Region Policy Optimization Policy Learning

Open paper AskPDF arXiv

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.

Abstract

Explore similar work

Group Adaptive Clipping Policy Optimization

On KL-Regularized Policy Optimization

Constrained Group Relative Policy Optimization