When KL Regularization Misfires in Group Policy Optimization

Fei Ding

cs.LGOct 8, 2026

When KL Regularization Misfires in Group Policy Optimization

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.

Figures & tables

Figure 1: KL placement and seven types of mismatch. The right panel illustrates the coefficient decomposition ( $d_{i}=\langle\kappa\rangle_{i}=\widetilde{D}_{i}$ , $K_{i}^{\kappa}=K_{i}$ ): a uniform shift represents the common KL offset, and the gold arrows indicate the relative calibration contribution. $S$ denotes the base surrogate, $\operatorname{sg}$ stops gradients, $r=\pi_{\theta}/\pi_{\rm old}$ is the probability ratio for a sampled token, and $\beta\geq 0$ is the KL weight; the independent term is illustrated using $\ell^{\rm KL}=e^{-\Delta}+\Delta-1$ ( $k_{3}$ ). $A_{i}=R_{i}-\bar{R}$ , $\hat{A}_{i}=A_{i}/(\sigma_{q}+\varepsilon_{\rm adv})$ , and $s_{q}=\max_{j}|A_{j}|$ ; $R,\sigma_{q},T,h,G,q$ denote the reward, reward standard deviation, length, prefix, group size, and prompt, respectively; overbars indicate group means, and $\varepsilon_{\rm adv}>0$ . Conditions illustrated: F2 assumes the same context/token, all gates open, and equal weights; F3 assumes unfiltered groups; F4 assumes no length averaging and $\ell^{\rm KL}_{i,t}\approx c>0$ ; F5 assumes nonzero denominators; F7 assumes $G\geq 2$ independent single-token responses sampled from the current policy for the same prompt, with distinct, full-support $\pi_{\theta},\pi_{\rm ref}$ . The coefficient objective first skips groups with $G<2$ or $s_{q}=0$ .

Figure 2: ZCPO coefficient construction. $A_{i}=R_{i}-\bar{R}$ , $B_{q}=\mathbf{1}\{G\geq 2,\ s_{q}>0\}$ . All tokens in a response share $\operatorname{sg}(C_{i})$ . Zero-sum coefficients do not imply a zero total gradient; conditional KL eliminates only token sampling noise conditional on a given prefix.

Model/method	AIME24 (%)	AIME25 (%)
DAPO without KL ( $\beta=0$ )	$47.3$	$37.1$
DAPO + standard KL, $\beta=0.04$	$31.2$	$18.9$
DAPO + standard KL, $\beta=0.001$	$44.5$	$34.7$
ZCPO	$\mathbf{56.8}$	$\mathbf{48.5}$
No KL centering ( $K_{i}\leftarrow D_{i}/T_{i}$ )	$45.6$	$36.7$
No KL length norm. ( $K_{i}\leftarrow D_{i}-\frac{1}{G}\sum_{j=1}^{G}D_{j}$ )	$53.9$	$43.5$

Table 1: Method comparisons, ablations, and F1 controls. Both AIME24 and AIME25 report avg@32 (%), with means over 3 random seeds. Each ZCPO ablation independently substitutes the definition shown in parentheses; all other training settings match full ZCPO.

Figure 3: KL failure-mode diagnostics. (a) F1: KL gradient share at clipped tokens. (b) F2: KL/total gradient norm ratios for groups in the top/bottom 10% by reward-gradient cancellation. (c) F4: at step 100, vanilla has mean lengths of 774 and 611 tokens at $\beta=0.04$ and $4$ , versus 1695 for ZCPO at $\beta=4$ . (d) F5: KL gradient shares in the shortest/longest 10% of groups. (e,f) F6: KL signal concentration and first-token drift. (g,h) F7: within-group dispersion of mean response KL and negative-estimate fractions, both before centering. Dashed/dotted curves in (g) give small-KL sampling-noise reference scales; the dashed curve in (h) gives the normal approximation for the k1 control. Vanilla denotes DAPO with independent KL loss.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration item	Setting
Training framework	verl
Training data	DAPO-Math-17K: approximately 17K prompts with integer answers
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Learning-rate warmup	Linear warmup over the first 20 rollout steps
Learning-rate schedule after warmup	Constant

Appendix

Table 2: Base training and evaluation settings and KL coefficients for the main experiments.

$\bm{\beta}$	AIME24 (%)
$0$	$48.1$
$4$	$56.8$
$20$	$53.7$

Appendix

Table 3: ZCPO sensitivity to

\beta

: AIME24 avg@32 (%) on Qwen2.5-32B, reporting means over 3 random seeds. The result at

\beta=4

is reproduced from Table 1 .

Reward normalization	KL calibration off ( $\beta=0$ )	KL calibration on ( $\beta=4$ )
Standard deviation	$47.3$	$55.4$
Maximum absolute advantage	$48.1$	$56.8$

Appendix

Table 4: Joint comparison of reward normalization and KL-coefficient calibration. Entries are collected from Tables 1 and 3 , reporting Qwen2.5-32B AIME24 avg@32 (%) means over 3 random seeds.

Method	$\bm{\beta}$	KL/total gradient norm ratio
vanilla	0.001	0.4%
vanilla	0.04	5.8%
ZCPO	4	7.0%
ZCPO	20	30.2%

Appendix

Table 5: Median ratio of the KL gradient norm to the total gradient norm in the diagnostic runs (steps 11–100).

Figure 4: Conditional sampling-noise comparisons. (a) Using the first mini-batch at each of the first 20 steps of ZCPO, the orange curve shows the residual RMSE of the k1 coefficient relative to ZCPO, and the green curve shows ZCPO’s own zero residual. ZCPO’s zero conditional sampling error refers only to the local estimation term obtained by replacing sampled $\Delta_{i,t}$ with $\kappa_{i,t}$ for a given prefix; variation in sampled prefixes remains. (b) GVPO training curves. “Noise” denotes the sampling residual of the log-ratio relative to its per-prefix conditional mean. Its magnitude is the measured RMS of $D_{i}^{\rm k1}-D_{i}=\sum_{t}(\Delta_{i,t}-\kappa_{i,t})$ after within-group centering, without multiplication by the KL weight. The green dashed line is the zero reference for conditional sampling error.

Model	DocMath	LBV2	Frames	MRCR	CorpusQA	LBV1-QA
Qwen3-4B-Thinking-2507	60.7	39.4	64.9	37.5	49.2	64.2
+ (GRPO + standard KL)	61.6	41.7	64.8	49.3	53.8	64.3
+ (GRPO without KL)	62.3	45.2	66.4	65.7	61.9	65.6
+ (GRPO-ZCPO)	63.8	48.2	68.2	66.0	69.6	66.4
+ (GRPO-ZCPO, $B_{q}=1$ )	63.3	48.0	67.6	66.3	68.5	66.1

Appendix

Table 6: Long-context evaluation results and group-gating ablation for Qwen3-4B. All methods trained in this work use the same data subset.

B_{q}=1

denotes the ablation that removes gating for groups with identical rewards, with all other ZCPO settings unchanged.

Data
Hyperparameter	Value
Max prompt length	160K
Max response length	16K
Responses per prompt	16
Optimization
Learning rate	2e-6

Appendix

Table 7: Training hyperparameters for the long-context experiments.

Parameter	Setting
Model, Data, and Method
Starting model	Qwen3-4B-Thinking-2507
Training data	A fixed subset of 8,000 GoLongRL examples; the same subset for all methods
Training framework and entry point	verl; recipe.dapo.main_dapo
Trainer configuration	dapo_megatron_trainer
Update coefficient	GRPO uses group-relative advantages; ZCPO uses the joint coefficient $C_{i}$ defined in the main text

Appendix

Table 8: Additional configuration for the Qwen3-4B long-context experiments. KL and group-gating settings that differ across methods are listed separately.

Model/method	AIME24 avg@32 (%)
GLM-4-9B-0414 + DAPO without KL	$24.5$
GLM-4-9B-0414 + ZCPO	$\mathbf{36.2}$

Appendix

Table 9: Cross-model math results on GLM-4-9B-0414 (AIME24 avg@32, %).

Explore similar work

Aug 31, 2026cs.LG

Group Adaptive Clipping Policy Optimization

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.

Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan +1

University of Toronto · Amazon

Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards Trust Region Policy Optimization Policy Learning

Ask PDF

Oct 6, 2026cs.LG

On KL-Regularized Policy Optimization

Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-

K

and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

Yifan Zhang

Princeton University

RL for LLM Agents KL-Regularized RL Off-Policy RL Asynchronous RL Critic-Free RL

Ask PDF

Feb 5, 2026cs.LG

Constrained Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) remains the dominant critic-free approach for fine-tuning LLMs and VLMs, but its compatibility with constrained policy optimization (e.g. for safety-critical domains) has not been carefully examined. In this work, we introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization. We show that the standard practice of scalarizing rewards before normalization introduces a critical Lagrangian-specific failure mode: GRPO's within-group normalization makes constrained optimization highly sensitive to how multi-component learning signals are aggregated. We show that scalarizing rewards before normalization introduces shared-denominator coupling, so that changing one multiplier alters not only the emphasis on its corresponding constraint, but also the relative weighting of the reward and other constraints. We address this with a simple but crucial modification: scalarizing standardized advantages rather than rewards. This yields a better-conditioned update by addressing the coupling induced by reward scalarization, resulting in better-behaved multiplier dynamics and more stable constraint enforcement in practice. Empirically, across a controlled gridworld, a real-world autonomous driving benchmark, and a mathematical reasoning task, Constrained GRPO consistently achieves better adherence to specified constraints while maintaining or improving task performance.

Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3

Mila – Quebec AI Institute Université de Montréal

Group Relative Policy Optimization Constrained RL Critic-Free RL Safe RL RL Fine-Tuning Policy Learning

Ask PDF