Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Haoqun Cao; Yurun Yuan; Tengyang Xie

cs.LGOct 19, 2026

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Authors: Haoqun Cao, Yurun Yuan, Tengyang Xie

Organizations: University of Wisconsin–Madison

Abstract

Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift. We show that for a general class of off-policy policy gradient updates, consistency, in the sense that the optimal policy remains a stationary point, requires one of two structures at the optimal policy: trajectory-level importance weighting, which suffers from the well-known curse of horizon, or KL regularization with a baseline equal to the optimal soft value, instantiated by the grouped mean. Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage. This also gives a theoretical account of a variant of the empirically successful GRPO objective. We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage. We instantiate this via the asymmetric REINFORCE objective, and connect it to the game-theoretic offline RL framework, showing a regret bound with single-policy coverage. Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.

Figures & tables

Figure 1: Roadmap of the paper: the off-policy policy-gradient update template, the two consistent routes it admits, the chain of equivalent objectives and games along the KL route, and the two finite-sample guarantees.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Defined in
Rewards and returns
$r^{}_{h}$ , $R^{}(\tau)$	true per-step reward and trajectory reward	Sec. 2.1
$r_{h}$ , $R$ , $\mathcal{R}$	additive reward model and reward class	Sec. 5
$R^{*}_{\pi,\beta}(\tau)$	KL-regularized return $R^{*}(\tau)-\beta\sum_{h}\log\frac{\pi_{h}(a_{h}\|s_{h})}{\pi_{\textnormal{ref},h}(a_{h}\|s_{h})}$	Sec. 2.1
$R_{\pi}$ , $\mathcal{R}_{\Pi}$	policy-induced reward $\beta\log\frac{\mathbb{P}^{\pi}(\tau\|s_{1})}{\mathbb{P}^{\pi_{\textnormal{ref}}}(\tau\|s_{1})}$ and $\{R_{\pi}:\pi\in\Pi\}$	App. F.2
Value functions and induced policies

Appendix

Table 1: Summary of notation.

Abbreviation	Meaning	Reference
IS	importance sampling	Sec. 3
PG	policy gradient	Sec. 2.2
GRPG	group-relative policy gradient; the KL-route update studied here, a variant of the GRPO update	Sec. 4, Shao et al. [2024]
VM	conditional variance minimization; the objective whose gradient field is the GRPG update	Sec. 4.1, Tang et al. [2025]
TBRM	trajectory Bellman residual minimization; the value-based objective whose minimizers induce the same policies as VM	Sec. 4.1, Yuan et al. [2025]
AsymRE	asymmetric REINFORCE; GRPG with a constant shift in the grouped-mean baseline	Sec. 5, Arnal et al. [2025]

Appendix

Table 2: Abbreviations for the objectives and algorithms discussed in the paper.

Method	$\delta$	$\delta/\delta_{\mathrm{thm}}$	Regret $\downarrow$	$U(\hat{\pi})$
GRPG	$0$	$0$	$1366$ [1180, 1523]	$853$
AsymRE	$0.0039$	$0.1$	$2.69$ [2.19, 3.12]	$3.33$
AsymRE	$0.0193$	$0.5$	$0.314$ [0.216, 0.385]	$0.576$
AsymRE (Thm. 5)	$0.0387$	$1$	$\mathbf{0.239}$ [0.178, 0.276]	$0.279$
AsymRE	$0.0773$	$2$	$0.414$ [0.335, 0.483]	$0.138$
AsymRE	$0.309$	$8$	$4.06$ [3.26, 4.77]	$0.032$

Appendix

Table 3: Baseline-shift ablation (

\epsilon=0.03

\beta=0.4

n=128

; 50 datasets), for which

\delta_{\mathrm{thm}}=0.0387

. Regret and

U

are in units of

10^{-3}

$\epsilon$	$C_{\mathrm{all}}$	$C_{\pi^{*}}$	GRPG regret $\downarrow$	AsymRE regret $\downarrow$
$0.01$	$10^{12}$	$19.8$	$1329$ [1170, 1395]	$\mathbf{59.1}$ [51.6, 65.6]
$0.1$	$10^{6}$	$0.14$	$905$ [777, 1138]	$\mathbf{2.84}$ [1.44, 4.56]
$0.2$	$1.6\times 10^{4}$	$2.38$	$253$ [119, 343]	$\mathbf{0.890}$ [0.657, 1.27]
$0.3$	$1.4\times 10^{3}$	$7.58$	$12.2$ [0.00, 35.7]	$\mathbf{5.06}$ [3.70, 6.42]

Appendix

Table 4: All-policy coverage ablation (

\beta=0.8

n=128

; 30 datasets). Regret is in units of

10^{-3}

$\beta$	$\pi^{*}(\bar{a})$	$C_{\pi^{*}}$	GRPG regret $\downarrow$	AsymRE regret $\downarrow$
$0.2$	$4.5\times 10^{-5}$	$0.21$	$886$ [676, 991]	$\mathbf{0.278}$ [0.205, 0.358]
$0.4$	$6.7\times 10^{-3}$	$0.023$	$1538$ [1329, 1628]	$\mathbf{0.240}$ [0.217, 0.278]
$0.8$	$0.076$	$19.8$	$1321$ [1199, 1397]	$\mathbf{58.4}$ [51.6, 64.5]
$1.6$	$0.22$	$3.6\times 10^{4}$	$782$ [714, 843]	$769$ [693, 831]

Appendix

Table 5: KL-parameter ablation (

\epsilon=0.01

, so

C_{\mathrm{all}}=10^{12}

;

n=128

; 20 datasets).

\pi^{*}(\bar{a})

is the probability that

\pi^{*}

takes the non-preferred action. Regret is in units of

10^{-3}

$n$	$\delta_{\mathrm{thm}}$	GRPG regret $\downarrow$	AsymRE regret $\downarrow$
$10^{2}$	$4.4\times 10^{-2}$	$1532$ [1470, 2778]	$\mathbf{3.27}$ [3.24, 7.91]
$10^{4}$	$4.4\times 10^{-3}$	$1497$ [1427, 1561]	$\mathbf{0.262}$ [0.233, 0.269]
$10^{6}$	$4.4\times 10^{-4}$	$1534$ [1533, 1542]	$\mathbf{0.186}$ [0.180, 0.197]

Appendix

Table 6: Large-

n

regime (

\epsilon=0.001

\beta=0.4

; 5 datasets per

n

). Regret is in units of

10^{-3}

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 158 papers

Oct 6, 2026

On KL-Regularized Policy Optimization

Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-

K

and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

Yifan Zhang

Princeton University

RL for LLM Agents KL-Regularized RL Off-Policy RL Asynchronous RL Critic-Free RL

Ask PDF

Jul 6, 2026

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.

Yu Li, Xiuyu Li, Mingyang Yi +4

Renmin University of China

LLM Alignment Importance Sampling RL Post-Training Rejection Sampling Off-Policy RL Importance Sampling for RL

Ask PDF

Jun 15, 2026

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

Policy gradient algorithms for language models optimize the same objective

J(θ) = \mathbb{E}*{τ\sim p*θ(τ)}[R(τ)]

, which has exactly two factors: the trajectory probability

p_θ(τ)

and the reward

R(τ)

. Every method from REINFORCE to PPO to GRPO and their descendants modifies one or both factors to address a specific failure in the preceding formulation. Existing surveys organize these methods by domain or chronology, which obscures the rationale behind each design choice and the precise location of its intervention within the gradient estimator. This survey revisits the landscape of LLM policy optimization from

J(θ)

on first principles and uses the trajectory side, induced by

p_θ(τ)

, and the reward side, induced by

R(τ)

, as the two axes along which methods are located. It covers the path from REINFORCE and PPO to GRPO, as well as post-GRPO variants, Agentic RL, and GRPO-OPD. The resulting framework is unified, diagnostic, and extensible: it analyzes methods from a shared objective, identifies which side each method modifies and why, and applies the same trajectory and reward axes across these settings. Across these settings, the framework also exposes compound failures that no single-side fix resolves and that therefore require joint design of the trajectory side and the reward side. The boundary cases and coupled failures identified by this map mark where existing solutions run out and provide a principled starting point for designing the next generation of LLM policy optimization algorithms.

Jianghan Shen, Siqi Luo, Yue Li +9

1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4

RL for Language Models Group Relative Policy Optimization Policy Gradient Methods Proximal Policy Optimization Policy Learning

Ask PDF

Abstract

Policy Gradient Methods

Explore similar work

On KL-Regularized Policy Optimization

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions