Target Policy Optimization

Jean Kaddour

cs.LGApr 7, 2026

Target Policy Optimization

Authors: Jean Kaddour

Abstract

In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce Target Policy Optimization (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution $q_i \propto p_i^{\,\mathrm{old}} \exp(u_i)$ and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is $p^θ- q$ , which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.

Figures & tables

Figure 1: TPO matches baselines on easy tasks and outperforms them under sparse reward. (a) On an MNIST contextual bandit with dense reward, TPO converges slightly faster than GRPO and DG. (b) On a sparse-reward token-reversal task (reward only at end of sequence), GRPO and DG stall near random while TPO solves the task. Both panels show mean

\pm

s.e. over 20 seeds.

q_{i}=\frac{p_{i}^{\text{old}}\exp(u_{i}/\eta)}{\sum_{j=1}^{K}p_{j}^{\text{old}}\exp(u_{j}/\eta)}.

Algorithm 1 Target Policy Optimization (TPO)

Figure 3: Single-context symmetric bandit (

K{=}100

B{=}100

, normalized steps). (a) TPO and DG converge fastest; GRPO and PG plateau at higher error. (b) TPO maintains the lowest misalignment to the oracle gradient throughout training.

Figure 4: Multi-context bandit (

N{=}100

K{=}10

, exact gradients). (a) All methods converge; the CE oracle is fastest. (b) TPO achieves near-zero misalignment to the CE oracle direction, confirming its update direction targets the optimal allocation.

Figure 5: MNIST contextual bandit: TPO converges fastest and reaches the lowest error. (a) Learning curves for all single-sample bandit updates, including the same-signal ablation Group PG. (b) At step 2,000, for each misclassified example we measure how much more each method increases the true-class probability

p_{y}

compared to a generic one-vs-rest baseline (Appendix C ), binned by wrong-class concentration

c=\max_{j\neq y}\pi_{j}/(1-\pi_{y})

. TPO’s extra gain grows with concentration; DG’s does not.

Figure 6: Token Reversal (bag-of-tokens reward,

K{=}8

token candidates). All methods use

B{=}100

prompts and follow one behavior trajectory each; TPO token and GRPO token additionally sample

K

next-token candidates at each prefix state. Columns vary vocabulary size

V\in\{2,4,8,16\}

	$V=2$	$V=4$	$V=8$	$V=16$
TPO token	58	74	103	102
GRPO token	904	141	124	148
DG	199	273	314	393
PPO	872	181	191	259

Table 1: Steps to 1% error. Token Reversal (bag-of-tokens reward,

K{=}8

token candidates). Bold : fastest method at each

V

. All methods use the same environment interactions per step.

Figure 7: Task variations, prompt- and interaction-matched. Top two rows : prompt-matched. Bottom two rows : interaction-matched. Within each pair, the first row is bag-of-tokens reward and the second is sequential reward. Columns vary target logic.

Reward	Target	TPO token	GRPO token	DG	PPO
Bag of tokens	Copy	81	338	219	170
	Flip	56	104	201	146
	Rev. copy	55	352	202	$-$
	Rev. flip	59	209	200	143
Sequential	Copy	295	$-$	439	$-$
	Flip	321	$-$	349	$-$

Table 2: Steps to 1% error, task variations (

K{=}8

token candidates). Bold : fastest per row. “

-

”: never reached within budget.

Figure 8: Terminal reward, prompt- and interaction-matched. Top row : prompt-matched ( $B{=}100$ for all methods). Bottom row : interaction-matched ( $B{\cdot}K{=}800$ rollouts per step, with single-sample batch size and learning rate scaled by $K$ and $\sqrt{K}$ respectively). Here grouped methods use $K{=}8$ candidates per prompt. Y-axis: exact-match error. TPO has the lowest error at each $H$ under both matching conditions.

	Prompt-matched				Interaction-matched
	$H{=}7$	$H{=}8$	$H{=}9$	$H{=}10$	$H{=}7$	$H{=}8$	$H{=}9$	$H{=}10$
TPO	6.9	8.6	6.1	7.4	1.8	2.8	5.3	19.0
GRPO	14.5	27.6	30.0	50.4	9.6	23.2	36.2	48.7
GRPO (no KL)	66.6	92.5	$-$	$-$	78.1	83.8	$-$	$-$
PPO	12.0	26.3	90.6	$-$	38.6	62.1	66.2	$-$
DG	33.8	58.8	$-$	$-$	47.7	69.4	$-$	$-$

Table 3: Exact-match error (%), terminal reward. Bold : best method. “

-

”:

>

95% (no meaningful learning). Left: prompt-matched. Right: interaction-matched. TPO attains the lowest error at each tested

H

Figure 9: Removing the anchor, KL penalty, or target matching each degrades learning. Terminal reward, reverse-copy targets, $V{=}2$ , $K{=}8$ , $B{=}100$ , 20 seeds. Shading shows $\pm 1$ s.e.

Figure 10: LLM RLVR. Top row: Qwen3-1.7B. Bottom row: DeepSeek-R1-Distill-Qwen-1.5B. All runs use

K{=}16

rollouts per prompt. Columns: GSM8K (held-out test accuracy, evaluated every 5 steps), Reasoning Gym graph coloring (train mean score), Reasoning Gym Knights & Knaves (train mean score).

Figure 11: TPO’s gradient self-extinguishes; GRPO’s does not (

H{=}8

V{=}2

K{=}32

). (a) Gradient L2 norms over training. (b) Per-candidate weight proxy on successful (solid) vs. failed (dashed) candidates: mean target mass

q_{i}

for TPO, mean

|A_{i}|

for GRPO.

Figure 12: Most groups carry no signal early in training; TPO eliminates them fastest (

H{=}8

V{=}2

K{=}32

). (a) Fraction of groups where all

K

candidates fail. (b) Fraction of prompts with at least one successful candidate.

Figure 13: Group-size sensitivity sweep (

H{=}8

V{=}2

, epochs

{=}4

). (a) TPO learning curves: steady improvement as

K

grows, with the strongest performance at

K{=}64

. (b) GRPO learning curves: larger groups help, but performance remains less stable and less monotonic. (c) Final error vs.

K

: TPO improves from 8.9% at

K{=}4

to 0.36% at

K{=}64

; GRPO improves from 19.4% at

K{=}4

to 4.4% at

K{=}32

and then worsens slightly at

K{=}64

(5.6%). 30 seeds, shading/bars

\pm

1 s.e.

Figure 14: Zero-variance masking (

H{=}8

V{=}2

K{=}32

, epochs

{=}4

). (a) Learning curves: GRPO (zv-masked) is substantially worse than both GRPO and TPO. (b) Final error: masking increases GRPO from 6.3% to 29.7%, while TPO reaches 0.05% without any masking. 30 seeds, shading/bars

\pm

1 s.e.

Figure 15: Multi-epoch extraction (

H{=}8

V{=}2

K{=}32

). (a) Error curves: TPO with 4 gradient epochs reaches 0.2% error at episode 400 while TPO with 1 epoch is at 1.1%, roughly

5\times

faster. Both eventually converge to

<

0.1%. DG, limited to a single epoch, plateaus at 14%. (b) Gradient norms: TPO (4 ep) gradient decays fastest; TPO (1 ep) shows a delayed spike and slower decay; DG’s gradient stays low but persistent.

Figure 16: Epoch-count ablation (

H{=}8

V{=}2

K{=}32

). (a) TPO learning curves across epoch counts: all converge smoothly and remain low-error throughout. (b) GRPO learning curves: 2 epochs is the worst setting, while 8 and 16 epochs recover strongly. (c) Final error comparison: TPO stays below 2.3% everywhere; GRPO is strongly non-monotonic (37.6% at 2 epochs, 1.1% at 16 epochs). 30 seeds, shading

\pm

1 s.e.

Method	Update rule	Group	Critic	Fixed ref.
REINFORCE	PG + baseline	✗	✗	✗
REINFORCE++	PG + per-token KL baseline	✗	✗	✓
ReMax	PG + greedy baseline	✗	✗	✓
TRPO	PG + KL trust region	✗	✗	✗
PPO	Clipped PG surrogate	✗	✗	✗
DG	Sigmoid-gated PG	✗	✗	✗

Table 4: Comparison of policy optimization methods. Group: update structurally compares candidates within a context. Fixed ref.: requires a frozen reference model beyond

\pi_{\text{old}}

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: TPO temperature ablation. All values in

[0.25,2]

converge within 141 episodes; only

\eta{=}4

is meaningfully slower. Performance is robust across a 16

\times

range.

$\eta$	Final error (%)	Steps to 1%
0.25	1.0	72
0.50	0.0	67
1.00 (default)	0.7	96
2.00	1.0	141
4.00	0.8	260

Appendix

Table 5: TPO temperature ablation on token reversal (reverse copy,

V{=}2

). Performance is robust across a wide range; only

\eta{=}4

degrades substantially.

Figure 18: DG epoch sensitivity across sparse- and dense-reward transformer tasks. (a) Reverse-copy transformer RLVR with terminal reward, 20 seeds: reusing each rollout batch for 4 DG gradient epochs keeps the error high (48.3% final) while the standard 1-epoch DG update reaches 2.0%. (b,c) Final error on the eight prompt-matched token-reversal variants from Section 3.5 (

H{=}10

V{=}2

K{=}8

token candidates, 10 seeds), split by reward type. DG with 4 epochs is worse in 7 of 8 settings, with the largest regressions on the sequential tasks. Shading and error bars show

\pm 1

s.e.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 224 papers

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
May 7, 2026Yun Qu, Qi Wang, Yixiu Mao +11RL for Language Model Reasoning Group Relative Policy Optimization
Department of Automation, Tsinghua University · 2LLM Department, Tencent
A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
Jun 15, 2026Jianghan Shen, Siqi Luo, Yue Li +9RL for Language Models Group Relative Policy Optimization
1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4
P^2O: Joint Policy and Prompt Optimization
Mar 23, 2026Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2

May 7, 2026J/K move · Enter open · S save

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou+10

Department of Automation, Tsinghua University · 2LLM Department, Tencent

RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization Reinforcement Learning with Verifiable Rewards RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.

Abstract

Policy Optimization

Explore similar work

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

P^2O: Joint Policy and Prompt Optimization