Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Zhiwei Wang; Yanxi Chen; Yaliang Li; Bolin Ding

cs.LGSep 29, 2026

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Authors: Zhiwei Wang, Yanxi Chen, Yaliang Li, Bolin Ding

Organizations: Department of Mathematical Sciences Tsinghua University · Alibaba Group

Abstract

We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version of REINFORCE -- referred to as RE(S) -- that updates the rollout distribution once every $S \ge 1$ gradient steps. Prior work in bandits and reinforcement learning has developed rich theory for policy gradient methods, and on-policy sampling (i.e., a small $S$ , ideally $1$ ) is often viewed as crucial to their success; yet in prominent application like post-training large language models, reward-guided self-training has proved to be effective even when the rollout distribution is updated infrequently, but theoretical understanding remains limited for the convergence properties of these off-policy methods. To bridge these gaps, we develop a unified theory for RE(S) that covers the full spectrum of $S \ge 1$ : it can be interpreted as a stage-wise optimization process, where each stage takes $S$ gradient steps for minimizing the Kullback-Leibler distance to a fixed reward-weighted rollout distribution. For multi-arm bandits with softmax policies, our in-depth analysis and numerical experiments reveal three key findings: (1) for any fixed $S$ , RE(S) enjoys global convergence to the optimal policy as the number of rollout distribution updates $B = \lfloor T / S \rfloor \rightarrow \infty$ , where $T$ denotes the number of gradient steps; (2) we prove tight two-sided bounds showing that the suboptimality gap of RE(S) achieves an asymptotic $Θ(1 / T)$ convergence rate, while $S$ only affects the length of a burn-in phase; (3) when initialized at a weak policy with a small optimal-action probability, RE(1) gets trapped around suboptimal policies for a long period, whereas RE(S) with a suitable $S$ avoids the detour and achieves significantly faster convergence to the global optimum, highlighting the benefits of off-policyness in this case.

Figures & tables

Figure 1: Empirical validation of RE(

S

) convergence rates in two concrete settings, both with

K=100

. Left: the reward-mean vector is

\mu=[1,0.1,\dots,0.1]

, the initial policy

\pi_{0,0}

has probability

0.9

on the optimal action and equal probabilities on the remaining actions, and the learning rate

\eta=0.095

. Right:

\mu=[1,0.6,\dots,0.6]

\pi_{0,0}

has probability

0.7

on the optimal action and equal probabilities on the remaining actions, and

\eta=1

Figure 2: Empirical validation of the benefits of off-policyness. Left:

K\in\{10,100\}

, the reward-mean vector is

\mu=[1,0.7,0.3,0.3,\dots]

, the initial policy

\pi_{0,0}

satisfies

\pi_{0,0}(1)=0.1/K

and

\pi_{0,0}(a)\propto e^{-2\mu(a)}

for

2\leq a\leq K

. Right: a minimal setting for visualizing the optimization trajectories of RE(

S

), with

K=3

\mu=[1,0.9,0.89]

, and

\pi_{0,0}=[0.0005,0.1,0.8995]

. The learning rate

\eta=0.5

for both experiments.

Policy Gradient Methods

Policy gradient methods optimize stochastic policies by estimating how parameter changes affect expected reward.

See all 158 papers

Oct 19, 2026

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift. We show that for a general class of off-policy policy gradient updates, consistency, in the sense that the optimal policy remains a stationary point, requires one of two structures at the optimal policy: trajectory-level importance weighting, which suffers from the well-known curse of horizon, or KL regularization with a baseline equal to the optimal soft value, instantiated by the grouped mean. Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage. This also gives a theoretical account of a variant of the empirically successful GRPO objective. We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage. We instantiate this via the asymmetric REINFORCE objective, and connect it to the game-theoretic offline RL framework, showing a regret bound with single-policy coverage. Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.

Haoqun Cao, Yurun Yuan, Tengyang Xie

University of Wisconsin–Madison

Policy Gradient Methods Off-Policy RL RL Post-Training KL-Regularized RL Importance Sampling for RL Offline RL

Ask PDF

May 7, 2026

On Training in Imagination

State-of-the-art model-based reinforcement learning methods train policies on imagined rollouts. These rollouts are trajectories generated by a learned dynamics model and are scored by a learned reward model, but without querying the true environment during policy updates. We study this training paradigm by quantifying how errors in learned dynamics and reward models affect returns and policy optimization. First, we extend the analysis of Asadi et al. (2018) to MDPs with learned reward models, and derive the optimal sample allocation--the ratio of dynamics samples to reward samples that minimizes a bound on return error under power-law scaling assumptions. We identify lower Lipschitz constants of the learned dynamics, reward, and policy as a representation desideratum that tightens this bound, and we connect this perspective to the temporal-straightening objective of Wang et al. (2026). Second, we examine how policy optimization with REINFORCE tolerates noisy rewards, which are often cheaper to obtain. We show that zero-mean reward noise leaves the gradient estimator unbiased and adds at most a variance term that decreases with the number of rollouts. This introduces a practical tradeoff: given a fixed budget, should one buy more rollouts with cheaper but noisier rewards, or fewer rollouts with more expensive but less noisy rewards? We reduce this choice to a one-dimensional optimization problem and characterize the optimum.

Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum +2

Weizmann Institute of Science · New York University · Columbia University

Reward Modeling Reinforcement Learning Policy Gradient Methods Policy Learning Model-Based RL

Ask PDF

Sep 1, 2026

Online Self-Weighted Fine-Tuning

Standard supervised fine-tuning (SFT) assigns the same explicit loss weight to every expert demonstration, regardless of the model's changing competence over training queries. Reinforcement learning (RL) based methods adapt update strength using model-generated rollouts, but often require substantially more sampling and can be unstable on hard tasks. We propose Online Self-Weighted Fine-Tuning (OSW-FT), a simple method that augments SFT with online, trajectory-level weighting. For each query, OSW-FT estimates the model's current success rate using a small number of inference-only rollouts and rescales the standard SFT loss accordingly. The optimization direction remains anchored to the expert trajectory, while the update magnitude adapts online. For binary-verifiable reasoning, we connect this weighting to SFT and RL at the gradient level, inspired by variance-reduction principles. The resulting estimator is unbiased for the exact OSW-FT surrogate update for any finite rollout count, and we analyze convergence with respect to the corresponding surrogate objective. Evaluated across Qwen3 series ranging from 0.6B to 4B on multiple challenging benchmarks (e.g., AIME), OSW-FT consistently improves over SFT on small-to-medium scale models. OSW-FT offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only 2 online rollouts.

Haiquan Wen, Yiwei He, Bei Peng +1

University of Liverpool · University of Sheffield

Supervised Fine-Tuning LLM Fine-Tuning Adaptive Loss Weighting

Ask PDF

Abstract

Policy Gradient Methods

Explore similar work

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

On Training in Imagination

Online Self-Weighted Fine-Tuning