Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT

Joery Ariën de Vries; Neil David Lawrence; Zhenwen Dai

cs.LGOct 2, 2026

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT

Authors: Joery Ariën de Vries, Neil David Lawrence, Zhenwen Dai

Organizations: Trent AI Limited, London, United Kingdom · Computer Laboratory, University of Cambridge, Cambridge, United Kingdom

Abstract

Critic-free reinforcement fine-tuning (RFT) for agentic large language models is often done through GRPO-style methods, which compute a group baseline over repeated rollouts to reduce target variance. However, this setup is ill-suited to agents acting in stateful environments such as live services or security sandboxes, where repeated rollouts are impractical to obtain and aggressive updates entrench the noise of long, sparsely verified trajectories. We propose Follow the Winners (FTW), a critic-free policy-learning algorithm that adapts the cross-entropy method to RFT, replacing group rollouts with an ordinal filter on replay-buffer samples that yields polynomial concentration in the order statistic of returns. We derive FTW through a control-as-inference lens, which also recovers GRPO and DPO as specific modelling choices, identifying GRPO as risk-neutral while DPO and FTW share a bounded risk-seeking offset that FTW controls. We identify this offset as an inherent trade-off of variance reduction through ordinal filters on samples, whereas a critic model induces a different trade-off between bias and variance. Scaled to agentic LLM post-training, FTW matches GRPO and PPO on Sokoban and Search-R1 baselines, showing a viable trade-off from a value model or group rollouts to CPU memory.

Figures & tables

Figure 2: Diagnostic bandits isolating algorithm properties (Sec. 4 ). Left (

\sigma

-trap): asymptotic

\pi(a^{*})

vs. distractor variance. GRPO is only stable for large

\sigma

when proportionally decreasing the learning rate. Center (Pareto-

\alpha

): under heavy tail returns, GRPO learning destabilizes whereas rank-based FTW/DPO remain stable. Right (state-aliasing): FTW achieves lower L1 distance

|\pi(a)-\pi^{*}(a)|

between the learned and optimal action distributions in the POMDP. Left all lines show mean and 95% bootstrapped CIs over replay buffer sizes

N\in\{16,64,256,1024,4096\}

and 10 seeds. Center and right: bold lines are averages over sampled learning-rates, translucent lines show sample trajectories.

Figure 3: Evaluation learning curves on HalfCheetah after 1B steps of training. Shaded regions are 95% Gaussian CIs over 10 seeds.

Figure 4: Sokoban test success rate curves (left, 128 puzzles, mean

\pm

one standard deviation over 5 seeds) and final exact-match accuracy

\pm

one standard deviation on the Search-R1 test set over 3 seeds (right, 50k+ examples). PPO and GRPO results are from Jin et al. (2025) .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Gaussian $\mathcal{N}(\mu,\sigma^{2})$	Sparse $m\,\mathrm{Ber}(p)$	Counts $m\,\mathrm{Poi}(\lambda)$	Exponential( $\theta$ )
GRPO	$0$	$0$	$0$	$0$
DPO	$(\mu-\mu_{0})\Big(\tfrac{\sigma_{0}}{\sqrt{\sigma_{0}^{2}+\sigma^{2}}}-1\Big)$	$0^{\dagger}$	$\approx$ Gaussian cell, $\sigma^{2}=m^{2}\lambda$	$\tfrac{1}{\theta_{0}}\ln\!\big(1+\tfrac{\theta_{0}}{\theta}\big)-\tfrac{1}{\theta}$
FTW top- $K$ / $C$	$\approx(\mu-t)\Big(\tfrac{\sigma_{0}s_{C,K}}{\sqrt{\sigma^{2}+\sigma_{0}^{2}s_{C,K}^{2}}}-1\Big)$	$0^{\dagger}$	$\approx$ Gaussian cell, $\sigma=m\sqrt{\lambda}$	$\approx$ Gaussian cell, $(a_{C,K},s_{C,K})$ via $H,H^{(2)}$
FTW smx-ranks	$\dfrac{\sigma^{2}}{2}\Big(\dfrac{f_{0}(\mu)}{\tau}-\dfrac{\mu-\mu_{0}}{\sigma_{0}^{2}}\Big)+o(\sigma^{2})$	$0^{\dagger}$	$\approx$ Gaussian cell	$\dfrac{\sigma^{2}\theta_{0}}{2}\Big(\dfrac{e^{-\theta_{0}\mu}}{\tau}-1\Big)+o(\sigma^{2})$
MNL-exp	$\dfrac{\sigma^{2}}{2\tau}$	$\tau\ln(1-p+pe^{m/\tau})-mp$	$\tau\lambda(e^{m/\tau}-1)-m\lambda$	$-\tau\ln\!\big(1-\tfrac{1}{\theta\tau}\big)-\tfrac{1}{\theta},\theta\tau>1$

Appendix

Table 1: Risk offsets

\mathcal{R}=J-\mu

per operator (rows) and return family (columns) ordered from least aggressive (top) to most aggressive (bottom). Sparse rewards use scaled returns

m

and success rate

p

, such that

\mu=mp

allowing the mean to be invariant to larger or smaller

p

for properly chosen

m

. Similarly, counts use

G=m\cdot\mathrm{Poi}(\lambda)

, so

\mu=m\lambda

and

\sigma^{2}=m^{2}\lambda

. The exponential family has rate

\theta

and mean

1/\theta

and requires the MNL-exp cell to satisfy

\theta\tau>1

to be defined. GRPO’s zeros are exact, while

0^{\dagger}

marks cells whose ranking is mean-consistent but whose offset is not expressible in return units (the certainty equivalent is interval-valued). The constants

a_{C,K}

and

s_{C,K}^{2}

are the mean and variance of the

K

-th largest value among

C-1

buffer returns standardized to mean

0

and variance

1

t=\mu_{0}+\sigma_{0}a_{C,K}

denotes the elite threshold,

H_{n}

and

H^{(2)}_{n}

are the harmonic numbers of first and second order, and

f_{0}

is the density of the population marginal

F_{0}

Bandit & Brax	SearchR1	Sokoban
Logit-Table / MLP	Qwen2.5-3B-Instruct	Qwen2.5-VL-3B-Instruct
JAX 0.9.2	verl-agent 0.3.1.dev	verl-agent 0.3.1.dev
brax 0.14.2	torch 2.8.0+cu128	torch 2.8.0+cu128
mujoco-mjx 3.6.0	vLLM 0.11.0	vLLM 0.11.0
flax 0.12.6	flash-attn 2.7.4.post1 †	gym 0.26.2
optax 0.2.8	faiss-gpu-cu12	gym-sokoban 0.0.6

Appendix

Table 2: Key software versions per experimental environment. Top section: policy model. Bottom section: core framework (first row) and other critical pinned packages. All LLM-agent runs additionally use transformers 4.57.6 and ray 2.54.1 . † flash-attn built from source to resolve ABI mismatch with torch 2.8.0+cu128 . SearchR1 requires a separate server for the wiki retrieval, making it the most involved environment to set up.

Parameter	FTW	SAC
Actor objective	BC on top- $K$ winners	$\alpha\log\pi-Q$
Elite filter $C$ / $J$ / $K$	$64$ / $256$ / $\{1,4\}$	n/a
Updates per env step	$8$ (BC)	$1$
Entropy	$0.05$ (fixed)	auto, target $\bar{H}{=}{-}\tfrac{1}{2}\|\mathcal{A}\|$
Critic	n/a	clipped double- $Q$ , Polyak $\tau$

Appendix

Table 3: Brax HalfCheetah hyperparameters. All methods share

\mathrm{lr}{=}3{\times}10^{-4}

(Adam),

\gamma{=}0.99

, replay size

2{\times}10^{6}

64

parallel envs, batch size

256

, episode length

1000

10^{9}

total env steps, warmup

8192

transitions, and

10

seeds. Any parameters not mentioned are all kept at their default values from the Brax 0.14.2 ( Freeman et al., 2021 ) implementation of SAC ( Haarnoja et al., 2018 ) .

Parameter	SearchR1	Sokoban
Learning rate	$1{\times}10^{-6}$	$1{\times}10^{-6}$
Weight decay	$0.01$	$0.01$
Gradient clip	$1.0$	$1.0$
Rollout batch (prompts)	$256$	$32$
Rollouts per prompt $n$ (GRPO only)	$5$	$8$
Mini-batch size	$512$	$64$

Appendix

Table 4: LLM-agent hyperparameters. All runs use bfloat16 rollouts and optimize with AdamW ( Loshchilov and Hutter, 2019 ) . Unless otherwise stated, we use the exact hyperparameter defaults as described in the respective baselines as implemented by verl-agent 0.3.1.dev ( Feng et al., 2025 ) . For FTW, the number of minibatches per update is

J=128/K

, so every update uses

J\cdot K=128

training targets regardless of

K

Figure 5: Online episode success rate on Sokoban (left) and Search-R1 (right) with EMA-smoothed means and

\pm

1 standard deviation over seeds, individual seeds are drawn faintly. Note that, the Search-R1 training split covers only NQ and HotpotQA, so the right panel is not on the same question distribution as the main-text reported test accuracy.

Figure 6:

\sigma

-trap solutions for FTW selection variants (10 seeds; shaded: bootstrap 95% CIs).

Figure 7: Bandit sweep across five return distributions (Beta, Binomial, scaled Binomial, Pareto, Student-

t

). Left column shows the scale-regret

\max_{k}\mu_{k}-\mathbb{E}_{\pi}[\mu_{k}]

, right column shows the probability of selecting the optimal arm. Color encodes the distribution-shape parameter, and we use the same legend as our Pareto-

\alpha

result in the main text.

Figure 8: State-aliasing (corridor) task with a learning-rate sweep at horizon $H=16$ . We plot average L1 distance $|\pi(a)-\pi^{*}(a)|$ between the learned and optimal action distributions, averaged across learning rates within each method. Lower is better.

Figure 9: Population solutions after

T=8\times 10^{5}

steps of training vs. maximum sample age

h

(100 seeds). Dashed line: on-policy GRPO mean and 95% CI. The top row shows results under Bernoulli returns and the bottom row under homoscedatic Gaussian noise

\mathcal{N}(\mu_{a},0.01)

returns.

Figure 10: Mean learning curves for Figure 9 (shaded: 95% CI). Evidently, convergence takes proportionally longer as

\delta

decreases.

Figure 11: Learning curves at

h=128

(30 seeds, mean and Gaussian

95\%

bands). Left to right: offline simple regret of the posterior predictive, cumulative regret divided by

t

, posterior width, and predictive entropy. Colors denote the method family and line styles its variants. Thompson sampling is an oracle handed the true linear-Gaussian structure whereas the policy methods must learn it through the variational learning rule.

Figure 12: Offline simple regret (30 seeds, Gaussian

95\%

bands). Top row: each method’s concentration knob at

h=128

. Bottom row: the buffer-depth ladder at each method’s best knob setting.

Critic-Free RL

Critic-Free Reinforcement Learning optimizes policies without a separately learned value critic, using rollout comparisons, return statistics, or derived values.

See all 23 papers

Sep 29, 2026

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that a pretrained critic's ability to predict future outcomes can make it a valuable asset for efficient long-horizon reasoning. First, we find that instability in critic-based RL for long chain-of-thought reasoning is largely an optimization artifact: keeping policy updates small and low in variance restores stable convergence. Second, a well-pretrained critic estimates the posterior probability of eventual success from later trajectory states and unfinished prefixes. Its predictions provide outcome-derived, dense, per-prefix learning signals that, during policy optimization, require neither completed rollouts, step-level annotations, nor external reward labels. Building on this insight, we introduce Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes. We further show that binarizing the debiased score stops the policy from exploiting the critic's length bias. Binarized, RFPO matches supervised PPO without a single label in the training loop, while cutting compute and memory overhead. This makes RFPO well suited to long-horizon reasoning tasks, where outcomes arrive late and generation dominates cost: because rollouts can be rewarded before they finish, training no longer has to pay for waiting on every trajectory to complete. Our findings challenge the prevailing critic-free paradigm and establish critic-based, reward-free optimization as a scalable and computationally efficient path for LLM post-training.

Hongyang Li, Xiao Li, Caesar Wu +3

University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Reinforcement Learning Value Function Estimation RL for Language Model Reasoning RL Post-Training Policy Learning

Ask PDF

Jan 26, 2026

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL.

Yibo Li, Zijie Lin, Ailin Deng +5

National University of Singapore, Singapore.

Continual Learning for LLM Agents Reinforcement Learning Test-Time RL Policy Learning RL for LLM Agents Continual RL+1

Ask PDF

Sep 29, 2026

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version of REINFORCE -- referred to as RE(S) -- that updates the rollout distribution once every

S \ge 1

gradient steps. Prior work in bandits and reinforcement learning has developed rich theory for policy gradient methods, and on-policy sampling (i.e., a small

S

, ideally

1

) is often viewed as crucial to their success; yet in prominent application like post-training large language models, reward-guided self-training has proved to be effective even when the rollout distribution is updated infrequently, but theoretical understanding remains limited for the convergence properties of these off-policy methods. To bridge these gaps, we develop a unified theory for RE(S) that covers the full spectrum of

S \ge 1

: it can be interpreted as a stage-wise optimization process, where each stage takes

S

gradient steps for minimizing the Kullback-Leibler distance to a fixed reward-weighted rollout distribution. For multi-arm bandits with softmax policies, our in-depth analysis and numerical experiments reveal three key findings: (1) for any fixed

S

, RE(S) enjoys global convergence to the optimal policy as the number of rollout distribution updates

B = \lfloor T / S \rfloor \rightarrow \infty

, where

T

denotes the number of gradient steps; (2) we prove tight two-sided bounds showing that the suboptimality gap of RE(S) achieves an asymptotic

Θ(1 / T)

convergence rate, while

S

only affects the length of a burn-in phase; (3) when initialized at a weak policy with a small optimal-action probability, RE(1) gets trapped around suboptimal policies for a long period, whereas RE(S) with a suitable

S

avoids the detour and achieves significantly faster convergence to the global optimum, highlighting the benefits of off-policyness in this case.

Zhiwei Wang, Yanxi Chen, Yaliang Li +1

Department of Mathematical Sciences Tsinghua University · Alibaba Group

Policy Gradient Methods Off-Policy RL Multi-Armed Bandits

Ask PDF

Abstract

Critic-Free RL

Explore similar work

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness