Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Siddarth Venkatraman; Matthieu Dinot; Laurence Aitchison

cs.LGAug 17, 2026

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Authors: Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

Organizations: Mistral AI · Mila – Quebec AI Institute · Université de Montréal

Abstract

Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.

Figures & tables

Figure 1: Privileged value functions. Let

h_{t}

be the policy state at time

t

, here represented as a Sudoku question and an intermediate chain of thought. To improve value estimation, a Privileged Value Function ( Pvf ) additionally uses context clues unavailable to the policy, such as a known reference solution

z_{1}

or leave-one-out group trajectories with rewards

z_{2}

. In this illustrative figure the value function conditions on both, producing the baseline

V_{t}=V_{\phi}(h_{t},z_{1},z_{2})

and unbiased policy advantage

A_{t}=R-V_{t}

Figure 2: Tether : adaptively combining group and value baselines. Our other contribution is a baseline that combines group Leave-One-Out mean and token level values through simple linear combination using an adaptive mixture ratio. The coefficient

\rho

is fit to minimize Monte Carlo return prediction error.

\rho=0

recovers the group baseline,

\rho=1

recovers the value function baseline, and intermediate values smoothly interpolate between the two endpoints.

Figure 3: RL with privileged value functions. Seed-averaged training reward curves (EMA smoothed); shaded regions denote one standard deviation across seeds. 2 seeds for RG runs and 3 seeds for CodeIO and Sudoku. We compare the group-mean ( Mean ), ordinary value function ( VF ), and privileged value function ( Pvf ) baselines.

K

is the rollout group size. The no-group

K=1

RG setting skips mean baseline. In RG and Sudoku, the privileged critic receives the ground-truth answer. In CodeIO, it receives the other

K-1=3

leave-one-out responses from the group and their rewards. Pvf is the best performing method in all settings. Figure 7 summarizes end-of-training rewards.

Figure 5: RL with Tether baseline. Seed-averaged training reward curves (EMA smoothed); shaded regions denote one standard deviation across seeds. Reasoning Gym uses two seeds, while CodeIO, Sudoku, and MiniF2F use three. We compare the group-mean ( Mean ), ordinary value-function ( VF ), and adaptive Tether baselines.

K

denotes the rollout group size. Tether improves over VF in all four settings. Tether outperforms Mean in RG and MiniF2F, matches it in CodeIO, and shrinks the gap in Sudoku. Figure 7 summarizes end-of-training rewards.

Figure 7: Aggregated final results. Bars show the mean raw reward over the final 50 policy steps, first averaged within each seed and then across seeds for every run; error bars denote one standard deviation across the per-seed means. This summary helps directly compare training reward differences between methods. Left: Pvf experiments. Right: Tether experiments.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Asynchronous value function evaluation and training. The value evaluator is the serving copy used in the inference and policy-training pipeline: it predicts the expected return from every partial response and these predictions are used to compute policy advantages. Evaluated trajectories are then added to the FIFO replay buffer. Independently, the value trainer samples replay batches and performs optimizer updates; the evaluator uses the latest published weights.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 410 papers

Off-Policy Value-Based Reinforcement Learning for Large Language Models
Mar 24, 2026Peng-Yuan Wang, Ziniu Li, Tian Xu +8Reinforcement Learning RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
Value-Gradient Hypothesis of RL for LLMs
May 20, 2026Arip Asadulaev, Daniil Ognev, Karim Salta +1RL for Language Models Policy Gradient
MBZUAI · Independent
LLM Value Functions for Credit Assignment in Agentic RL
Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM Agents Agentic RL
Apple

Mar 24, 2026J/K move · Enter open · S save

Off-Policy Value-Based Reinforcement Learning for Large Language Models

Peng-Yuan Wang, Ziniu Li, Tian Xu, Bohan Yang+7

National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data+1

Reinforcement Learning RL for Language Model Reasoning Off-Policy RL RL Fine-Tuning LLM Reasoning

Open paper AskPDF arXiv

Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.

Abstract

RL for Language Models

Explore similar work

Off-Policy Value-Based Reinforcement Learning for Large Language Models

Value-Gradient Hypothesis of RL for LLMs

LLM Value Functions for Credit Assignment in Agentic RL