LLM Value Functions for Credit Assignment in Agentic RL

Kaijie Zhu; Weixiang Yan; Omar Attia; Abbas Kazerouni; Alexander Toshev; Andrew Szot

cs.LGOct 11, 2026

LLM Value Functions for Credit Assignment in Agentic RL

Authors: Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot

Organizations: Apple

Abstract

Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.

Figures & tables

Figure 1: Overview of the proposed framework. ( 1 ) Group rollout: We sample a group of trajectories from the current policy and use their sparse outcome rewards to compute the empirical group success rate

\mu_{G}

. ( 2 ) LLM value estimation: A frozen LLM critic scores all trajectory states, conditioned on group context. Estimates are anchored to empirical outcomes: the initial-state value to the group success rate and the terminal-state value to the actual reward. ( 3 ) Step advantages: We compute unnormalized step advantages as consecutive value differences (

\overline{A}_{t}=V_{\phi}(s_{t+1})-V_{\phi}(s_{t})

) and normalize them across the group, yielding a dense, per-action credit signal. ( 4 ) Policy optimization: These step-level advantages drive standard PPO updates in LVRL, or act as a gating mechanism in LVOPD to restrict teacher supervision to actions that decrease the estimated value.

Table 2

Split	Door distance	Layouts	Median optimal actions	Max optimal actions
Train	1	95	14	34
	2	95	24	48
	3	95	32	60
Evaluation ID	1	100	14	36
	2	100	23	46
	3	100	32	51

Table 5: BabyAI training and evaluation pools. Optimal actions include rotations, forward movements, door toggles, and the final pickup. The three splits use disjoint seed ranges and share no layouts.

Panel of Figure 3 — Figure 3: Representative BabyAI evaluation layouts at increasing door distance. The blue line is the shortest collision-free path from the agent to a cell adjacent to the target; the action count below each panel additionally includes turns, door toggles, and the final pickup. The path crosses one, three, and five doors respectively, which is what door distance measures. Every layout holds the same inventory of $18$ objects, so distractor identities carry no information and only the spatial arrangement and the choice of target vary. The agent never observes this map: it receives a textual egocentric description (Figure 4 ).

Environment / student	Queries	Input tokens	Output tokens	USD / query
AppWorld / Qwen3-4B	92,718	9.01k	0.66k	0.061
AppWorld / Qwen3-8B	99,968	9.43k	0.61k	0.062
BabyAI / Qwen3-8B	8,838	11.86k	1.90k	0.107
R2E-Gym / Qwen3-8B	8,030	20.35k	1.55k	0.140

Table 6: Estimated cost per logged value-estimation query. Token counts are means computed with cl100k_base , an uncalibrated proxy for Claude tokenization. Each query scores one target trajectory. Input counts include peer summaries, but the cost of generating those summaries is excluded. Dollar estimates use standard Claude Opus 4.6 rates.

Setting	AppWorld	BabyAI
Training updates	200	150
Rollout-step budget	50	128
Training nodes	1	2
Training world size	8	16
Minibatch size per rank	4	4
Gradient-accumulation steps	8	4

Table 7: Configured rollout and optimization settings for LVOPD. The effective optimization batch size is the product of the per-rank minibatch size, gradient-accumulation steps, and training world size.

Figure 9: RL training reward curves across three training environments. For AppWorld and BabyAI, curves show the mean over three seeds; shaded regions denote one standard deviation.

Figure 10: OPD training reward curves on AppWorld across model sizes. Curves show the mean over three seeds; shaded regions denote one standard deviation. Both gated variants consistently outperform the OPD baseline after roughly

100

training steps.

	BabyAI
Method	Qwen3-4B	Qwen3-8B
Base	$0.0{\scriptstyle\,\pm\,0.0}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$\mathbf{0.7}{\scriptstyle\,\pm\,0.0}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+OPD	$0.5{\scriptstyle\,\pm\,0.1}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.6{\scriptstyle\,\pm\,0.3}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+Trajectory-gated	$0.4{\scriptstyle\,\pm\,0.2}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.3{\scriptstyle\,\pm\,0.3}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+LVOPD (ours)	$\mathbf{0.7}{\scriptstyle\,\pm\,0.2}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.4{\scriptstyle\,\pm\,0.1}$ / $0.0{\scriptstyle\,\pm\,0.0}$

Table 8: Evaluation performance under on-policy distillation.

Threshold	Minimum difference	Worst-case tail probability	Expected spurious events
$1\sigma$	$4/32$	$0.3820$	$664$
$2\sigma$	$8/32$	$0.0599$	$72$
$3\sigma$	$12/32$	$0.0037$	$3.4$

Table 9: Sampling noise for differences between two independent Monte-Carlo estimates with

M=32

. The final column estimates the number of spurious events over the full evaluation set under the null.

Figure 11: Agreement between predicted value changes and Monte-Carlo events across normalized trajectory position

t/L

, where

L

is the number of actions. (a) Fraction of MC events in each position bin for which the predicted value changes in the same direction as the MC estimate: a decrease for pitfalls and an increase for recoveries. (b) Fraction of MC events for which the model predicts no value change (

\Delta_{t}^{\mathrm{pred}}=0

), despite an MC increase or decrease large enough to meet the event threshold (

|\Delta_{t}^{\mathrm{MC}}|\geq 3\sigma

Position	Same sign	No change	Opposite sign
First third ( $n=34$ )	$44\%$	$29\%$	$26\%$
Middle third ( $n=54$ )	$78\%$	$13\%$	$9\%$
Final third ( $n=26$ )	$88\%$	$4\%$	$8\%$

Table 10: Step-level agreement by normalized trajectory position. No change denotes

\Delta^{\mathrm{pred}}=0

. Percentages may not sum to

100\%

because of rounding.

	Task	Outcome	$\Delta^{\mathrm{MC}}$	$\Delta^{\mathrm{pred}}$	Detected
Pitfall, no execution error	3c13f5a_2	✗	$-0.938$	$-0.70$	✓
Recovery despite an error	34d9492_2	✓	$+0.438$	$+0.30$	✓
Missed pitfall	27e1026_1	✗	$-0.500$	$+0.05$	✗

Table 11: The three transitions examined in this section. The predicted value tracks the Monte-Carlo value on the first two transitions and moves in the wrong direction on the third.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 583 papers

Apr 12, 2026

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Zikang Shan, Han Zhong, Liwei Wang +1

Peking University · Microsoft Research Asia · Shanghai Jiao Tong University

RL for Language Models Reinforcement Learning Value Function Estimation Credit Assignment in RL Actor-Critic Methods

Ask PDF

Mar 7, 2026

Hindsight Credit Assignment for Long-Horizon LLM Agents

Large Language Model (LLM) agents often face significant credit assignment challenges in long-horizon, multi-step tasks due to sparse rewards. Existing value-free methods, such as Group Relative Policy Optimization (GRPO), encounter two fundamental bottlenecks: inaccurate step-level Q-value estimation and misaligned value baselines for intermediate states. To address these limitations, we introduce HCAPO, the first framework to integrate hindsight credit assignment into LLM agents. HCAPO leverages the LLM itself as a post-hoc critic to refine step-level Q-values through hindsight reasoning. Furthermore, HCAPO's multi-scale advantage mechanism effectively supplements the inaccurate value baselines at critical decision states. Evaluations across three challenging benchmarks, including WebShop and ALFWorld, demonstrate that HCAPO consistently outperforms state-of-the-art RL methods. Notably, HCAPO achieves a 7.7% improvement in success rate on WebShop and a 13.8% on ALFWorld over GRPO using the Qwen2.5-7B-Instruct model. These results indicate that HCAPO significantly enhances exploration efficiency, promotes concise decision-making, and ensures scalability in complex, long-horizon tasks.

Hui-Ze Tan, Xiao-Wen Yang, Hao Chen +7

State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · FiT, Tencent, Shenzhen, China +2

Credit Assignment in RL Long-Horizon LLM Agents RL for LLM Agents Step-Level Credit Assignment in RL

Ask PDF

Aug 17, 2026

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.

Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

Mistral AI · Mila – Quebec AI Institute · Université de Montréal

RL for Language Models RL for Language Model Reasoning Token-Level Credit Assignment

Ask PDF

Abstract

RL for LLM Agents

Explore similar work

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Hindsight Credit Assignment for Long-Horizon LLM Agents

Le Critique: Privileged Value Functions for LLM Reinforcement Learning