LLM Value Functions for Credit Assignment in Agentic RL

Kaijie Zhu; Weixiang Yan; Omar Attia; Abbas Kazerouni; Alexander Toshev; Andrew Szot

cs.LGOct 11, 2026

LLM Value Functions for Credit Assignment in Agentic RL

Authors: Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot

Organizations: Apple

Abstract

Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.

Figures & tables

Figure 1: Overview of the proposed framework. ( 1 ) Group rollout: We sample a group of trajectories from the current policy and use their sparse outcome rewards to compute the empirical group success rate

\mu_{G}

. ( 2 ) LLM value estimation: A frozen LLM critic scores all trajectory states, conditioned on group context. Estimates are anchored to empirical outcomes: the initial-state value to the group success rate and the terminal-state value to the actual reward. ( 3 ) Step advantages: We compute unnormalized step advantages as consecutive value differences (

\overline{A}_{t}=V_{\phi}(s_{t+1})-V_{\phi}(s_{t})

) and normalize them across the group, yielding a dense, per-action credit signal. ( 4 ) Policy optimization: These step-level advantages drive standard PPO updates in LVRL, or act as a gating mechanism in LVOPD to restrict teacher supervision to actions that decrease the estimated value.

Table 2

Split	Door distance	Layouts	Median optimal actions	Max optimal actions
Train	1	95	14	34
	2	95	24	48
	3	95	32	60
Evaluation ID	1	100	14	36
	2	100	23	46
	3	100	32	51

Table 5: BabyAI training and evaluation pools. Optimal actions include rotations, forward movements, door toggles, and the final pickup. The three splits use disjoint seed ranges and share no layouts.

Panel of Figure 3 — Figure 3: Representative BabyAI evaluation layouts at increasing door distance. The blue line is the shortest collision-free path from the agent to a cell adjacent to the target; the action count below each panel additionally includes turns, door toggles, and the final pickup. The path crosses one, three, and five doors respectively, which is what door distance measures. Every layout holds the same inventory of $18$ objects, so distractor identities carry no information and only the spatial arrangement and the choice of target vary. The agent never observes this map: it receives a textual egocentric description (Figure 4 ).

Environment / student	Queries	Input tokens	Output tokens	USD / query
AppWorld / Qwen3-4B	92,718	9.01k	0.66k	0.061
AppWorld / Qwen3-8B	99,968	9.43k	0.61k	0.062
BabyAI / Qwen3-8B	8,838	11.86k	1.90k	0.107
R2E-Gym / Qwen3-8B	8,030	20.35k	1.55k	0.140

Table 6: Estimated cost per logged value-estimation query. Token counts are means computed with cl100k_base , an uncalibrated proxy for Claude tokenization. Each query scores one target trajectory. Input counts include peer summaries, but the cost of generating those summaries is excluded. Dollar estimates use standard Claude Opus 4.6 rates.

Setting	AppWorld	BabyAI
Training updates	200	150
Rollout-step budget	50	128
Training nodes	1	2
Training world size	8	16
Minibatch size per rank	4	4
Gradient-accumulation steps	8	4

Table 7: Configured rollout and optimization settings for LVOPD. The effective optimization batch size is the product of the per-rank minibatch size, gradient-accumulation steps, and training world size.

Figure 9: RL training reward curves across three training environments. For AppWorld and BabyAI, curves show the mean over three seeds; shaded regions denote one standard deviation.

Figure 10: OPD training reward curves on AppWorld across model sizes. Curves show the mean over three seeds; shaded regions denote one standard deviation. Both gated variants consistently outperform the OPD baseline after roughly

100

training steps.

	BabyAI
Method	Qwen3-4B	Qwen3-8B
Base	$0.0{\scriptstyle\,\pm\,0.0}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$\mathbf{0.7}{\scriptstyle\,\pm\,0.0}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+OPD	$0.5{\scriptstyle\,\pm\,0.1}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.6{\scriptstyle\,\pm\,0.3}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+Trajectory-gated	$0.4{\scriptstyle\,\pm\,0.2}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.3{\scriptstyle\,\pm\,0.3}$ / $0.0{\scriptstyle\,\pm\,0.0}$
+LVOPD (ours)	$\mathbf{0.7}{\scriptstyle\,\pm\,0.2}$ / $0.0{\scriptstyle\,\pm\,0.0}$	$0.4{\scriptstyle\,\pm\,0.1}$ / $0.0{\scriptstyle\,\pm\,0.0}$

Table 8: Evaluation performance under on-policy distillation.

Threshold	Minimum difference	Worst-case tail probability	Expected spurious events
$1\sigma$	$4/32$	$0.3820$	$664$
$2\sigma$	$8/32$	$0.0599$	$72$
$3\sigma$	$12/32$	$0.0037$	$3.4$

Table 9: Sampling noise for differences between two independent Monte-Carlo estimates with

M=32

. The final column estimates the number of spurious events over the full evaluation set under the null.

Figure 11: Agreement between predicted value changes and Monte-Carlo events across normalized trajectory position

t/L

, where

L

is the number of actions. (a) Fraction of MC events in each position bin for which the predicted value changes in the same direction as the MC estimate: a decrease for pitfalls and an increase for recoveries. (b) Fraction of MC events for which the model predicts no value change (

\Delta_{t}^{\mathrm{pred}}=0

), despite an MC increase or decrease large enough to meet the event threshold (

|\Delta_{t}^{\mathrm{MC}}|\geq 3\sigma

Position	Same sign	No change	Opposite sign
First third ( $n=34$ )	$44\%$	$29\%$	$26\%$
Middle third ( $n=54$ )	$78\%$	$13\%$	$9\%$
Final third ( $n=26$ )	$88\%$	$4\%$	$8\%$

Table 10: Step-level agreement by normalized trajectory position. No change denotes

\Delta^{\mathrm{pred}}=0

. Percentages may not sum to

100\%

because of rounding.

	Task	Outcome	$\Delta^{\mathrm{MC}}$	$\Delta^{\mathrm{pred}}$	Detected
Pitfall, no execution error	3c13f5a_2	✗	$-0.938$	$-0.70$	✓
Recovery despite an error	34d9492_2	✓	$+0.438$	$+0.30$	✓
Missed pitfall	27e1026_1	✗	$-0.500$	$+0.05$	✗

Table 11: The three transitions examined in this section. The predicted value tracks the Monte-Carlo value on the first two transitions and moves in the wrong direction on the third.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 583 papers

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
Apr 12, 2026Zikang Shan, Han Zhong, Liwei Wang +1RL for Language Models Reinforcement Learning
Peking University · Microsoft Research Asia · Shanghai Jiao Tong University
Hindsight Credit Assignment for Long-Horizon LLM Agents
Mar 7, 2026Hui-Ze Tan, Xiao-Wen Yang, Hao Chen +7Credit Assignment in RL Long-Horizon LLM Agents
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · FiT, Tencent, Shenzhen, China +2
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
Aug 17, 2026Siddarth Venkatraman, Matthieu Dinot, Laurence AitchisonRL for Language Models RL for Language Model Reasoning
Mistral AI · Mila – Quebec AI Institute · Université de Montréal

Apr 12, 2026J/K move · Enter open · S save

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Zikang Shan, Han Zhong, Liwei Wang, Li Zhao

Peking University · Microsoft Research Asia · Shanghai Jiao Tong University

RL for Language Models Reinforcement Learning Value Function Estimation Credit Assignment in RL Actor-Critic Methods

Open paper AskPDF arXiv

Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Abstract

RL for LLM Agents

Explore similar work

Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

Hindsight Credit Assignment for Long-Horizon LLM Agents

Le Critique: Privileged Value Functions for LLM Reinforcement Learning