Large language model (LLM) agents need to choose actions whose consequences unfold over multiple future steps, yet poor decisions do not reveal whether the model failed to represent those consequences or failed to use them. Mechanistic interpretability has studied internal representations of environment state and plans, but sequential decision making also requires evaluating how candidate actions change future outcomes. Reinforcement learning formalizes this as action value. We study this computation in Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B across a partially observed Grid environment and WebShop, a simulated web shopping environment. We find that the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. We call this gap between value representation and policy choice value-policy dissociation. To investigate how represented value relates to action selection, we compare direct interventions along the value readout direction with Natural Value Transport (NVT), a transformation learned from hidden state changes that naturally result from differences in action value. While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice.
Figures & tables
Figure 1: Value–policy dissociation from a motivating example to a systematic pattern. Top: a looping Grid trajectory in which the agent repeats an inferior action despite observations favoring a better alternative. Bottom: Generalizing to broader scenarios, a linear readout of the hidden state favors the better action while the policy selects the worse one; this probe–policy mismatch recurs across states.
Grid
Model
Success ↑
Loop rate ↓
Qwen2.5-7B-Instruct
12.64%
81.2%
GPT-OSS-20B
54.57%
32.5%
Qwen2.5-32B-Instruct
58.14%
41.6%
WebShop
Model
Reward ↑
Loop rate ↓
Table 1: Zero-shot performance (Success Rate on Grid and Reward on WebShop) of tested LLM agents. Loop rate is the fraction of trajectories containing at least one local loop of actions (“ A→B→A→B ”).
Figure 2: Layer-wise value readout in Grid and WebShop. (a) Grid held-out top-1 ranking accuracy, measuring whether the action with highest decoded value matches the gold best-value action; the dashed line shows the Manhattan baseline. (b) WebShop held-out pairwise ranking accuracy, measuring whether the readout correctly orders two candidate actions by empirical continuation value; dashed lines show the TF-IDF baselines. Shaded regions show 95% cluster-bootstrap intervals for the learned readouts. Results reported on a held-out validation set.
Policy correct
Policy wrong
Model
Probe correct
Probe wrong
Probe correct
Probe wrong
Grid
Qwen2.5-7B-Instruct
39.6%
1.8%
58.3%
0.3%
GPT-OSS-20B
62.2%
0.1%
37.5%
0.2%
Qwen2.5-32B-Instruct
64.0%
0.2%
34.2%
1.6%
WebShop
Table 2: Joint correctness of policy decisions and value readouts. Each entry reports the percentage of decision states falling into one of four categories: whether the policy choice is correct and whether the value probe correctly identifies the better action. Bold entries highlight states where the policy chooses the worse action while the probe correctly identifies the better one.
Figure 3: Layer-wise steering slope for Natural Value Transport (NVT) and control interventions across Grid and WebShop. Probe (calibrated) uses the scale implied by the value readout; Probe (same-norm), Shuffled NVT, and Random match the magnitude of the NVT perturbation while changing its direction. Positive slope indicates increased policy preference for the action favored by the value readout. NVT identifies a more stable late-layer policy-responsive region across tasks. Results are reported on a held-out validation set.
Task
Model
n
Outcome ↑
Loop ↓
Flip rate
Grid
Qwen2.5-7B-Instruct
100
0.140 → 0.300
85.0% → 64.0%
24.2%
Grid
GPT-OSS-20B
100
0.580 → 0.640
43.0% → 38.0%
16.9%
Grid
Qwen2.5-32B-Instruct
100
0.590 → 0.680
41.0% → 32.0%
37.3%
WebShop
Qwen2.5-7B-Instruct
671
0.473 → 0.501
7.2% → 6.1%
42.3%
WebShop
GPT-OSS-20B
503
0.417 → 0.458
15.5% → 14.9%
16.5%
WebShop
Qwen2.5-32B-Instruct
454
0.358 → 0.452
2.0% → 1.8%
55.1%
Table 3: Behavioral effects of value steering. Paired entries report baseline → steered performance evaluated over n episodes. Outcome denotes success rate for Grid and mean reward for WebShop. Flip rate is the fraction of evaluated decisions whose selected action changes under the intervention.
Setting
Model
Invalid rate ↓
Grid
Qwen2.5-7B-Instruct
0.0% → 0.17%
Grid
GPT-OSS-20B
0.0% → 0.0%
Grid
Qwen2.5-32B-Instruct
0.0% → 0.0%
WebShop
Qwen2.5-7B-Instruct
0.0% → 1.0%
WebShop
GPT-OSS-20B
0.1% → 0.4%
WebShop
Qwen2.5-32B-Instruct
9.7% → 6.2%
Table 4: Invalid-action rates under value steering. Entries report baseline → steered rates.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Probe
Majority Baseline
Model
Cell
Exact map
Cell
Exact map
Qwen2.5-7B-Instruct
98.20
79.73
60.69
0.00
Qwen2.5-32B-Instruct
97.89
79.02
60.69
0.00
GPT-OSS-20B
96.28
72.18
61.46
0.00
Appendix
Table 5: Readout of the 25-cell F/O/U state map. Probe layers are selected by validation macro-F1. The majority baseline assigns each position its most frequent training label. Exact-map accuracy requires all 25 cells to be correct.
Figure 4: Layer-wise linear readout of the full Grid value map. Left: MAE over all finite cells in the 5×5 optimistic distance-to-goal field. Right: MAE on detour cells, where known obstacles make the shortest-path distance differ from Manhattan distance. Top: continuous predictions. Bottom: predictions rounded to the nearest integer before computing MAE. Dashed and dotted lines denote the Manhattan baselines for the shared Qwen states and the GPT-OSS native states, respectively. Markers indicate layers selected by validation detour-cell MAE.
Figure 5: Example global value-map readout on a held-out Grid state. Left: gold optimistic remaining distance to the goal. Right: prediction from the linear readout. Known obstacles are marked by O , the goal by G , and dashed cells indicate locations whose shortest-path distance differs from Manhattan distance because of the observed obstacles.
Model
Max Steps
Reward
Purchase Rate
Mean Steps
Loop Rate
Qwen2.5-7B-Instruct
15
0.509
76.0%
6.63
8.9%
Qwen2.5-7B-Instruct
30
0.517
78.0%
9.80
9.5%
GPT-OSS-20B
15
0.250
41.0%
10.89
36.3%
GPT-OSS-20B
30
0.284
46.9%
19.43
53.3%
Qwen2.5-32B-Instruct
15
0.293
51.5%
7.01
29.4%
Qwen2.5-32B-Instruct
30
0.297
52.8%
11.28
30.0%
Appendix
Table 6: WebShop performance under different maximum interaction budgets. All models are evaluated on the same 1,502 shared tasks. Reward is averaged over tasks, purchase rate denotes the fraction of episodes containing a click[Buy Now] action, and loop rate follows the trajectory-level loop definition used in the main experiments.
Task
Model
Sensor
Actuator
α
Grid
Qwen 7B
19
21
12.5
Grid
GPT-OSS 20B
17
20
4
Grid
Qwen 32B
51
50
12.5
WebShop
Qwen 7B
17
21
8
WebShop
GPT-OSS 20B
17
15
12.5
WebShop
Qwen 32B
36
51
8
Appendix
Table 7: Configurations used for closed-loop value steering. Sensor and actuator refer to the value-readout and NVT intervention layers, respectively. Layer indices are zero-based.
Language-model agents often reason before they act, and the value of monitoring that reasoning depends on how it relates to their actions. We ask how an agent's actions depend on its reasoning when the correct action depends on a changing state. In a small file-system task, background events announce which file's lock has been toggled but not its new value. The agent therefore has to reconstruct the current locks from its history before it can move files safely. With Qwen3-8B and Qwen3.5-9B, we separate two roles of reasoning: generating it before each action, and retaining earlier reasoning in the prompt. Most of the reduction in lock-violating commands comes from reasoning written at the moment of action. Retaining earlier reasoning adds little to rule-following, although it helps one model complete the task. The benefit depends on what the reasoning says. Filler text, reasoning written for another history and an accurate table of the state do not reproduce it. When we give the agent reasoning written for a history that differs in one lock value, the agent acts on that reasoning even against its own history. Without reasoning, linear probes recover the state from the model's activations about as well as a text classifier recovers it from the transcript. However, decoding accuracy does not predict which interventions change the agent's actions. In this setting, the benefit of reasoning for rule-following comes mainly from generating it at each step. The state that can be decoded without reasoning is a limited guide to how the agent acts.
Language model (LM) agents can fail for different reasons, requiring different explanations and interventions. We study how state beliefs, action values, and action preferences interact as LM agents reason and commit to decisions in text-rendered grid worlds. We estimate task-relevant beliefs using report-based and representation-based probes, and action values using inverse reinforcement learning. We find that task-relevant state and value information is often represented more reliably than it is reflected in behaviour. Tracking action preferences throughout reasoning further shows that decisions sharpen around a commitment point and reported belief uncertainty decreases near commitment, although individual belief changes do not consistently explain shifts in action preference. Finally, forced actions follow reasoning prefixes from other decisions when transplanted, whereas the tested reasoning-window activation edits provide limited control over the action. Together, these results suggest that some failures arise not from missing task-relevant information, but from how available information is translated into action. Our measurements distinguish information availability from belief-action consistency without by themselves identifying the cause of each failure. Our code is available at: https://anonymous.4open.science/r/belief-action-gap-F745/
Ayse Sila Okcu, Weisheng Wang, Tongmeng Xie +4
University of Cambridge · Utrecht University · University of York +3
Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.