The belief-action gap in language model agents

Ayse Sila Okcu; Weisheng Wang; Tongmeng Xie; Niall Dalton; Angelos Nalmpantis; Raghu Arghal; Mario Giulianelli

cs.LGOct 12, 2026

The belief-action gap in language model agents

Authors: Ayse Sila Okcu, Weisheng Wang, Tongmeng Xie, Niall Dalton, Angelos Nalmpantis, Raghu Arghal, Mario Giulianelli

Organizations: University of Cambridge · Utrecht University · University of York · University College London, University of London ·

Abstract

Language model (LM) agents can fail for different reasons, requiring different explanations and interventions. We study how state beliefs, action values, and action preferences interact as LM agents reason and commit to decisions in text-rendered grid worlds. We estimate task-relevant beliefs using report-based and representation-based probes, and action values using inverse reinforcement learning. We find that task-relevant state and value information is often represented more reliably than it is reflected in behaviour. Tracking action preferences throughout reasoning further shows that decisions sharpen around a commitment point and reported belief uncertainty decreases near commitment, although individual belief changes do not consistently explain shifts in action preference. Finally, forced actions follow reasoning prefixes from other decisions when transplanted, whereas the tested reasoning-window activation edits provide limited control over the action. Together, these results suggest that some failures arise not from missing task-relevant information, but from how available information is translated into action. Our measurements distinguish information availability from belief-action consistency without by themselves identifying the cause of each failure. Our code is available at: https://anonymous.4open.science/r/belief-action-gap-F745/

Figures & tables

Panel of Figure 1 — Figure 1: Example DoorKey environment shown in text and spatial form. The language model receives the text-rendered grid in (a) , while (b) visualises the same underlying environment used to define states, transitions, and planner-optimal actions. $A$ denotes the agent, $K$ the key, $D$ the door, $G$ the goal, and filled cells denote walls.

	All Steps		All Probes Correct		Probe Held-out
Quantity	Pre	Post	Pre	Post	Pre	Post
Expected local regret $\Delta$	0.681	0.582	0.676	0.554	0.700	0.670
Belief–action gap $R$	0.467	0.268	0.514	0.288	0.316	0.199
Value of perfect state information $I$	0.214	0.314	0.163	0.266	0.384	0.472
Decoded Belief entropy $H$	1.494	2.085	1.106	1.811	2.788	3.020

Table 1: Mean expected local regret

\Delta

, belief–action gap

R

, expected value of perfect state information

I

, and decoded belief entropy

H

before and after reasoning. The expected local regret decomposes exactly as

\Delta=R+I

Figure 2: Example expected local regret under the pre-reasoning and post-reasoning conditions. Cell numbers show the probe-estimated location probabilities, with shading proportional to probability mass. Arrows mark the successor state preferred by the agent-supervised value model and the action actually emitted by the agent. In this example, reasoning increases the decoded belief entropy ( $\operatorname{H}:0.261\rightarrow 1.572$ ) while reducing the expected local regret ( $\mathrm{\Delta}:2.386\rightarrow 0.205$ ).

Figure 3: Evolution of action preference over reasoning (95 trajectories, 1,276 decisions, 98,310 evaluated prefixes). left. The mean probability assigned to the eventual action and inverse normalised action entropy as reasoning progresses, together with the same quantities before the first decisional action mention. right. The probability assigned to the eventual action separately by action. Bands are 95% trajectory-clustered bootstrap intervals.

Figure 4: Decodability and intervention effectiveness at the reasoning-window site. Left: decodability of action decisiveness against positional and length-based baselines. Right: effectiveness of the tested activation interventions, distinguishing any change in the forced action from changes to the intended target action.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Query family	State–question pairs	Greedy	Ten-sample modal
Wall	132	93.9%	97.0%
Goal adjacency	132	100.0%	100.0%
Door adjacency	132	99.2%	100.0%
Key adjacency	132	67.4%	97.0%

Appendix

Table 2: Report accuracy on the 33-state failure-mode set. Each family contains four directional questions per state. The modal result uses ten samples per question at temperature 0.7.

Direction	Report-based	Probe (Pre-reasoning)	Probe (Post-reasoning)
left	100.0%	73.3%	73.3%
right	100.0%	86.7%	93.3%
up	100.0%	100.0%	93.3%
down	100.0%	80.0%	93.3%

Appendix

Table 3: Accuracy of report-based and representation-based estimates of whether the adjacent cell in each direction is blocked, evaluated on the same 15 decision states.

Metric	Pre-reasoning	Post-reasoning
has_key accuracy	100.0%	89.5%
door_open accuracy	100.0%	88.6%
Location top-1 accuracy	34.8%	13.1%
Top-1 within Manhattan distance 1	95.5%	60.8%
Top-1 within Manhattan distance 2	99.0%	83.5%
Mean probability mass within distance 1	93.1%	59.2%

Appendix

Table 4: Expanded representation-based state decoding on held-out environment states. Location top-1 accuracy requires an exact match. “Top-1 within Manhattan distance

k

” measures whether the most probable decoded location lies within distance

k

of the true location. “Mean probability mass within distance

k

” is the probability assigned to all locations within that distance of the true position.

Figure 5: Example learned state values for planner-supervised IRL (left) and agent-supervised IRL (right), using post-reasoning activation-derived state representations. Darker cells indicate larger learned values. Because the one-step objective identifies relative successor-state preferences, absolute values should not be interpreted as goal distances; the comparison is intended to visualise differences in the learned state-preference structure.

Panel of Figure 6 — Figure 6: Additional commitment timing and stability results. (a) Cumulative distributions of the first decisional action mention, the action-commitment prefix, and the point after which the eventual-action probability remains above 0.8. (b) Left: probability that the preferred action changes at the next evaluated prefix as reasoning progresses. Right: probability of at least one later preference switch as a function of the current probability assigned to the eventual action.

Figure 7: Sentence-function distributions around action commitment for 1,004 decisions and 141,377 sentences. The top panel compares function frequencies before and after commitment. The bottom panel shows their local evolution around the commitment position and compares the observed change with a position-matched placebo. Meta-checking should be interpreted as exploratory because its annotation precision is low.

Measure	Before	At or after	Change [95% CI]
Belief entropy (bits)	0.392	0.336	$-0.056\;[-0.105,-0.009]$
Report accuracy	82.8%	83.0%	$+0.2\;[0.0,0.6]$ pp

Appendix

Table 5: Changes in report-based beliefs around retrospective action commitment. Values are averaged within decisions before being aggregated across decisions. Intervals are 95% trajectory-clustered bootstrap intervals.

Transition	Events	Any report change	$\Delta$ error [95% CI]
Any action change	682	14 (2.1%)	$0.0005\;[-0.0022,0.0027]$
Optimal to suboptimal	161	6 (3.7%)	$0.0000\;[-0.0065,0.0055]$
Suboptimal to optimal	182	3 (1.6%)	$-0.0018\;[-0.0080,0.0017]$

Appendix

Table 6: Changes in report correctness at action transitions. “Any report change” counts events where the correctness of at least one of the six current-state reports changes. Error changes compare the preceding sentence with the transition sentence.

Cluster	True commitment	Matched control	Excess
$C_{10}$	14.0%	5.9%	$+8.2$ pp
$C_{9}$	11.6%	8.4%	$+3.2$ pp
$C_{2}$	9.3%	6.8%	$+2.5$ pp
$C_{5}$	15.1%	13.5%	$+1.6$ pp
$C_{7}$	5.5%	4.5%	$+1.0$ pp

Appendix

Table 7: Frequency of selected activation clusters at true commitment sentences and position-matched control sentences from the same decisions.

Cluster	Action change	Optimal to suboptimal	Suboptimal to optimal
$C_{6}$	$1.65\times$	$2.07\times$	$1.77\times$
$C_{10}$	$1.64\times$	$1.48\times$	$1.59\times$
$C_{9}$	$1.41\times$	$1.45\times$	$1.44\times$
$C_{5}$	$1.33\times$	$1.26\times$	$1.35\times$

Appendix

Table 8: Enrichment of sentence-level activation clusters at changes in preferred action and preferred-action optimality relative to within-decision shuffled-event controls.

Figure 8: Whole-chain reasoning-text patching. The prompt from decision $A$ is paired with a reasoning prefix from decision $B$ , whose emitted action differs from that of $A$ . Across early, middle, and late cuts, the forced-action distribution preferentially follows the transplanted reasoning rather than the original prompt. The experiment uses 396 mismatched pairs and 396 congruent controls.

Figure 9: Single-sentence intervention results across 240 reasoning prefixes and 720 independent forced-action evaluations. Left: probability that the forced top action becomes the injected runner-up action as a function of current action confidence. Right: mean probability movement of the injected action under counter-evidence, neutral, and supportive interventions.

Figure 10: Belief–action decomposition: worked example and empirical summary. (A–B) A constructed DoorKey example assigns belief probability to nine positions, all with no key and a locked door; blank belief cells have zero probability. The green map assigns values to all 39 non-wall positions. For this illustration only,

g(x,y,h,d)=v(x,y)

across valid states; the door-cell value applies when the door is open. With

Q(s,a)=g(T_{G}(s,a))

, the emitted RIGHT action has expected value

2.47

, the best single action UP has value

3.95

, and choosing after learning the state yields

5.34

. Thus the belief–action gap is

R=1.48

, the value of perfect state information is

I=1.39

, and expected local regret is

\Delta=R+I=2.87

. RIGHT is blocked at the outlined actual position. These beliefs, values, and action are illustrative, not fitted observations or task rewards. (C) Empirical endpoint means across all 3,194 decisions ( Table 1 ), using representation-based beliefs and agent-supervised value models fitted separately before and after reasoning.

R

decreases, while

I

and decoded-belief entropy

H

increase.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 237 papers

Lost with a Map: Conversational State and Behavioral Reliability in Language Models
Sep 27, 2026Atahan Dokme, Larry HeckSpoken Dialogue Systems LLM Reliability
Georgia Institute of Technology
Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents
Oct 19, 2026Pedram Mouseli, Massieh MoayediCoT Reasoning Multi-Turn Language Model Reasoning
University of Toronto
CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models
Aug 21, 2026Souptik Kumar Majumdar, Fabian Kögel, Andreas BullingVLM Reasoning Reasoning for VLA Models
Universität Stuttgart 70569, Stuttgart Germany

Sep 27, 2026J/K move · Enter open · S save

Lost with a Map: Conversational State and Behavioral Reliability in Language Models

Atahan Dokme, Larry Heck

Georgia Institute of Technology

Spoken Dialogue Systems LLM Reliability State Tracking Belief Updating in LLMs Task-Oriented Dialogue Systems

Open paper AskPDF arXiv

Task-oriented dialogue requires maintaining and updating information across turns, yet language models expose no explicit belief-state object. We study how conversational state is represented, updated, and used inside eight instruction-tuned language models from four families on MultiWOZ and SGD. Structure and values separate: which domains, slots, and requests are active is linearly readable just before the model acts, whereas exact values are far more readable where the user stated them. After a user changes a value, both values remain accessible at their mentions, and causal interventions show that both continue to influence the model's action. In natural closed-loop interaction, query failures separate into cases of weak structural support, incorrect value resolution, and failure to deploy otherwise-supported constraints, with targeted interventions producing systematically different repair behavior across these cases. These findings motivate a state-action controller that starts from the base model action and selectively edits it using structural readouts, without requiring a complete predicted belief state as an intermediate representation. On held-out MultiWOZ interaction across five models, it raises the base model exact-query accuracy from .318 to .621 and task success from .272 to .371 at negligible added cost. Overall, reliable interaction requires not only retaining conversational information, but resolving which available constraints currently apply and ensuring that they govern action.

Abstract

LLM Decision-Making

Explore similar work

Lost with a Map: Conversational State and Behavioral Reliability in Language Models

Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents

CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models