Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents

Pedram Mouseli; Massieh Moayedi

cs.LGOct 19, 2026

Computation, Not Memory: Chain-of-Thought and State Tracking in LLM Agents

Authors: Pedram Mouseli, Massieh Moayedi

Organizations: University of Toronto

Abstract

Language-model agents often reason before they act, and the value of monitoring that reasoning depends on how it relates to their actions. We ask how an agent's actions depend on its reasoning when the correct action depends on a changing state. In a small file-system task, background events announce which file's lock has been toggled but not its new value. The agent therefore has to reconstruct the current locks from its history before it can move files safely. With Qwen3-8B and Qwen3.5-9B, we separate two roles of reasoning: generating it before each action, and retaining earlier reasoning in the prompt. Most of the reduction in lock-violating commands comes from reasoning written at the moment of action. Retaining earlier reasoning adds little to rule-following, although it helps one model complete the task. The benefit depends on what the reasoning says. Filler text, reasoning written for another history and an accurate table of the state do not reproduce it. When we give the agent reasoning written for a history that differs in one lock value, the agent acts on that reasoning even against its own history. Without reasoning, linear probes recover the state from the model's activations about as well as a text classifier recovers it from the transcript. However, decoding accuracy does not predict which interventions change the agent's actions. In this setting, the benefit of reasoning for rule-following comes mainly from generating it at each step. The state that can be decoded without reasoning is a limited guide to how the agent acts.

Figures & tables

Figure 1: Design (schematic). (a) An abridged episode. Lock states are given at the start and then toggled by announced events whose result is not stated. (b) Rollout conditions vary whether the agent reasons before acting (rows) and which part of each earlier reply is retained in later prompts (columns). Observations and executed commands are retained in every condition. (c) Fixed-history arms: the same saved prompt is completed with different contents of the thought block.

Figure 2: Generation versus persistence. (a) Violation rate, Eq. ( 2 ) (violations/attempts above the bars). (b) Effects of generating and of retaining reasoning on the violation rate,

\Delta_{\text{gen}}

and

\Delta_{\text{ret}}

in Eq. ( 1 ), in percentage points and compared episode by episode. (c) Strict task success. Base: Qwen3-8B-Base in the plain-text format (right of the dashed lines). Qwen3-8B: 500 episodes per condition; other models: 300. Error bars: 95% episode-bootstrap intervals.

Model	Arm	AUROC	Viol./Att.	Rate per attempt (%)	$\Delta$ Viol. per 100 prompts
Qwen3-8B	no reasoning	0.866	21/43	48.8 [34.0, 64.1]	—
	filler	0.847	27/58	46.6 [32.8, 60.4]	$+1.3$ [ $-0.7$ , 3.4] †
	other-history reasoning	0.572	23/28	82.1 [65.6, 95.5]	$+0.3$ [ $-2.7$ , 3.3] †
	state table (oracle)	1.000	32/101	31.7 [22.8, 41.3]	$+2.8$ [0.5, 5.1]
	own reasoning	0.935	3/52	5.8 [0.0, 13.0]	$-4.5$ [ $-6.6$ , $-2.4$ ]
Qwen3.5-9B	no reasoning	0.759	21/52	40.4 [27.1, 53.1]	—

Table 1: Fixed histories (400 held-out prompts per model; 144 episodes for Qwen3-8B and 90 for Qwen3.5-9B). AUROC of the state readout, Eq. ( 3 ), over the four files; lock-violating commands and eligible attempts; the violation rate per attempt; and the change in violations per 100 prompts relative to the no-reasoning arm. Brackets: 95% episode-bootstrap intervals. † The filler and other-history arms come from a second evaluation pass over the same prompts, whose own no-reasoning arm had 22 violations in 44 attempts (Qwen3-8B) and 23 in 54 (Qwen3.5-9B); their changes are computed against that pass.

Figure 3: Reasoning swap on counterfactual pairs of log files (Qwen3-8B: 114 pairs from 68 episodes; Qwen3.5-9B: 104 pairs from 29 episodes). Each history (locked or unlocked version) is completed with the reasoning written for the locked version (hatched) or for the unlocked version (solid). Solid bars on the unlocked history and hatched bars on the locked history show the model’s own reasoning. (a) Share of pairs in which the greedy command moves the file (counts above bars); in the locked history, this move violates the rule. (b) Mean log-probability of the command mv

f

. (c) Mean state readout

s_{f}

. Error bars: 95% episode-bootstrap intervals.

Figure 4: Activation patching with the state readout (Qwen3-8B, 120 held-out counterfactual pairs; Qwen3.5-9B in Appendix I ). (a) Recovered fraction

R

, Eq. ( 4 ), when the activations of the locked version are copied into the unlocked version at one layer and one group of positions. (b) Patching only the component of the difference along one direction at the final token. The directions, each estimated on separate training data, are the mean difference between counterfactual pairs; the mean difference between locked and unlocked cases on ordinary prompts (a mass-mean probe); logistic probes fitted at the same token or at the agent prompt; the difference of the unembedding vectors of the two answers; and a random direction. Bands: 95% episode-bootstrap intervals.

Figure 5: State readout and direct action on identical prefixes. For each counterfactual pair, the same prefix (the history up to the current command) is followed either by the state question or by CMD: . The plot shows the recovered fraction, Eq. ( 4 ), of the state readout (blue) and of

\log p(\text{mv }f)

(orange) when the locked version’s activations are patched at the source token, at the rest of the history, or at the question. Qwen3-8B: 120 pairs from 74 episodes; Qwen3.5-9B: 119 pairs from 38 episodes. Bands: 95% episode-bootstrap intervals.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Condition	Violations / attempts	Rate % [95% CI]	Attempts / episode
Qwen3-8B	direct/cmd	130 / 570	22.8 [19.5, 26.2]	1.14
	think/cmd	14 / 737	1.9 [1.0, 3.0]	1.47
	think/body	17 / 740	2.3 [1.2, 3.5]	1.48
	think/full	10 / 696	1.4 [0.6, 2.4]	1.39
Qwen3.5-9B	direct/cmd	72 / 474	15.2 [11.6, 18.8]	1.58
	think/cmd	20 / 415	4.8 [2.8, 7.0]	1.38

Appendix

Table 2: Lock-violating commands among eligible attempts over all turns, with 95% episode-bootstrap intervals, and eligible attempts per episode.

Figure 7: Effects of generating reasoning ( direct/cmd minus think/cmd ) and of retaining it ( think/cmd minus think/full ) on the violation rate, in percentage points, for all five rollout runs (compared episode by episode; 95% intervals). “Bound” is the one-sided 95% upper bound on the ratio of the persistence effect to the generation effect: for example, 7.6% means that, with 95% confidence, retaining reasoning has at most 7.6% of the effect of generating it. The bound is not shown when the generation effect is close to zero.

Figure 8: (a) Characters of reasoning generated per turn (mean, log scale) and (b) share of turns with fewer than 20 characters of reasoning, by run and persistence setting.

Model	Condition	Strict success	Stricter	Goal locked	Logs left	Archived	Lock on log
Qwen3-8B	direct/cmd	0.44	0.34	0.75	0.29	0.82	0.40
	think/cmd	0.76	0.76	0.87	0.09	1.43	0.00
	think/body	0.74	0.74	0.87	0.09	1.43	0.00
	think/full	0.76	0.76	0.86	0.11	1.36	0.00
Qwen3.5-9B	direct/cmd	0.35	0.35	0.57	0.34	1.10	0.00
	think/cmd	0.53	0.53	0.72	0.23	1.28	0.00

Appendix

Table 3: Task outcomes per condition. Strict success as defined in Section 3 ; the stricter version also fails episodes with a lock command on a log file. Goal locked: report.txt locked at the end. Logs left: unlocked log files left in /logs at the end, per episode. Archived: log files successfully moved, per episode. Lock on log: lock commands aimed at log files, per episode.

Figure 9: Task outcomes per episode. (a) Goal file locked at the end; (b) unlocked log files left in /logs ; (c) lock commands aimed at log files; (d) goal file locked at the end, among episodes in which it was unlocked at some point. Error bars: 95% episode-bootstrap intervals.

Figure 10: (a,b) Lock-violating commands per eligible attempt at each turn; grey lines mark where a toggle is announced. (c) Share of opportunities (every file in /logs at every turn) at which the agent moves or deletes the file, for locked (solid) and unlocked (hatched) files, split by the history of the file: as stated (no toggle and no lock / unlock by the agent) or ever toggled. Error bars: 95% episode-bootstrap intervals.

Figure 11: Commands per episode by command, for each primary condition.

Figure 12: Qwen3 models from 0.6B to 14B parameters (300 episodes per condition, chat format). (a) Violation rate (

n

: eligible attempts); (b) eligible attempts per episode; (c) reasoning per turn and share of turns that reach the 2,500-token budget ( think/body ); (d) strict task success. Error bars: 95% episode-bootstrap intervals.

Figure 13: (a) Qwen3-8B with 8- and 16-turn episodes. (b) Goal swap: the task names alpha.log instead of report.txt as the file that must be locked at the end. Error bars: 95% episode-bootstrap intervals.

Figure 14: Fixed histories (400 prompts per model). (a) State-readout AUROC over the four files; for the state table this is a check that the model reads the table. (b) Lock-violating commands per 100 prompts (counts above bars). (c) Violations per eligible attempt, with the number of attempts. Error bars: 95% episode-bootstrap intervals.

Figure 15: Fixed histories in detail. (a,b) State-readout AUROC per file and arm. (c) Next command in each arm on the 400 Qwen3-8B prompts.

Figure 16: Counterfactual log-file pairs (Qwen3-8B: 114 pairs from 68 episodes; Qwen3.5-9B: 104 pairs from 29 episodes). (a) State-readout separation,

s_{f}

in the locked version minus

s_{f}

in the unlocked version. (b) Action separation, the log-probability of mv

f

in the unlocked version minus that in the locked version. (c) Share of pairs in which the greedy command moves the file, in each version (counts above bars). Reasoning is generated separately for each version. Error bars: 95% episode-bootstrap intervals.

		Qwen3-8B (114 pairs)			Qwen3.5-9B (104 pairs)
History	Reasoning written for	Moves	$\log p$	$s_{f}$	Moves	$\log p$	$s_{f}$
locked	locked version	0	$-29.3$	8.7	0	$-12.7$	4.3
locked	unlocked version	43	$-13.6$	$-3.2$	41	$-7.0$	1.5
unlocked	locked version	0	$-21.8$	5.4	0	$-10.1$	3.2
unlocked	unlocked version	45	$-12.1$	$-6.2$	41	$-6.5$	0.8

Appendix

Table 4: Reasoning swap on log-file pairs. Rows: the version of the history and the version for which the reasoning was written. Moves: pairs whose greedy command moves

f

(in the locked history, a lock-violating command).

\log p

: mean log-probability of the command “ mv

f

”.

s_{f}

: mean state readout,

\log p(\text{locked})-\log p(\text{unlocked})

after the state question; for an individual example, a positive score means that “locked” is more likely than “unlocked”.

Figure 17: Targeted edit of the model’s own reasoning on log-file pairs. Blue: the unchanged reasoning. Grey (control): the lock words in the explicit statements about

f

are written in capital letters (“LOCKED”), which keeps their meaning. Orange: the same lock words are replaced by the opposite value.

n

: pairs whose reasoning contains at least one editable statement. (a) Share of pairs whose greedy command moves

f

(counts above bars); (b) mean log-probability of mv

f

; (c) mean state readout

s_{f}

. Error bars: 95% episode-bootstrap intervals.

Lock variable	Layer	Probe	Shuffled labels	TF-IDF	Last value	Replay
alpha.log	16	.792	.504	.754	.774	1.000
beta.log	4	.800	.465	.792	.788	1.000
gamma.log	12	.762	.490	.751	.776	1.000
report.txt	20	.811	.501	.803	.703	1.000
Mean		.791	.490	.775	.760	1.000

Appendix

Table 5: Held-out balanced accuracy of lock-state classifiers on Qwen3-8B think/body (test set: 150 episodes, 1,155 turns). Probes: logistic regression on the last prompt token, layer (counted from 0) chosen on validation data. Last value: the last lock value written for the file. Replay: symbolic replay of the history.

Figure 18: Probes on Qwen3-8B think/body (test set). (a) Balanced accuracy by layer (solid) and with shuffled labels (dotted); dots mark the layer chosen on validation data. (b) The selected probe against the TF-IDF classifier and the last-value rule. (c) Probe accuracy by the number of turns since the file’s lock value was last written in the visible text.

Figure 19: State-readout AUROC before and after rewriting every visible statement of the file’s lock value (Qwen3-8B, 300 prompts per file).

Figure 20: Activation patching with the state readout on Qwen3.5-9B (120 held-out pairs from 42 episodes), as in Figure 4 .

Figure 21: Decoding accuracy against patching effect. Each point is one direction at one late layer: its decoding AUROC on held-out ordinary prompts (

x

) and the share of the full final-token patch that it recovers (

y

Figure 22: Activation patching with the direct action as output,

x=\log p(\text{mv }f\mid\text{prompt ending in {CMD:}})

, on Qwen3-8B (a,b; 120 pairs from 71 episodes) and Qwen3.5-9B (c,d; 112 pairs from 33 episodes). (a,c) Recovered fraction by position group; bands are 95% episode-bootstrap intervals. (b,d) At the last prompt token, the full patch and patches along single directions: the action’s own mean difference, the two state-readout directions of Figure 4 b, and a random direction.

Figure 23: The history group of Figure 22 split into non-overlapping parts, each patched on its own (same pairs). Later mentions of

f

are averaged over the pairs that contain one (29 and 49 pairs); the other curves over all pairs. Bands: 95% episode-bootstrap intervals.

Figure 24: Controls for the history split (same pairs; some layers). (a,b) In pairs with a later mention of

f

: patching the tokens of these mentions against patching as many randomly chosen mentions of other files, and the paired difference with a 95% episode-bootstrap interval. (c) The rest of the later turns split into mentions of other files, chat-template tokens and all other tokens, at the layer where the history carries most (numbers: mean tokens).

Figure 25: Shift of the state readout towards “locked” when a direction is added at the final token (Qwen3-8B, 300 held-out prompts per file). Grey: mean

\pm

2 standard deviations and maximum over 24 random directions. Solid: answers about the target file; dashed: answers about the other files under the same edit.

\alpha=64

equals the mean residual norm at this position.

Run	Model	Conditions	Episodes	Budget	Context	At budget	Max tokens	GPU time
Main (chat)	Qwen3-8B	3 think + direct	500	4,000	32,768 a	2.7–6.9%	13,571	5.4 h
Second run (chat)	Qwen3-8B	3 think + direct	300	2,500	24,576	6.8–28.7%	10,357	2.8 h
Second model (chat)	Qwen3.5-9B	3 think + direct	300	1,200	16,384	2.0–4.8%	7,752	0.8 h
Plain text	Qwen3-8B	think/cmd, think/full, direct	300	1,200 / 400	16,384	1.8% / 15.8%	3,779	0.2 h
Plain text	Qwen3-8B-Base	think/cmd, think/full, direct	300	400	16,384	$<$ 1%	1,408	0.1 h
Scale (chat)	Qwen3-0.6B–14B	think/body + direct	300	2,500	24,576	4.9–29.2%	3,028	2.0 h

Appendix

Table 6: Rollout configurations. All runs use vLLM with bfloat16 weights, temperature 0.7, top-

p

0.9, episode seeds

0,\ldots,N-1

, and one NVIDIA RTX A6000 (48 GB); direct-mode replies are limited to 12 tokens. Budget: maximum tokens per reasoning-mode reply. a 16,384 for the direct condition. At budget: share of reasoning-mode turns whose reply reached the budget. Max tokens: the longest prompt plus reply at the final turn of an episode. GPU time: generation time summed over the conditions of a run.

CoT Reasoning

Chain-of-thought (CoT) reasoning studies how language models use intermediate steps, visible or latent, to solve multi-step tasks.

See all 223 papers

How Strongly Should Task State Influence an LLM Agent?
Sep 22, 2026Chenyu Zhang, Wonbin Kweon, Jiawei HanLLM Agent Evaluation Runtime Enforcement for AI Agents
University of Waterloo · Sungkyunkwan University · University of Illinois Urbana-Champaign
The Window Is in the Reasoning: When LLM Agents Decide to Follow Prompt Injections
Oct 16, 2026Haoxuan Jia, Yang Liu, Bin Chong +13Prompt Injection Attacks on AI Agents Prompt Injection Detection
Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com +6
Plans Fade Fast: Why Context Management Is Load-Bearing for LLM Agents
Jun 22, 2026Aman Mehta, Anupam DattaLong-Horizon LLM Agents LLM Context Management
Snowflake AI Research

Sep 22, 2026J/K move · Enter open · S save

How Strongly Should Task State Influence an LLM Agent?

Chenyu Zhang, Wonbin Kweon, Jiawei Han

University of Waterloo · Sungkyunkwan University · University of Illinois Urbana-Champaign

LLM Agent Evaluation Runtime Enforcement for AI Agents State Tracking Long-Horizon LLM Agents LLM Agent Reliability

Open paper AskPDF arXiv

Long-horizon assigned work requires an LLM agent to track the state of a task: which steps are done, blocked, cancelled, or open to repetition. Agent systems either keep this state as text in the prompt and rely on the model to read that text, or move the state into a module that enforces it, and each system is evaluated as a whole, so no one knows how much reliability comes from the state being shown, told, or enforced. We fix the task rules, the model, and paired episodes and vary how strongly task state reaches the agent: a raw transcript, an exact checklist, per-turn directives from a state machine compiled from the brief and advanced only by execution receipts, or an enforcement gate on that machine that refuses state-violating actions; every episode is scored by exact payload matching against dynamic ground truth. Across three models, two reasoning regimes, and two domains, four findings hold without per-turn reasoning: displaying accurate state is unreliable, an unverified ledger the agent writes itself beats an accurate checklist it is shown, directives help in proportion to the model's obedience, and enforcement needs no obedience but is bounded by the correctness of its state and by the matcher that maps requests to steps; per-turn reasoning at a 235B agent compresses these separations without repairing the text rungs. The same gate, compiled from

τ^2

-bench's airline policy, raises a 235B agent's pass

^1

from 0.39 to 0.54 and changes nothing for a 35B agent that rarely violates the policy; on PM-Bench, where acting turns on recognizing a cue rather than on state, showing the record is the best rung--matching or beating both gates and reversing the ledger-over-checklist finding--and enforcing the matcher's judgement drops a 35B agent below its raw transcript. Enforcement pays when failures are state-decidable and frequent, and hurts when the gate's judgement is wrong.

Abstract

CoT Reasoning

Explore similar work

How Strongly Should Task State Influence an LLM Agent?

The Window Is in the Reasoning: When LLM Agents Decide to Follow Prompt Injections

Plans Fade Fast: Why Context Management Is Load-Bearing for LLM Agents