Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations

Ahmed Karim; Fatima Sheaib; Zein Khamis; Maggie Chlon; Jad Awada; Leon Chlon

stat.MLFeb 22, 2026

Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations

Authors: Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon

Organizations: Hassana Labs · University of Oxford

Abstract

Large language models can follow complex procedures yet fail at a seemingly trivial final step: reporting a value they themselves computed moments earlier. We study this phenomenon as procedural hallucination: failure to execute a verifiable, prompt-grounded specification even when the correct value is present in context. In long-context binding tasks with a known single-token candidate set, we find that many errors are readout-stage routing failures. Specifically, failures decompose into Stage~~2A (gating) errors, where the model does not enter answer mode, and Stage~~2B (binding) errors, where it enters answer mode but selects the wrong candidate (often due to recency bias). In the hard regime, Stage~~2B accounts for most errors across model families in our tasks (Table~~1). On Stage~~2B error trials, a linear probe on the final-layer residual stream recovers the correct value far above chance (e.g., 74% vs.\ 2% on Qwen2.5-3B; Table~~2), indicating that the answer is encoded but not used. We formalize ``present but not used'' via available vs.\ used mutual information and pseudo-prior interventions, yielding output-computable diagnostics and information-budget certificates. Finally, an oracle checkpointing intervention that restates the true binding near the query can nearly eliminate Stage~~2B failures at long distance (e.g., Qwen2.5-3B $0/400 \rightarrow 399/400$ at $k = 1024$ ; Table~~8).

Figures & tables

Figure 1: Framework overview: Stage 2A gating and Stage 2B binding failures correspond to low routing efficiency

I_{\mathrm{used}}/I_{\mathrm{avail}}

, diagnosed via pseudo-prior interventions.

competing_vars	primacy_recency
KEY1 = [apple]	KEY = [alpha]
[k filler tokens]	[k filler tokens]
KEY2 = [banana]	KEY = [beta]
[k filler tokens]	[k filler tokens]
What is KEY1? KEY1 = [	KEY = [gamma]
	[k filler tokens]

Table 2

Figure 2: Spotlight summary of core empirical claims. (A) Stage decomposition for representative hard-regime settings: most errors are Stage 2B (binding) rather than Stage 2A (gating), i.e., the model enters answer mode but selects the wrong candidate. (B) Checkpointing (restating the true binding every 128 tokens near the query) substantially recovers long-distance binding for Qwen2.5-3B; on competing_vars at

k=1024

, it converts 0/400

\to

399/400 correct. Error bars are 95% Wilson binomial confidence intervals over

n=400

trials per cell.

Model	Task	$k$	Acc	Cand Acc	Frac 2A	Frac 2B	GateGap	ValueGap
Qwen2.5-3B	competing	256	0.107	0.158	0.350	0.650	0.56	-1.81
Qwen2.5-3B	primacy	256	0.043	0.045	0.185	0.815	0.96	-3.40
Qwen2.5-3B-Inst.	competing	256	0.276	0.360	0.402	0.598	0.61	-0.77
Qwen2.5-3B-Inst.	primacy	256	0.061	0.059	0.072	0.928	2.10	-3.96
Gemma-2-2B	competing	256	0.995	0.995	0.000	1.000	3.03	2.28
Gemma-2-2B	primacy	256	0.801	0.804	0.013	0.987	1.65	0.47

Table 1: Stage decomposition (

n=800

, greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

single-token candidates). Acc is

\mathbb{P}(\hat{Y}=V)

(top token), Cand-Acc is

\mathbb{P}(Y=V)

(best candidate). Frac-2A/Frac-2B are fractions of errors (conditional on

\hat{Y}\neq V

), decomposing errors into gating (non-candidate output) vs. binding (wrong candidate) failures. For Llama we report

k=2048

to reach comparable difficulty.

Task	$k$	Acc	Cand Acc	Frac 2B	Probe@0	Probe@18	Probe@35
competing	256	0.200	0.255	0.641	0.019	0.083	0.739
competing	512	0.000	0.000	0.968	0.008	0.029	0.390
primacy	256	0.069	0.074	0.754	0.020	0.040	0.354
primacy	512	0.000	0.000	0.994	0.020	0.020	0.124

Table 2: Probing on Stage 2B errors (Qwen2.5-3B,

n=800

; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Frac-2B is a fraction of errors (conditional on

\hat{Y}\neq V

) attributable to Stage 2B binding failures. Layers 0/18/35 are embedding, mid, and final residual streams. Chance is

\approx

2%. At

k=256

, the final-layer probe recovers the correct answer on 74% of error trials.

Model	Task	$k$	Baseline	+Checkpoint
Qwen2.5-3B	competing	1024	0.000	0.998
Qwen2.5-3B	primacy	1024	0.000	0.763
Llama-3.2-3B-Inst.	competing	2048	0.000	0.853
Llama-3.2-3B-Inst.	primacy	1024	0.003	0.915
Gemma-2-2B-it	primacy	1024	0.440	0.968

Table 3: Checkpointing summary (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Oracle checkpointing inserts a compact serialization of the true binding every 128 tokens in the tail filler, and can convert near-zero accuracy into near-perfect accuracy at long distance; see Appendix Table 8 for full results. Tasks are competing_vars ( competing ) and primacy_recency ( primacy ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$k$	Acc(%)	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
Mistral-7B (baseline)	1024	98.3	54.8	100.0	97.0	3.0
Mistral-7B (baseline)	2048	87.0	68.6	99.5	63.7	35.8

Appendix

Table 4: Trace-budget audit (

\tau=0.75

). At large distance, errors concentrate in flagged traces.

Variant	$n$	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)	$\mathbb{E}[p_{1}]$	$\mathbb{E}[p_{0}^{\mathrm{NE}}]$
baseline	369	21.1	83.3	61.5	21.8	0.398	0.036
rag	378	25.1	81.1	60.4	20.6	0.428	0.039
checkpoint	383	24.0	82.6	59.5	23.2	0.430	0.037

Appendix

Table 5: QuALITY audit with null-family envelope certification (

\tau=0.75

, Llama-3.1-8B-Instruct). Pass means the budget test holds for all null operators

\nu\in\mathcal{N}

(redact, delete, same-length mask, and no-evidence).

\mathbb{E}[p_{0}^{\mathrm{NE}}]

reports the no-evidence pseudo-prior.

Head	Mean patch restoration	Mean ablation $\Delta$ margin	$n$
L25H2 (misbinding/interference)	0.169	+0.508	159
L22H3 (counterbalances recency)	0.121	-0.321	159

Appendix

Table 6: Gemma-2-2b-it head-level patching/ablation on primacy_recency (

k=256

; bracket prompt style; clean filler random ; corrupt filler decoy_heavy with decoy_reps =12). Patch restoration is normalized margin recovery; ablation

\Delta

margin is

m_{\mathrm{ablated}}-m_{\mathrm{corrupt}}

(positive means ablating the head helps correct-vs-competitor margin; negative means ablation hurts).

Task	$k$	Qwen2.5-3B	Llama-3.2-3B-Inst.	Gemma-2-2b-it
competing_vars	256	0.203	0.993	1.000
	512	0.008	0.993	1.000
	1024	0.000	0.455	0.998
	2048	—	0.000	—
primacy_recency	256	0.060	0.868	0.963
	512	0.003	0.553	0.798

Appendix

Table 7: Baseline accuracy (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

single-token candidates). All models eventually fail on primacy_recency at sufficient distance.

Model	Task	$k$	Baseline	+Checkpoint	$\Delta$ Acc	ValueGap ${}_{\text{base}}$	ValueGap ${}_{\text{chk}}$
Qwen2.5-3B	competing_vars	256	0.203	0.973	+0.770	-1.02	+2.81
	competing_vars	512	0.008	0.993	+0.985	-5.75	+2.61
	competing_vars	1024	0.000	0.998	+0.998	-9.93	+2.88
	primacy_recency	256	0.060	0.760	+0.700	-2.88	+0.80
	primacy_recency	512	0.003	0.573	+0.570	-6.53	+0.26
	primacy_recency	1024	0.000	0.763	+0.763	-9.05	+1.01

Appendix

Table 8: Checkpointing results (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Checkpoints insert a compact serialization of the true binding every 128 tokens in the tail filler. Qwen recovers from 0% to 99.8% at

k=1024

. Exception: Llama at

k=2048

on primacy_recency shows checkpoint failure, as Stage 2A gating has collapsed and cannot be recovered by re-statement.

	Meta-Llama-3.1-8B-Instruct			Gemma-2-9b-it
$k$	Acc (base)	$P[Z]$ (base)	Acc (+Chk)	Acc (base)	$P[Z]$ (base)	Acc (+Chk)
0	0.435	0.185	0.510	0.330	0.490	1.000
64	0.575	0.130	0.590	0.415	0.300	1.000
256	0.550	0.155	0.550	0.640	0.150	0.995
1024	0.435	0.105	0.580	0.740	0.015	1.000
2048	0.430	0.055	0.600	0.820	0.000	1.000

Appendix

Table 9: Naturalistic notes_binding at 8B–9B scale with randomized choice ordering (

n=200

per cell).

P[Z]

is the abstention rate (Stage 2A failure). Llama degrades at long distance and is partially recovered by checkpointing; Gemma is near-perfect under checkpointing and exhibits a large reduction in abstention as

k

increases.

Qwen2.5-3B-Instruct				Llama-3.2-3B-Instruct				Gemma-2-2b-it
L	C	Rest	$\Delta$ M	L	C	Rest	$\Delta$ M	L	C	Rest	$\Delta$ M
28	attn	-0.03	-0.50	20	attn	-0.04	0.04	18	attn	0.02	-0.41
28	mlp	0.03	0.59	20	mlp	0.04	0.03	18	mlp	0.01	0.17
29	attn	0.06	1.02	21	attn	0.23	0.06	19	attn	0.10	0.12
29	mlp	-0.01	-0.24	21	mlp	-0.04	0.00	19	mlp	-0.03	-0.05
30	attn	-0.04	-0.71	22	attn	-0.03	0.09	20	attn	-0.01	0.02

Appendix

Table 10: Complete layer-component patching results (paired clean/corrupt prompts; greedy next-token decoding; bracket prompt style;

k=256

; clean filler random ; corrupt filler decoy_heavy with decoy_reps =12;

n\approx 160

Stage 2B pairs per model). Rest is mean normalized restoration

(m_{\mathrm{patched}}-m_{\mathrm{corrupt}})/(m_{\mathrm{clean}}-m_{\mathrm{corrupt}})

of the target-vs-competitor logit margin

m

;

\Delta M

is the mean change in margin

(m_{\mathrm{patched}}-m_{\mathrm{corrupt}})

. Qwen uses competing_vars ; Llama/Gemma use primacy_recency .

Task	Filler	$k$	Acc	Cand-Acc	Frac-2A	Frac-2B	GateGap	ValueGap
competing_vars	decoy_heavy	128	0.999	0.999	0.000	1.000	3.74	5.12
competing_vars	decoy_heavy	256	0.107	0.158	0.350	0.650	0.56	-1.81
competing_vars	decoy_heavy	512	0.003	0.001	0.023	0.977	1.99	-7.26
competing_vars	decoy_heavy	1024	0.000	0.000	0.033	0.968	1.79	-10.10
competing_vars	repeat	128	0.030	0.080	0.570	0.430	-0.07	-3.39
competing_vars	repeat	256	0.026	0.069	0.792	0.208	-0.80	-4.33

Appendix

Table 11: Complete Stage 2A/2B results for Qwen2.5-3B (

n=800

per row; greedy next-token decoding; bracket prompt style;

|\mathcal{C}|\approx 50

single-token candidates). Acc is

\mathbb{P}(\hat{Y}=V)

(top token), Cand-Acc is

\mathbb{P}(Y=V)

(best candidate). Frac-2A/Frac-2B are fractions of errors (conditional on

\hat{Y}\neq V

Task	$k$	Cand-Acc	$I_{\mathrm{used}}\geq$	Probe@35 (Stage 2B err)	$I_{\mathrm{probe,err}}^{\mathrm{proxy}}$
competing_vars	256	0.255	0.45	0.739	2.32
competing_vars	512	0.000	0.00	0.390	0.87
primacy_recency	256	0.074	0.04	0.354	0.75
primacy_recency	512	0.000	0.00	0.124	0.13

Appendix

Table 12: Converting accuracies into mutual-information quantities using the Fano expression

\Phi_{M}(\varepsilon)

from 1 with

M=50

(candidate set size

\approx 50

; nats). For the model’s candidate decision (Cand-Acc; unconditional, with

V

uniform by construction),

\Phi_{M}

is a valid lower bound on

I_{\mathrm{used}}

. For the final-layer probe accuracy restricted to Stage 2B error trials, we report the same quantity as an interpretable proxy (conditioning can change

H(V)

, so the lower-bound interpretation need not hold). For finite-sample cells with Cand-Acc

<1/M

we clamp the bound to 0.

$\tau$	Variant	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
0.65	baseline	24.7	83.5	60.4	23.1
0.65	checkpoint	27.4	81.0	59.0	22.0
0.65	rag	28.6	79.6	60.0	19.6
0.70	baseline	22.5	84.3	60.8	23.5
0.70	checkpoint	25.3	82.5	59.1	23.4
0.70	rag	26.2	81.8	59.9	22.0

Appendix

Table 13: Robustness of the QuALITY null-family envelope audit across reliability targets

\tau

. Passing is consistently associated with substantially higher answer accuracy.

Null operator	Variant	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
no-evidence	baseline	22.8	79.8	62.1	17.7
no-evidence	checkpoint	25.1	81.2	59.6	21.7
no-evidence	rag	26.5	79.0	60.8	18.2
redact-span	baseline	27.4	81.2	60.4	20.7
redact-span	checkpoint	30.8	78.8	58.9	19.9
redact-span	rag	31.2	80.5	58.8	21.7

Appendix

Table 14: Sensitivity of QuALITY audit to the concrete pseudo-prior operator at

\tau=0.75

. “Envelope” requires the budget test to pass for all null operators; it provides a conservative, assumption-light certificate while retaining strong predictive lift.

$k$	$n$	Pass(%)	$\mathbb{E}[p_{1}]$	$\mathbb{E}[p_{0}^{\min},p_{0}^{\max}]$
512	200	66.0	0.756	[0.179, 0.272]
1024	200	0.0	0.728	[0.097, 0.184]
2048	200	100.0	0.841	[0.181, 0.361]

Appendix

Table 15: Null-family statistics for a synthetic binding audit (

\tau=0.75

). Although

p_{0}

varies across null operators, the pass/fail decision is invariant across the null family in this setting (envelope equals any single-null decision).

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
May 1, 2026Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1LLM Evaluation Instruction Following
Indian Institute of Technology Gandhinagar · Soket AI
Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors
Jul 1, 2026Yangfan Hu, Xuhan Tong, Haoyue Bai +5Knowledge Conflicts in Language Models LLM Prompting
University of Wisconsin–Madison
When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models
Oct 19, 2026Guanqun Jiang, Pengfei Cao, Kang Liu +1Hallucination in Language Models Knowledge Conflicts in Language Models
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Xi’an Jiaotong-Liverpool University, Suzhou, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

May 1, 2026J/K move · Enter open · S save

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh

Indian Institute of Technology Gandhinagar · Soket AI

LLM Evaluation Instruction Following LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.

Abstract

Hallucination in Language Models

Explore similar work

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors

When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models