Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations

Ahmed Karim; Fatima Sheaib; Zein Khamis; Maggie Chlon; Jad Awada; Leon Chlon

stat.MLFeb 22, 2026

Attention Deficits in Language Models: Causal Explanations for Procedural Hallucinations

Authors: Ahmed Karim, Fatima Sheaib, Zein Khamis, Maggie Chlon, Jad Awada, Leon Chlon

Organizations: Hassana Labs · University of Oxford

Abstract

Large language models can follow complex procedures yet fail at a seemingly trivial final step: reporting a value they themselves computed moments earlier. We study this phenomenon as procedural hallucination: failure to execute a verifiable, prompt-grounded specification even when the correct value is present in context. In long-context binding tasks with a known single-token candidate set, we find that many errors are readout-stage routing failures. Specifically, failures decompose into Stage~~2A (gating) errors, where the model does not enter answer mode, and Stage~~2B (binding) errors, where it enters answer mode but selects the wrong candidate (often due to recency bias). In the hard regime, Stage~~2B accounts for most errors across model families in our tasks (Table~~1). On Stage~~2B error trials, a linear probe on the final-layer residual stream recovers the correct value far above chance (e.g., 74% vs.\ 2% on Qwen2.5-3B; Table~~2), indicating that the answer is encoded but not used. We formalize ``present but not used'' via available vs.\ used mutual information and pseudo-prior interventions, yielding output-computable diagnostics and information-budget certificates. Finally, an oracle checkpointing intervention that restates the true binding near the query can nearly eliminate Stage~~2B failures at long distance (e.g., Qwen2.5-3B $0/400 \rightarrow 399/400$ at $k = 1024$ ; Table~~8).

Figures & tables

Figure 1: Framework overview: Stage 2A gating and Stage 2B binding failures correspond to low routing efficiency

I_{\mathrm{used}}/I_{\mathrm{avail}}

, diagnosed via pseudo-prior interventions.

competing_vars	primacy_recency
KEY1 = [apple]	KEY = [alpha]
[k filler tokens]	[k filler tokens]
KEY2 = [banana]	KEY = [beta]
[k filler tokens]	[k filler tokens]
What is KEY1? KEY1 = [	KEY = [gamma]
	[k filler tokens]

Table 2

Figure 2: Spotlight summary of core empirical claims. (A) Stage decomposition for representative hard-regime settings: most errors are Stage 2B (binding) rather than Stage 2A (gating), i.e., the model enters answer mode but selects the wrong candidate. (B) Checkpointing (restating the true binding every 128 tokens near the query) substantially recovers long-distance binding for Qwen2.5-3B; on competing_vars at

k=1024

, it converts 0/400

\to

399/400 correct. Error bars are 95% Wilson binomial confidence intervals over

n=400

trials per cell.

Model	Task	$k$	Acc	Cand Acc	Frac 2A	Frac 2B	GateGap	ValueGap
Qwen2.5-3B	competing	256	0.107	0.158	0.350	0.650	0.56	-1.81
Qwen2.5-3B	primacy	256	0.043	0.045	0.185	0.815	0.96	-3.40
Qwen2.5-3B-Inst.	competing	256	0.276	0.360	0.402	0.598	0.61	-0.77
Qwen2.5-3B-Inst.	primacy	256	0.061	0.059	0.072	0.928	2.10	-3.96
Gemma-2-2B	competing	256	0.995	0.995	0.000	1.000	3.03	2.28
Gemma-2-2B	primacy	256	0.801	0.804	0.013	0.987	1.65	0.47

Table 1: Stage decomposition (

n=800

, greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

single-token candidates). Acc is

\mathbb{P}(\hat{Y}=V)

(top token), Cand-Acc is

\mathbb{P}(Y=V)

(best candidate). Frac-2A/Frac-2B are fractions of errors (conditional on

\hat{Y}\neq V

), decomposing errors into gating (non-candidate output) vs. binding (wrong candidate) failures. For Llama we report

k=2048

to reach comparable difficulty.

Task	$k$	Acc	Cand Acc	Frac 2B	Probe@0	Probe@18	Probe@35
competing	256	0.200	0.255	0.641	0.019	0.083	0.739
competing	512	0.000	0.000	0.968	0.008	0.029	0.390
primacy	256	0.069	0.074	0.754	0.020	0.040	0.354
primacy	512	0.000	0.000	0.994	0.020	0.020	0.124

Table 2: Probing on Stage 2B errors (Qwen2.5-3B,

n=800

; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Frac-2B is a fraction of errors (conditional on

\hat{Y}\neq V

) attributable to Stage 2B binding failures. Layers 0/18/35 are embedding, mid, and final residual streams. Chance is

\approx

2%. At

k=256

, the final-layer probe recovers the correct answer on 74% of error trials.

Model	Task	$k$	Baseline	+Checkpoint
Qwen2.5-3B	competing	1024	0.000	0.998
Qwen2.5-3B	primacy	1024	0.000	0.763
Llama-3.2-3B-Inst.	competing	2048	0.000	0.853
Llama-3.2-3B-Inst.	primacy	1024	0.003	0.915
Gemma-2-2B-it	primacy	1024	0.440	0.968

Table 3: Checkpointing summary (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Oracle checkpointing inserts a compact serialization of the true binding every 128 tokens in the tail filler, and can convert near-zero accuracy into near-perfect accuracy at long distance; see Appendix Table 8 for full results. Tasks are competing_vars ( competing ) and primacy_recency ( primacy ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$k$	Acc(%)	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
Mistral-7B (baseline)	1024	98.3	54.8	100.0	97.0	3.0
Mistral-7B (baseline)	2048	87.0	68.6	99.5	63.7	35.8

Appendix

Table 4: Trace-budget audit (

\tau=0.75

). At large distance, errors concentrate in flagged traces.

Variant	$n$	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)	$\mathbb{E}[p_{1}]$	$\mathbb{E}[p_{0}^{\mathrm{NE}}]$
baseline	369	21.1	83.3	61.5	21.8	0.398	0.036
rag	378	25.1	81.1	60.4	20.6	0.428	0.039
checkpoint	383	24.0	82.6	59.5	23.2	0.430	0.037

Appendix

Table 5: QuALITY audit with null-family envelope certification (

\tau=0.75

, Llama-3.1-8B-Instruct). Pass means the budget test holds for all null operators

\nu\in\mathcal{N}

(redact, delete, same-length mask, and no-evidence).

\mathbb{E}[p_{0}^{\mathrm{NE}}]

reports the no-evidence pseudo-prior.

Head	Mean patch restoration	Mean ablation $\Delta$ margin	$n$
L25H2 (misbinding/interference)	0.169	+0.508	159
L22H3 (counterbalances recency)	0.121	-0.321	159

Appendix

Table 6: Gemma-2-2b-it head-level patching/ablation on primacy_recency (

k=256

; bracket prompt style; clean filler random ; corrupt filler decoy_heavy with decoy_reps =12). Patch restoration is normalized margin recovery; ablation

\Delta

margin is

m_{\mathrm{ablated}}-m_{\mathrm{corrupt}}

(positive means ablating the head helps correct-vs-competitor margin; negative means ablation hurts).

Task	$k$	Qwen2.5-3B	Llama-3.2-3B-Inst.	Gemma-2-2b-it
competing_vars	256	0.203	0.993	1.000
	512	0.008	0.993	1.000
	1024	0.000	0.455	0.998
	2048	—	0.000	—
primacy_recency	256	0.060	0.868	0.963
	512	0.003	0.553	0.798

Appendix

Table 7: Baseline accuracy (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

single-token candidates). All models eventually fail on primacy_recency at sufficient distance.

Model	Task	$k$	Baseline	+Checkpoint	$\Delta$ Acc	ValueGap ${}_{\text{base}}$	ValueGap ${}_{\text{chk}}$
Qwen2.5-3B	competing_vars	256	0.203	0.973	+0.770	-1.02	+2.81
	competing_vars	512	0.008	0.993	+0.985	-5.75	+2.61
	competing_vars	1024	0.000	0.998	+0.998	-9.93	+2.88
	primacy_recency	256	0.060	0.760	+0.700	-2.88	+0.80
	primacy_recency	512	0.003	0.573	+0.570	-6.53	+0.26
	primacy_recency	1024	0.000	0.763	+0.763	-9.05	+1.01

Appendix

Table 8: Checkpointing results (

n=400

; greedy next-token decoding; bracket prompt style; decoy_heavy filler;

|\mathcal{C}|\approx 50

). Checkpoints insert a compact serialization of the true binding every 128 tokens in the tail filler. Qwen recovers from 0% to 99.8% at

k=1024

. Exception: Llama at

k=2048

on primacy_recency shows checkpoint failure, as Stage 2A gating has collapsed and cannot be recovered by re-statement.

	Meta-Llama-3.1-8B-Instruct			Gemma-2-9b-it
$k$	Acc (base)	$P[Z]$ (base)	Acc (+Chk)	Acc (base)	$P[Z]$ (base)	Acc (+Chk)
0	0.435	0.185	0.510	0.330	0.490	1.000
64	0.575	0.130	0.590	0.415	0.300	1.000
256	0.550	0.155	0.550	0.640	0.150	0.995
1024	0.435	0.105	0.580	0.740	0.015	1.000
2048	0.430	0.055	0.600	0.820	0.000	1.000

Appendix

Table 9: Naturalistic notes_binding at 8B–9B scale with randomized choice ordering (

n=200

per cell).

P[Z]

is the abstention rate (Stage 2A failure). Llama degrades at long distance and is partially recovered by checkpointing; Gemma is near-perfect under checkpointing and exhibits a large reduction in abstention as

k

increases.

Qwen2.5-3B-Instruct				Llama-3.2-3B-Instruct				Gemma-2-2b-it
L	C	Rest	$\Delta$ M	L	C	Rest	$\Delta$ M	L	C	Rest	$\Delta$ M
28	attn	-0.03	-0.50	20	attn	-0.04	0.04	18	attn	0.02	-0.41
28	mlp	0.03	0.59	20	mlp	0.04	0.03	18	mlp	0.01	0.17
29	attn	0.06	1.02	21	attn	0.23	0.06	19	attn	0.10	0.12
29	mlp	-0.01	-0.24	21	mlp	-0.04	0.00	19	mlp	-0.03	-0.05
30	attn	-0.04	-0.71	22	attn	-0.03	0.09	20	attn	-0.01	0.02

Appendix

Table 10: Complete layer-component patching results (paired clean/corrupt prompts; greedy next-token decoding; bracket prompt style;

k=256

; clean filler random ; corrupt filler decoy_heavy with decoy_reps =12;

n\approx 160

Stage 2B pairs per model). Rest is mean normalized restoration

(m_{\mathrm{patched}}-m_{\mathrm{corrupt}})/(m_{\mathrm{clean}}-m_{\mathrm{corrupt}})

of the target-vs-competitor logit margin

m

;

\Delta M

is the mean change in margin

(m_{\mathrm{patched}}-m_{\mathrm{corrupt}})

. Qwen uses competing_vars ; Llama/Gemma use primacy_recency .

Task	Filler	$k$	Acc	Cand-Acc	Frac-2A	Frac-2B	GateGap	ValueGap
competing_vars	decoy_heavy	128	0.999	0.999	0.000	1.000	3.74	5.12
competing_vars	decoy_heavy	256	0.107	0.158	0.350	0.650	0.56	-1.81
competing_vars	decoy_heavy	512	0.003	0.001	0.023	0.977	1.99	-7.26
competing_vars	decoy_heavy	1024	0.000	0.000	0.033	0.968	1.79	-10.10
competing_vars	repeat	128	0.030	0.080	0.570	0.430	-0.07	-3.39
competing_vars	repeat	256	0.026	0.069	0.792	0.208	-0.80	-4.33

Appendix

Table 11: Complete Stage 2A/2B results for Qwen2.5-3B (

n=800

per row; greedy next-token decoding; bracket prompt style;

|\mathcal{C}|\approx 50

single-token candidates). Acc is

\mathbb{P}(\hat{Y}=V)

(top token), Cand-Acc is

\mathbb{P}(Y=V)

(best candidate). Frac-2A/Frac-2B are fractions of errors (conditional on

\hat{Y}\neq V

Task	$k$	Cand-Acc	$I_{\mathrm{used}}\geq$	Probe@35 (Stage 2B err)	$I_{\mathrm{probe,err}}^{\mathrm{proxy}}$
competing_vars	256	0.255	0.45	0.739	2.32
competing_vars	512	0.000	0.00	0.390	0.87
primacy_recency	256	0.074	0.04	0.354	0.75
primacy_recency	512	0.000	0.00	0.124	0.13

Appendix

Table 12: Converting accuracies into mutual-information quantities using the Fano expression

\Phi_{M}(\varepsilon)

from 1 with

M=50

(candidate set size

\approx 50

; nats). For the model’s candidate decision (Cand-Acc; unconditional, with

V

uniform by construction),

\Phi_{M}

is a valid lower bound on

I_{\mathrm{used}}

. For the final-layer probe accuracy restricted to Stage 2B error trials, we report the same quantity as an interpretable proxy (conditioning can change

H(V)

, so the lower-bound interpretation need not hold). For finite-sample cells with Cand-Acc

<1/M

we clamp the bound to 0.

$\tau$	Variant	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
0.65	baseline	24.7	83.5	60.4	23.1
0.65	checkpoint	27.4	81.0	59.0	22.0
0.65	rag	28.6	79.6	60.0	19.6
0.70	baseline	22.5	84.3	60.8	23.5
0.70	checkpoint	25.3	82.5	59.1	23.4
0.70	rag	26.2	81.8	59.9	22.0

Appendix

Table 13: Robustness of the QuALITY null-family envelope audit across reliability targets

\tau

. Passing is consistently associated with substantially higher answer accuracy.

Null operator	Variant	Pass(%)	Acc(pass)(%)	Acc(flag)(%)	Lift(pp)
no-evidence	baseline	22.8	79.8	62.1	17.7
no-evidence	checkpoint	25.1	81.2	59.6	21.7
no-evidence	rag	26.5	79.0	60.8	18.2
redact-span	baseline	27.4	81.2	60.4	20.7
redact-span	checkpoint	30.8	78.8	58.9	19.9
redact-span	rag	31.2	80.5	58.8	21.7

Appendix

Table 14: Sensitivity of QuALITY audit to the concrete pseudo-prior operator at

\tau=0.75

. “Envelope” requires the budget test to pass for all null operators; it provides a conservative, assumption-light certificate while retaining strong predictive lift.

$k$	$n$	Pass(%)	$\mathbb{E}[p_{1}]$	$\mathbb{E}[p_{0}^{\min},p_{0}^{\max}]$
512	200	66.0	0.756	[0.179, 0.272]
1024	200	0.0	0.728	[0.097, 0.184]
2048	200	100.0	0.841	[0.181, 0.361]

Appendix

Table 15: Null-family statistics for a synthetic binding audit (

\tau=0.75

). Although

p_{0}

varies across null operators, the pass/fail decision is invariant across the null family in this setting (envelope equals any single-null decision).

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

May 1, 2026

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.

Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1

Indian Institute of Technology Gandhinagar · Soket AI

LLM Evaluation Instruction Following LLM Reasoning

Ask PDF

Jul 1, 2026

Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors

Large language models often produce hallucinated answers that violate prompt-level constraints. A key diagnostic question is whether these failures reflect missing knowledge, or whether the model has the relevant information but follows the wrong inference path. We study this phenomenon as inference misalignment: a mismatch between the answer supported by the prompt and the answer favored by statistically salient latent associations. We formalize this view with a latent key-task model, in which pretraining-frequency imbalance can cause a shortcut path to dominate the constraint-sensitive path and induce positive inference loss. The framework predicts two failure modes: task-retrieval bias in entity disambiguation and key-selection bias in action choice. We introduce TrapQA, a controlled diagnostic testbed with two components. ScientistQA tests disambiguation among similar scientists with supplementary factual probes, while Real-Life Constrained QA tests everyday constraint following under salient shortcuts. Our results show that hallucination can arise from biased latent inference rather than absent knowledge alone.

Yangfan Hu, Xuhan Tong, Haoyue Bai +5

University of Wisconsin–Madison

Knowledge Conflicts in Language Models LLM Prompting Hallucination in Language Models Language Model Probing

Ask PDF

Oct 19, 2026

When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models

Language models increasingly use skills that provide reusable instructions and procedures for task execution. After a relevant skill has been selected, the model may need to apply its governing procedure together with retrieved task context. When the retrieved context is useful but contains cues that violate the skill, the model must retain compatible information while rejecting the conflicting cues. We study failures in which the model adopts such cues and produces a retrieval-supported but skill-invalid answer, which we call procedural hallucination. We introduce SourceClash, a matched benchmark with 1,271 bundles and 6,355 prompts across ten procedural domains. Each bundle fixes the task, skill specification, and answer space while varying the relation between the skill and retrieval. Across nine models, valid-answer rates average 25.7% under conflicting retrieval, compared with 78.2% and 72.8% under two supportive retrieval settings. Mechanistic analysis of open-weight models shows that the answer-preference difference typically reaches its maximum shortly after generation begins. Residual replacement moves the model toward valid answers, while attention decomposition attributes most of the repair to value vectors. Intervention at the identified value pathway also improves final answer validity. These results characterize how partly conflicting retrieval changes skill-guided generation.

Guanqun Jiang, Pengfei Cao, Kang Liu +1

The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Xi’an Jiaotong-Liverpool University, Suzhou, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Hallucination in Language Models Knowledge Conflicts in Language Models

Ask PDF

Abstract

Hallucination in Language Models

Explore similar work

When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors

When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models