Large language models can infer and follow specific rules through in-context learning (ICL), yet may fail to remain faithful to provided evidence, giving rise to out-of-context (OOC) hallucination. Existing work has largely investigated the underlying mechanisms of ICL and OOC hallucination along separate lines. However, this separation obscures the mechanistic basis for these seemingly opposing outcomes. In this work, we study this contrast through the unified view of functional roles in dynamic context. The key observation is that role information can be internally represented without reliably constraining output. We derive a first-order decomposition to characterize the role effect, showing that its substantial influence depends on two necessary conditions, namely sufficient contribution magnitude and coherent signed aggregation. Causal analysis further shows that role span contributions affect key token choices involving response structure, content selection, and factual values. We further show that role-consistent choices depend on sufficient role span support at these key decisions, while failing choices can be reversed when this support overcomes the deficit. These findings support a common mechanistic view of ICL and OOC hallucination: each functional role defines the pattern of output decisions over which role span contributions must remain effective, and differences in this organization account for the seemingly opposing outcomes.
Figures & tables
Figure 1: Examples of matched contexts . Construction details are provided in Appendix C.1 .
Figure 2: Structured residual responses . Left panels show rule span responses, and right panels show evidence span responses. Each violin shows the fraction of Δh energy captured by a 10-dimensional PCA subspace; points are colored by the percentile of ∥Δh∥ . The dashed gray line denotes the expected random subspace baseline k/H≈0.2% , with k=10 PCA directions and residual dimension.
Figure 3
Figure 4: Response preference and contribution organization . (a) Fraction of positive candidate margins. For Qwen2.5-7B, (b) layerwise profiles of ∣ΔBℓ,t∣ and ∣∇ψt∣ , (c) signed layerwise contributions and sign coherence, and (d) preference margin Mf under baseline, role span pathway ablation, and random ablation. More results are in Appendix D.1 .
(a) First-divergence localization
(b) Retention of role-specified output content (%)
Rule span
Evidence span
Rule span
Evidence span
Grounded
Hallucinated
Format
Entity
Grounded
Hallucinated
Model
Attr.
Dec.
Attr.
Dec.
Attr.
Dec.
Cf
Rem.
Cf
Rem.
Cf
Rem.
Cf
Rem.
Qwen2.5-7B
87.5
85.0
75.0
92.5
88.0
88.2
100.0
4.8
57.1
18.1
100.0
10.2
62.3
11.3
Mistral-7B
82.5
85.7
87.5
77.5
89.8
90.1
100.0
2.4
58.1
33.3
100.0
16.7
61.5
7.7
Llama-3.1-8B
80.0
90.0
85.0
95.0
92.4
91.9
100.0
4.8
53.3
18.1
100.0
10.0
61.5
13.5
Table 2: Role span effects on output decisions. (a) Role span influence at the first divergence and the output decision. (b) Retention rate of the content specified by Cf under the original ( Cf ) and pathway-removed (Rem.) conditions. Additional results are provided in Appendix D.6 .
(a) Target token margin
(b) Token and task outcomes at α=1
Targeted attenuation
Non-role comparison
Targeted attenuation
Non-role comparison
Model
α=0
α=0.25
α=0.5
α=0.75
α=1
α=0
α=1
Flip (%)
Task (%)
Flip (%)
Task (%)
Rule span
Qwen2.5-7B
4.62
5.00
-0.38
-12.89
-16.25
4.62
5.00
84.0
22.3
17.0
100.0
Mistral-7B
6.69
6.06
2.09
-4.16
-5.03
6.69
5.25
77.5
32.4
32.4
94.1
Llama-3.1-8B
4.69
3.25
2.06
-1.94
-4.06
4.69
2.38
76.5
28.6
25.5
86.7
Table 3: Sustained attenuation of role span contributions . The contribution is scaled by (1−α) , with α=1 denoting complete attenuation. (a) Median target token margin under targeted attenuation and non-role comparison. (b) Token flips and task performance under full attenuation.
(a) Restoration at β=1
(b) Restoration across weights
Baseline
Margin
Selection
Task
Model
margin
change
(%)
(%)
Metric
β=0
β=0.5
β=1
β=2
β=4
β=8
β=16
Rule span
Qwen2.5-7B
-14.6
-0.61
0.0
0.0
Selection
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Mistral-7B
-19.0
+0.06
0.0
4.8
Margin
-14.6
-14.6
-14.5
-14.1
-13.0
-12.8
-11.7
Llama-3.1-8B
-11.3
+0.00
0.0
0.0
Task
0.0
0.0
1.3
2.5
6.3
3.8
6.3
Table 4: Restoration at failing decisions . We restore the contribution vector at the decision, scaled by a weight β . (a) Restoring the contribution at β=1 . (b) Restoration across multiple weights.
Figure 5: Matched preference construction . Each context pair differs only in the role-bearing span, while each candidate pair differs only in the choice specified by that role.
Figure 6: Organization of contributions for Mistral-7B . (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of ∣ΔBℓ,t∣ and ∣∇ψt∣ ; and (c) preference margin Mf under the baseline, role span pathway ablation, and random ablation.
Figure 7: Organization of contributions for Llama-3.1-8B . (a) Signed layerwise contributions and sign coherence; (b) layerwise profiles of ∣ΔBℓ,t∣ and ∣∇ψt∣ ; and (c) preference margin Mf under the baseline, role span pathway ablation, and random ablation.
Rule span
Evidence span
Model
Corr.
Sign (%)
Corr.
Sign (%)
Qwen2.5-7B
0.93
77.1
0.86
95.0
Mistral-7B
0.94
87.6
0.88
97.0
Llama-3.1-8B
0.92
91.4
0.85
94.0
Appendix
Table 5: Correspondence between first-order contributions and exact score shifts .
Rule
Evidence
Model
Direct ΔB
Reweighting ΔW
Non-role
Direct ΔB
Reweighting ΔW
Non-role
Qwen2.5-7B
73.5
0.0
0.1
85.9
1.9
1.4
Mistral-7B
88.3
0.6
−0.9
92.0
1.9
−0.7
Llama-3.1-8B
87.7
0.8
−0.2
93.9
0.4
0.8
Appendix
Table 6: Direct recovery of role effects. Median normalized recovery (%) after substituting the direct role term ( ΔB ), attention reweighting ( ΔW ), or matched non-role positions.
(a) Carry drop by depth
(b) Update families
Model
ℓ/L=0.25
0.5
0.75
Attention
MLP
Rule
Qwen2.5-7B
−0.01 [ −0.13 , 0.12]
−0.22 [ −0.64 , 0.14]
11.25 [7.18, 16.11]
30.79 [24.51, 37.68]
26.14 [21.55, 31.06]
Mistral-7B
0.05 [ −0.01 , 0.11]
0.77 [0.15, 1.42]
12.56 [9.76, 15.32]
15.64 [12.20, 19.21]
13.40 [10.84, 15.97]
Llama-3.1-8B
−0.04 [ −0.10 , 0.03]
3.25 [2.10, 4.55]
9.11 [8.20, 10.00]
13.74 [11.59, 15.96]
8.77 [7.71, 9.68]
Evidence
Appendix
Table 7: Accumulated carry and update-family effects at role-sensitive decisions. (a) The carried state entering the layer at three fractional depths. (b) Each update family at all layers.
(a) Rule induction
(b) Contribution
(c) Decision effect
Model
Task acc. (%) orig./perm.
Mapping follow (%)
No-demo/ broken (%)
Positive margin (%)
Recovery (%) demo/control
Margin drop [95% CI]
Accuracy drop [95% CI] (pp)
SST-2
Qwen2.5-7B
77.1 / 86.7
84.8
46.7 / 54.3
81.9
99.4 / 0.1
2.27 [1.7, 2.8]
25.7 [15.2, 35.2]
Mistral-7B
82.9 / 82.9
82.9
49.5 / 48.6
81.9
99.8 / 0.0
1.24 [0.8, 1.7]
20.0 [7.6, 31.4]
Llama-3.1-8B
73.3 / 77.1
79.0
41.0 / 41.9
77.1
96.2 / −1.0
0.63 [0.3, 0.9]
15.2 [6.7, 23.8]
TREC
Appendix
Table 8: Role effects for rules induced from demonstrations. The three column groups report rule induction, contribution alignment, and the effect of targeted attenuation.
(a) Recovery (%)
(b) Margin drop vs. control
Model
All
Labels
Inputs
All
Labels
Inputs
SST-2
Qwen2.5-7B
99.4
98.4
4.4
2.27
2.09
0.29
Mistral-7B
99.8
98.7
2.4
1.24
1.37
0.38
Llama-3.1-8B
96.2
95.1
3.2
0.63
0.76
0.23
TREC
Appendix
Table 9: Position-wise organization of induced role effects. Recovery and margin drop are compared across all demonstration tokens, label tokens, and input tokens.
(a) Selection (%)
(b) Matched deficit (%)
(c) Prediction at β=1 (%)
Direction
β=4
β=8
β=16
Group
Recovery
Group
Recovery
Demonstration
12.9
18.8
22.5
Single-label rule
6.5
β^t≤1
59.5
Random
5.5
0.0
0.0
Task-defined evidence
47.6
β^t>1
0.4
Real-hallucinated
58.8
Appendix
Table 10: Restoration of failing single-label rule decisions in few-shot ICL. (a) Target selection under amplification. (b) Recovery at matched deficits. (c) Recovery by the predicted threshold.
(a) Grounded preference and pathway recovery
Grounded preference (%)
Matched effect
Pathway recovery
Model
Original
Counterfactual
No evidence
Median Me
Recovery difference (nats)
Qwen2.5-7B
82.7
46.9
63.3
36.47
5.08 [4.0, 6.2]
Mistral-7B
89.8
43.9
69.4
35.11
5.07 [4.2, 6.0]
Llama-3.1-8B
98.0
50.0
76.5
26.08
4.30 [3.6, 5.0]
Appendix
Table 11: Evidence contributions under distributed multi-hop grounding. (a) Grounded preference and pathway recovery. (b) Dependence on each supporting passage and its direct effect at the answer decision.
(a) Context-conditioned preference and pathway recovery
Primary preference (%)
Matched effect
Pathway recovery
Model
Original
Swapped
Closed book
Median Me
Positive Me (%)
Recovery difference (nats)
Qwen2.5-7B
28.4
25.3
26.3
1.06
53.7 [43.2, 64.2]
4.99 [4.0, 6.0]
Mistral-7B
31.6
13.7
22.1
1.90
65.3 [55.8, 74.7]
4.63 [3.9, 5.3]
Llama-3.1-8B
34.7
5.3
20.0
4.03
81.1 [72.6, 88.4]
4.70 [4.1, 5.3]
Appendix
Table 12: Evidence contributions under competing factual sources. (a) Context-conditioned preference and pathway recovery. (b) Source-specific effects under targeted attenuation.
(a) Role span influence (%)
(b) Position of generation changes
Model
Decision percentile
Matched control
Median offset
Before dec. (%)
At dec. (%)
Rule
Qwen2.5-7B
85.0
67.4
+0
46.9
53.1
Mistral-7B
85.7
63.6
−1
55.9
44.1
Llama-3.1-8B
90.0
63.0
−5
55.3
44.7
Evidence
Appendix
Table 13: Role span behavior at semantic decisions. (a) Role span influence at semantic decisions: the median within-response percentile of ∣δ^t∣ at the decision position and at matched positions in other responses. (b) Position of the first generation change relative to the semantic decision.
(a) Factual decision (%)
(b) First divergence
Direct evidence removed
Matched non-evidence
Indirect share median (%)
Margin median
All models
31.6
1.7
91.8
0.6
Appendix
Table 14: Direct and indirect evidence effects on a fixed prefix. (a) Changes in the selected fact at the factual decision. (b) Pathway share and margin at the first divergence.
Rule span
Evidence span
β
Predicted (%)
Observed (%)
Predicted (%)
Observed (%)
0.5
0.0
0.0
23.3
25.2
1
0.0
1.3
26.2
32.0
2
0.0
2.5
38.8
47.6
4
0.0
6.3
52.4
53.4
8
0.0
3.8
64.1
55.3
Appendix
Table 15: Task-level recovery prediction under role contribution amplification. Predicted versus observed task success across amplification strengths.
In-Context Learning (ICL) has become a cornerstone of modern LLM deployment. However, existing ICL post-training methods have a critical blind spot: they excel at extracting patterns from demonstrations while often neglecting context authority, the ability to determine whether contextual information should govern the final answer. To benchmark this capability, we introduce FakeContextBench, which contains pseudoscientific claims across seven domains. Our evaluation of commercial and open-source models shows that large-scale pre-training alone is insufficient for reliable context-authority discrimination. Moreover, prevalent ICL fine-tuning methods can increase susceptibility to misleading context, reducing reality accuracy by up to 14.95 percentage points relative to the base model. To address this trade-off, we propose Jurisdiction In-Context Learning (J-ICL), a post-training framework that incorporates context validation into the training objective. Across four model backbones, J-ICL improves ICLEval by an average of 5.84 percentage points and reality accuracy by 9.20 points over the corresponding base models. It also raises the Reality Rate by an average of 18.09 points relative to MetaICL and Symbol Tuning. These results demonstrate that ICL capability and resistance to deceptive context can be improved together. The benchmark is available at https://github.com/peilin717/FakeContext-Bench.
Pei-lin Li, Qingle Liu, Junyang Feng +4
Department of Computer Science and Technology, Tsinghua University · School of Integrated Circuits, Huazhong University of Science and Technology
Reasoning hallucinations in large language models (LLMs) often appear as fluent yet unsupported conclusions that violate either the given context or underlying factual knowledge. Although such failures are widely observed, the mechanisms by which decoder-only Transformers produce them remain poorly understood. We model next-token prediction as a graph search process over an underlying graph, where entities correspond to nodes and learned transitions form edges. From this perspective, contextual reasoning is a constrained search over a sampled subgraph (intrinsic reasoning), while context-free queries rely on memorized structures in the underlying graph (extrinsic reasoning). We show that reasoning hallucinations arise from two fundamental mechanisms: Path Reuse, where memorized knowledge overrides contextual constraints during early training, and Path Compression, where frequently traversed multi-step paths collapse into shortcut edges in later training. Together, these mechanisms provide a unified explanation for reasoning hallucinations in LLMs and connected to well-known behaviors observed in downstream applications.
Xinnan Dai, Kai Yang, Cheng Luo +3
Michigan State University · University of Michigan · Independent Researcher
Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challenge for reliable deployment. In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis. We propose that hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream. Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation. Based on this formulation, we introduce Dynamic Contextual Orthogonalization (DCO), an inference-time intervention method. DCO utilizes the input residual stream as a dynamic context anchor to perform orthogonal decomposition on attention head outputs. To distinguish between context-aligned semantic updates and divergent noise, DCO employs a layer-wise Z-score suppression mechanism that selectively attenuates outlier orthogonal components based on statistical distributions. Evaluations on Llama-3-8B and 70B across benchmarks such as XSum, NQ-Swap, and IFEval demonstrate that DCO achieves superior contextual faithfulness compared to state-of-the-art intervention baselines. Furthermore, DCO maintains high performance on knowledge-intensive tasks like TriviaQA and TruthfulQA, effectively mitigating the trade-off between hallucination suppression and parametric knowledge retention often observed in existing methods. Our findings validate the geometric interpretation of hallucinations and establish DCO as a computationally efficient approach for enforcing manifold alignment.Our code is available at https://github.com/Harry-Miral/DCO
Mingkuan Zhao, Wentao Hu, Tianchen Huang +6
Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom AI Technology (Beijing) Co., Ltd. · University of Science and Technology of China +2