Large language models (LLMs) hallucinate: they produce fluent outputs that are factually incorrect. We present a geometric dynamical systems framework in which hallucinations arise from task-dependent basin structure in latent space. Using autoregressive hidden-state trajectories across multiple open-source models and benchmarks, we find that separability is strongly task-dependent rather than universal: factoid settings can show clearer basin separation, whereas summarization and misconception-heavy settings are typically less stable and often overlap. We formalize this behavior with task-complexity and multi-basin theorems, characterize basin emergence in L-layer transformers, and show that geometry-aware steering can reduce hallucination probability without retraining.
Figures & tables
Symbol
Definition
h(ℓ)
Hidden state at layer ℓ∈{0,…,L}
d
Dimension of hidden states ( h(ℓ)∈Rd )
μ(ℓ)
Reference/centroid state at layer ℓ (basin center)
Table 2: Variance Analysis by Task Type. Factoid tasks show up to 4 × variance expansion (factual states occupy larger volume), while summarization maintains parity (high-dimensional manifolds). Basin separation d measured as ∥μfact−μhall∥2 . ρvar=Var[factual]/Var[hallucinated] .
Figure 1: Task-Dependent Basin Geometry. Llama-3.2-3b’s performance on various tasks and 3D PCA projected outputs. (a) shows performance on MuSiQue, (b) shows performance on HaluEvalQA, (c) shows performance on HaluEvalSummarization, (d) shows performance on TruthfulQA.
Model
Data
Lay
Centroid (95% CI)
Maha (95% CI)
N
B?
gemma-2-2b
FEVER
L14
0.515(0.489,0.548)
0.514(0.487,0.535)
9999
×
gemma-2-2b
HaluEval_qa
L14
0.727(0.703,0.744)
0.725(0.710,0.745)
20000
✓
gemma-2-2b
HaluEval_summ
L20
0.508(0.491,0.519)
0.479(0.457,0.502)
20000
×
gemma-2-2b
MuSiQue
L26
0.912(0.894,0.932)
0.926(0.909,0.944)
4834
✓
gemma-2-2b
TruthfulQA
L14
0.607(0.547,0.662)
0.597(0.535,0.651)
1580
×
llama-3.2-1b
FEVER
L8
0.670(0.641,0.700)
0.680(0.659,0.702)
9999
×
Table 3: Centroid and Mahalanobis (Maha) AUROC reported with 95% CI. Note: Lay indicates Layer with highest AUROC, N indicates number of data samples, and B? indicates whether a basin exists or not. For larger models, due to computational constraints, this was limited to N=1000 .
Figure 2: Causal Intervention: Factual → Basin. (Left) Dose-response curve fold increase in hallucination probability as factual hidden states are in-model steered toward the hallucination centroid (interpolation strength α on the horizontal axis). Right: bar plot comparing the maximum fold increase produced by steering along the basin direction versus two controls (random direction and an orthogonal direction). See Appendix D.3 and F.1 .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
σhall2←∣Hhall∣1∑h∈Hhall∥h−μref∥2
Appendix
Algorithm 1 Multi-Basin Detection for Misconceptions
Figure 3: Multi-basin Voronoi structure across models on TruthfulQA. Each panel shows distinct hallucination basins corresponding to different misconception modes.
Figure 4: Efficacy of Algorithm 2 in hallucination reduction as a function of the steering strength λ .
Figure 5: Irreversibility summary under autoregressive decoding (HaluEval QA, Llama-3.2-1B, best layer). We report basin-entry, conditional irreversibility, escape-after-entry, and factual entry rates. This verifies Theorem 5.9 .
Figure 6: Layer-wise attention entropy for factual versus hallucinated generations under uninformative contexts (autoregressive extraction). Entropy trends provide a complementary signal to basin-separation metrics. Supports the uniform attention assumption.
We address hallucination detection in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) by framing the problem through the lens of dynamical systems stability theory. Rather than treating hallucination as a straightforward classification task, we conceptualize (M)LLMs as dynamical systems, where factual knowledge is represented by stable equilibrium points within the representation space. Our main insight is that hallucinations tend to arise at the boundaries of knowledge-transition regions separating stable and unstable zones. To capture this phenomenon, we propose Lyapunov Probes: lightweight networks trained with derivative-based stability constraints that enforce a monotonic decay in confidence under input perturbations. By performing systematic perturbation analysis and applying a two-stage training process, these probes reliably distinguish between stable factual regions and unstable, hallucination-prone regions. Experiments on diverse datasets and models demonstrate consistent improvements over existing baselines.
Bozhi Luan, Gen Li, Yalan Qin +6
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University · Shanghai University · School of Electronic and Information Engineering, State Key Laboratory of CNS/ATM, Beihang University +2
Hallucination in large language models (LLMs), defined as the generation of factually incorrect or unsupported content, remains a critical barrier to reliable deployment. We present BEACON (Behavioral Entropy Aggregation for Cross-model hallucination detectiON), a black-box hallucination detection framework that operates purely on model outputs without requiring access to internal representations or external knowledge bases. BEACON extracts a 31-dimensional feature vector from structured multi-pass generation, integrating NLI-based semantic entropy, embedding geometry, chain-of-thought consistency, and paraphrase stability signals. A gradient-boosted classifier trained on 7,617 labeled examples across seven benchmarks achieves 0.8123 +/- 0.0102 AUROC (95% CI: 0.7632-0.8251), outperforming standalone semantic entropy (+0.2298) and SelfCheckGPT-style consistency baselines (+0.2457). Feature importance analysis shows that hallucination is inherently multi-dimensional, requiring combined uncertainty signals. An efficient 5-call variant achieves 0.7795 AUROC, enabling practical deployment across black-box LLM APIs.
Naveen Bera, Pulijala Sai Nikhila, Kondaguduru Abhiram +6
Large language models hallucinate--producing fluent, confident, factually wrong outputs--with a consistency that persists across generations and scales. Existing taxonomies classify hallucination by output type, distinguishing intrinsic from extrinsic failures and faithfulness from factuality divergence. These frameworks are descriptively rigorous but do not identify which internal mechanism produced a given instance. This paper analyses hallucination as a structural consequence of three architectural decisions that together form a compound failure system. Self-attention's co-occurrence learning substitutes statistical proximity for semantic meaning and produces entity confusion, fact misattribution, and semantic drift. The maximum likelihood estimation training objective optimises next-token probability without factual constraint, rewarding statistically plausible outputs regardless of their truth value. Autoregressive decoding's permanent left-to-right commitment under exposure bias ensures that a single wrong token cascades forward through the entire output sequence without revision. Dataset pathologies--long-tail deficiencies, training bias, and synthetic pollution--amplify these vulnerabilities but do not independently cause them. We make three contributions. First, we map each mechanism to a specific output category in the Alansari and Luqman taxonomy, locating intrinsic hallucination in self-attention, extrinsic hallucination in MLE, and logical inconsistency in autoregressive decoding. Second, we show that each commonly cited dataset pathology exploits one of these mechanisms rather than originating hallucination independently. Third, we identify the diagnostic limitation of output-type-only classification and contrast it with inference-layer mitigation approaches.