A Geometric Analysis of Small-sized Language Model Hallucinations
Authors: Emanuele Ricco, Elia Onofri, Lorenzo Cima, Stefano Cresci, Roberto Di Pietro
Organizations: Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) division, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Istituto di Informatica e Telematica (IIT), National Research Council of Italy (CNR), Pisa, Italy · Department of Information Engineering, University of Pisa, Pisa, Italy
Hallucinations -- plausible but factually incorrect responses -- pose a major challenge to the reliability of Large Language Models (LLMs), especially in multi-step or agentic settings. Existing work largely frames hallucinations as a consequence of missing knowledge; we show instead that, even when the relevant factual knowledge is present, models still produce hallucinated answers, pointing to retrieval instability rather than knowledge gaps. Building on this observation, we introduce APORIA (Aggregate Prompt-wise Observation Retrieving Instability via Asymmetry -- the state of puzzlement-in-contradiction that hallucinations embody), a geometric framework that studies repeated responses to the same prompt in sentence-embedding space. Our central hypothesis is that genuine responses cluster more tightly than hallucinated ones; we empirically validate this and show that, after Fisher projection, the two response classes become consistently separable. We leverage this asymmetry in geometry via APORIA-LP, an efficient label-propagation method that classifies large collections of responses from as few as 30--50 annotations, achieving F1 scores above 90% across ten small-sized LLMs. To support further research, we release SOCRATES-300K, a fully labelled dataset of 300,000 responses, together with the code for both dataset generation and result reproduction. Our key finding -- framing hallucinations from a geometric perspective in the embedding space -- complements traditional knowledge-centric and single-response evaluation paradigms, paving the way for further research.
Figures & tables
Figure 1: Pipeline of the proposed method. (Left) 200 different prompts are fed into 10 LLMs, generating 150 responses each. (Center) Responses are tagged as Genuine (G, green) or Hallucinated (H, red) under Claude Sonnet 4.5, and embedded through SBERT (blue circle) in R384 (here reduced with T-SNE for plots). Two analyses follow. (Structural, top right) In APORIA , mutual distances intra-G, intra-H, and inter-(GH) (blue) are evaluated in the embedding space, showing significant differences in the distributions (cf. Figures 2 and 3 ), yet being non directly separable; data points are embedded into the linear space maximising Fisher criterion, achieving optimal separability in terms of spatial and distances distribution (cf. Table 1 ). (Label Propagation, bottom right) In APORIA-LP , the structural analysis is exploited to classify previously unseen responses generated under the same setting: here, separability in the embedding space is potentially scarce, requiring the usage of Fisher space for reliable performances (cf. Table 2 ): points are classified by the distributions of the point-to-clusters distances.
Figure 2: Distributions of mutual intra-class distances for Genuine ( D\textscgg , green) and Hallucinated ( D\textschh , red) responses. Inter-class distance distributions D\textscgh are overlaid as blue boxplots, with medians highlighted. Statistical significance, assessed using Wilcoxon rank-sum test (Mann-Whitney U ), refers to differences between D\textscgg and D\textschh : *** p<.001 , ** p<.01 , * p<.05 , ns otherwise.
Figure 3: Statistical relevance of the distributional separation between D\textscgg and D\textschh over the various prompts for qwen2.5-32B . (Top) The observed Wasserstein distance W(D\textscgg,D\textschh) , represented by ordered dots, is compared with the same distance evaluated on the null hypothesis H0 obtained by permuting labels 100 times. (Bottom) Fraction of genuine responses is reported for each prompt, also encoded in the colour channel.
Model
Original space
Fisher space
mistral-7B
1.09 (0.08)
7.45 (5.68)
deepseek-7B
1.12 (0.10)
9.44 (5.71)
llama3-8B
1.11 (0.09)
7.71 (5.23)
gemma2-9B
1.08 (0.07)
6.51 (4.25)
yi1.5-9B
1.21 (0.13)
9.53 (6.21)
solar1-11B
1.09 (0.07)
5.29 (5.73)
Table 1: Mean (std) ratio between intra- and inter-class distances.
Model
Accuracy
F1
Signed Margin
Absolute Margin
[%]
[%]
G
H
G
H
mistral-7B
86.8 (6.6)
92.1 (4.4)
-0.1 (0.3)
-0.6 (0.3)
0.4 (0.2)
0.7 (0.2)
deepseek-7B
90.9 (6.2)
94.1 (4.8)
0.4 (0.5)
-1.1 (0.5)
0.8 (0.3)
1.2 (0.4)
llama3-8B
89.1 (6.3)
92.5 (5.7)
0.3 (0.5)
-0.9 (0.5)
0.7 (0.3)
1.0 (0.3)
gemma2-9B
86.1 (7.1)
91.5 (5.0)
-0.0 (0.4)
-0.7 (0.4)
0.5 (0.2)
0.7 (0.3)
yi1.5-9B
92.5 (4.8)
95.3 (3.7)
0.5 (0.5)
-1.1 (0.5)
0.9 (0.3)
1.1 (0.3)
Table 2: Mean (std) of the label propagator performances.
Figure 4: Evolution of the label-propagator F1-score with the training-set size, from 5 to 100 responses (step 5). Each model is plotted as mean ( x -axis) versus standard deviation ( y -axis): the further right and down, the better (cf. Figure 9 ).
Figure 5: Sensitivity analysis of the λ regularisation parameter for each of the models. Best choice is reported as a dashed black line.
Encoder
Dim
W ratio
Fisher sep.
LP F1 [%]
L6-v2 (main)
384
1.13
7.21
86.1
L12-v2
384
1.13
7.38
86.3
base-v2
768
1.10
10.47
87.2
mpnet-base-v2
768
1.11
10.59
87.4
Table 3: Ablation of structural statistics and LP metrics with respect to the choice of sentence encoder.
Baseline
Space
Accuracy
F1
[%]
[%]
INSIDE
SBERT 384D
71.0 (27.8)
74.9 (25.9)
SemanticEntropy
NLI clusters
78.5 (5.5)
86.3 (4.3)
SelfCheckGPT (NLI)
—
82.0 (5.7)
89.8 (3.8)
APORIA-LP
SBERT 384D
87.2 (7.4)
90.7 (7.1)
Table 4: Mean (std) performances achieved by external baseline.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Question Template
0
Who won the Australian Open men’s singles title in [year]?
1
Who won the Australian Open women’s singles title in [year]?
2
Who won the Roland Garros men’s singles title in [year]?
3
Who won the Roland Garros women’s singles title in [year]?
4
Who won the Wimbledon men’s singles title in [year]?
5
Who won the Wimbledon women’s singles title in [year]?
Appendix
Table 5: Complete prompt set (100 total). Each template is instantiated for both 2020 and 2022.
Type
Count
%
Description
(i) Minor imprecisions
27
22.30%
Response is substantively correct but contains a minor inaccuracy (e.g., approximate date, slightly imprecise name, wrong minor detail); Claude penalises strictly while human annotators judge the core answer as correct.
(ii) Temporal ambiguity
12
9.96%
The question targets a specific year, but the correct answer spans an ambiguous boundary (e.g., an event announced in late 2019 and completed in 2020), or the model answered with a correct fact which did not occur in the requested period.
(iii) Prompt repetition
11
9.08%
The response contains the correct answer but also repeats or paraphrases the original question; Claude treats the repetition as a sign of low-quality generation and flags it as hallucinated.
(iv) Unrequested information
40
33.05%
Response provides the correct answer alongside additional (unrequested but factually correct) information; Claude flags the extra content as potentially fabricated.
(v) Inaccessible ground truth
31
25.61%
Claude consistently labels correct answers to the same prompt as incorrect (or vice-versa), likely due to limited access to the ground-truth information.
Appendix
Table 6: Categorisation of disagreements between Claude’s labels and human annotations on the manually validated subset.
Model
Hallucination Rate [%]
Dropped Prompts [%]
2020
2022
Global
2020
2022
Global
mistral-7B
86.1 (7.8)
89.0 (6.2)
87.6 (7.1)
28.0
26.0
27.0
deepseek-7B
84.5 (11.6)
85.0 (9.7)
84.8 (10.7)
29.0
35.0
32.0
llama3-8B
80.4 (13.0)
82.5 (11.6)
81.4 (12.3)
21.0
16.0
18.5
gemma2-9B
85.6 (6.8)
86.2 (6.6)
85.9 (6.7)
24.0
5.0
14.5
yi1.5-9B
85.9 (11.1)
88.1 (8.5)
86.9 (10.0)
36.0
47.0
41.5
Appendix
Table 7: Prompt-level hallucination rates and dropped prompt percentages across models for years 2020, 2022, and global aggregate. Standard deviations in parentheses.
Model
Full responses
Stemmed responses
Original space
Fisher space
Original space
Fisher space
mistral-7B
1.09 (0.08)
7.45 (5.68)
1.07 (0.06)
7.80 (5.43)
deepseek-7B
1.12 (0.10)
9.44 (5.71)
1.11 (0.10)
9.68 (6.02)
llama3-8B
1.11 (0.09)
7.71 (5.23)
1.10 (0.08)
7.91 (5.42)
gemma2-9B
1.08 (0.07)
6.51 (4.25)
1.06 (0.05)
6.82 (4.31)
yi1.5-9B
1.21 (0.13)
9.53 (6.21)
1.18 (0.12)
9.88 (6.18)
Appendix
Table 8: Mean (std) of ratio between inter-class distances to intra-class distances on full (cf. Table 1 ) and stemmed responses.
Model
Accuracy [%]
F1 [%]
Full responses
Stemmed responses
Signed Margin
Absolute Margin
Signed Margin
Absolute Margin
Full
Stemmed
Full
Stemmed
G
H
G
H
G
H
G
H
mistral-7B
86.8 (6.6)
86.7 (6.8)
92.1 (4.4)
92.1 (4.6)
-0.1 (0.3)
-0.6 (0.3)
0.4 (0.2)
0.7 (0.2)
-0.1 (0.3)
-0.6 (0.3)
0.4 (0.2)
0.7 (0.2)
deepseek-7B
90.9 (6.2)
90.6 (6.4)
94.1 (4.8)
93.9 (5.0)
0.4 (0.5)
-1.1 (0.5)
0.8 (0.3)
1.2 (0.4)
0.3 (0.6)
-1.1 (0.5)
0.8 (0.3)
1.2 (0.4)
llama3-8B
89.1 (6.3)
88.8 (6.4)
92.5 (5.7)
92.3 (5.8)
0.3 (0.5)
-0.9 (0.5)
0.7 (0.3)
1.0 (0.3)
0.2 (0.5)
-0.9 (0.4)
0.7 (0.3)
1.0 (0.3)
gemma2-9B
86.1 (7.1)
86.1 (7.3)
91.5 (5.0)
91.6 (5.0)
-0.0 (0.4)
-0.7 (0.4)
0.5 (0.2)
0.7 (0.3)
-0.1 (0.4)
-0.7 (0.4)
0.5 (0.2)
0.7 (0.3)
Appendix
Table 9: Label propagation accuracy, F1 and margin across models both for full and stemmed responses.
Model
Full responses
Stemmed responses
%W>WˉH0
E[W]
E[WˉH0]
%W>WˉH0
E[W]
E[WˉH0]
mistral-7B
84.1
0.93 (0.47)
0.36 (0.21)
77.8
0.73 (0.38)
0.33 (0.20)
deepseek-7B
88.0
1.07 (0.71)
0.37 (0.23)
83.2
0.92 (0.66)
0.36 (0.23)
llama3-8B
86.2
0.94 (0.52)
0.32 (0.22)
84.1
0.80 (0.47)
0.31 (0.22)
gemma2-9B
81.5
0.78 (0.49)
0.32 (0.20)
77.2
0.62 (0.40)
0.29 (0.19)
yi1.5-9B
92.7
1.81 (0.83)
0.44 (0.23)
91.4
1.51 (0.76)
0.42 (0.23)
Appendix
Table 10: Observed Wasserstein distances versus null hypothesis. Percentage of prompts exceeding null baseline, mean observed E[W] and null E[WˉH0] distances (std in parentheses).
Figure 6: Statistical relevance of the distributional separation between D\textscgg and D\textschh over the various prompts for gemma2-9B . (Top) The observed Wasserstein distance W(D\textscgg,D\textschh) , represented by ordered dots, is compared with the same distance evaluated on the null hypothesis H0 obtained by permuting labels 100 times. (Bottom) Fraction of genuine responses is reported for each prompt, also encoded in the colour channel.
Figure 7: Fraction of (model, prompt) pairs showing distributional separation between D\textscgg and D\textschh as a function of subsample size N . For each pair and each N , three criteria are evaluated over 20 repeated subsamples: the Mann-Whitney U test on D\textscgg vs. D\textschh (solid, primary), the significance of Wasserstein distance against null permutations (dashed), and its point estimate Wobs>WˉH0 without a significance threshold (dotted). Numerical values are reported in the table on the right. The plot on the left also shows per-model Mann-Whitney rates (faint coloured lines) and the point N=30 at which label-propagation performance saturates (cf. Figures 4 and 9 ).
Figure 8: Label propagation performance across models and prompts. Box plots display the distribution of classification accuracy (left) and F1 score (right) across all 200 prompts for each model, evaluated using stratified shuffle split cross-validation to account for class imbalance at the prompt level. Each box represents the interquartile range of performance metrics computed independently for each prompt’s response set.
Figure 9: Evolution of the label-propagator accuracy with the training-set size, from 5 to 100 responses (step 5). The left plot shows model-wise results as mean ( x -axis) versus standard deviation ( y -axis): the further right and down, the better (cf. Figure 4 ). The right table reports aggregate values as mean (std).
Model
Prompts
Total Responses
Hallucinations
Rate [%]
LLaMA-13B
199
29,850
17,927
60.06
LLaMA-7B
199
29,850
20,169
67.57
OPT-6.7B
199
29,850
21,236
71.14
Total
598
89,550
66,262
66.26
Appendix
Table 11: Hallucination rates across models on CoQA-89K .
Figure 10: Fraction of (model, prompt) pairs showing distributional separation between D\textscgg and D\textschh as a function of subsample size N when the methodology is evaluated on the CoQA-89K bridge dataset (plots mirror the structure of Figure 7 ).
Figure 11: Label-propagator statistics when applied to the CoQA-89K bridge dataset (cf. Figure 9 ). Results are reported across training-set sizes from 5 to 100 responses (step 5). Left and right plots show model-wise F1 and accuracy as mean ( x -axis) versus standard deviation ( y -axis): the further right and down, the better (cf. Figure 4 ). The central table reports aggregate values as mean (std).
Model
APORIA structural
APORIA-LP
INSIDE
W ratio
Fisher sep.
Acc [%]
F1 [%]
AUC-ROC [%]
( Chen et al., 2024 ) [%]
LLaMA-7B
2.17
66.92
98.1
96.6
78.2
80.8
LLaMA-13B
2.26
107.74
98.5
97.3
83.8
80.8
OPT-6.7B
1.90
64.30
97.8
97.3
69.0
77.5
Average
2.11
79.65
98.1
97.1
77.0
79.7
Appendix
Table 12: Cross-dataset results on CoQA-89K for the three models evaluated in INSIDE ( Chen et al., 2024 ) .
Method
Fisher 1D
SBERT 384D
Acc [%]
F1 [%]
Acc [%]
F1 [%]
APORIA-LP (Wass.)
87.0 (8.1)
90.5 (8.8)
72.7 (14.5)
79.9 (14.3)
Centroid (Euclidean)
86.0 (8.4)
89.7 (9.0)
80.1 (10.8)
84.9 (11.1)
Centroid (Cosine)
–
–
80.0 (10.7)
84.8 (11.0)
Linear Regression
88.3 (7.7)
91.4 (9.2)
87.8 (8.0)
91.1 (9.1)
SVM
88.0 (7.8)
91.3 (8.9)
83.0 (9.3)
87.7 (9.3)
Appendix
Table 13: Comparison of label-propagation methods across embedding spaces. Results are mean (std) over models and prompts.
Method
# Components
Accuracy [%]
F1 [%]
M
∣M∣
% Agree
Fisher
1
86.9 (8.1)
90.5 (8.8)
-0.60 (0.55)
0.78 (0.29)
100.0 (0.0)
UMAP
1
67.9 (16.8)
70.1 (23.6)
-0.05 (11.86)
13.11 (4.45)
71.5 (16.8)
2
73.7 (16.3)
75.2 (23.4)
-0.73 (9.86)
11.63 (2.99)
77.7 (15.9)
3
73.9 (16.0)
75.4 (23.3)
-0.86 (9.80)
11.61 (2.85)
77.9 (15.6)
5
73.8 (16.0)
75.2 (23.4)
-0.87 (9.85)
11.66 (2.81)
77.8 (15.6)
10
73.9 (15.9)
75.2 (23.4)
-0.94 (9.80)
11.56 (2.82)
77.9 (15.4)
Appendix
Table 14: Performances of different projection methods (Fisher vs. UMAP vs. wPCA vs. EP) over the label propagation scheme.
Figure 12: Label propagation accuracy and F1 score vs. the number of components for different projection methods.
Large language models are known to hallucinate, generating linguistically plausible but incorrect answers to questions. Uncertainty quantification has been proposed as a strategy to detect such behaviour, but existing methods lack a unified framework to assess reliability at both the prompt and answer level. We introduce a geometric framework which quantifies language model uncertainty at both levels by explicitly modelling a prompt-conditioned semantic distribution in answer embedding space. Our approach is black-box and sampling-based; we generate multiple answers per prompt, and use archetypal analysis to estimate a geometric support for the answer distribution. At the prompt level, we approximate the distribution entropy to quantify uncertainty; for each individual answer, we then use notions of atypicality to assess its reliability relative to the batch. We employ our framework to not only detect hallucinations but correct them, by selecting the batch example deemed most reliable. Experiments show that our framework performs comparably to or better than prior methods on short form question-answering datasets, and achieves superior results on medical datasets where hallucinations carry particularly critical risks. Beyond pure performance, we suggest the theoretical grounding of our work provides support for semantic distributions as useful objects of study for language model uncertainty.
Edward Phillips, Sean Wu, Soheila Molaei +3
Department of Engineering Science, University of Oxford · GlaxoSmithKline · Oxford Suzhou Centre for Advanced Research
As large language models (LLMs) evolve from conversational assistants into agents capable of handling complex tasks, they are increasingly deployed in high-risk domains. However, existing benchmarks largely rely on mixed queries and posterior evaluation, output-level scoring, which quantifies hallucination severity but offers limited insight into where and why hallucinations arise in the generation pipeline. We therefore reformulate hallucination evaluation as a diagnostic problem and propose PRISM, a controlled benchmark that disentangles hallucinations into four dimensions: knowledge missing, knowledge errors, reasoning errors, and instruction-following errors, grounded in three stages of generation (memory, instruction, and reasoning). PRISM contains 9,448 instances across 65 tasks and supports fine-grained, stage-aware diagnostic evaluation. Evaluating 24 mainstream open-source and proprietary LLMs, we uncover consistent trade-offs across instruction following, memory retrieval, and logical reasoning, showing that mitigation strategies often improve specific dimensions at the expense of others. We hope PRISM provides a framework for understanding the specific mechanisms behind LLMs hallucinations, ultimately accelerating the development of trustworthy large language models.
Large language models can hallucinate even when the knowledge required for a correct answer is already available. We study this failure through a latent-key view of inference, where answer selection depends on competition among associations acquired during pretraining. We show that model predictions can be highly sensitive to individual query keywords, that these influential keywords exhibit entity-specific binding, and that their effects are systematically shaped by pretraining frequency. Multiple bindings can also compete and exhibit higher-order interactions within the same query. Based on this mechanism, we introduce a two-stage keyword-perturbation method for hallucination detection. By removing influential keywords and measuring how the model reorganizes its prediction, the method distinguishes errors caused by misleading key associations from correct decisions supported by diagnostic evidence. Across multiple models and benchmarks, perturbation provides a strong and transferable detection signal, reaching 0.910 AUROC on probe-known ScientistQA. Finally, we extend the same probabilistic framework to four hallucination regimes: knowledge deficit, wrong knowledge, context distraction, and unstable inference. Their operational distributions across benchmarks provide diagnostic context for why different detector families succeed in different settings.