From Prerequisites to Predictions: Validating a Geometric Hallucination Taxonomy Through Controlled Induction

Matic Korun

cs.CLFeb 27, 2026

From Prerequisites to Predictions: Validating a Geometric Hallucination Taxonomy Through Controlled Induction

Authors: Matic Korun

Organizations: Independent Researcher Ljubljana, Slovenia

Abstract

We test whether a geometric hallucination taxonomy -- classifying failures as center-drift (Type~~1), wrong-well convergence (Type~~2), or coverage gaps (Type3) -- can distinguish hallucination types through controlled induction in GPT-2. Using a two-level statistical design with prompts ( $N = 15$ /group) as the unit of inference, we run each experiment 20 times with different generation seeds to quantify result stability. In static embeddings, Type3 norm separation is robust (significant in 18/20 runs, Holm-corrected in 14/20, median $r = +0.61$ ). In contextual hidden states, the Type~~3 norm effect direction is stable (19/20 runs) but underpowered at $N = 15$ (significant in 4/20, median $r = -0.28$ ). Types~~1 and2 do not separate in either space ( ${\leq}\,3/20$ runs). Token-level tests inflate significance by 4--16 $\times$ through pseudoreplication -- a finding replicated across all 20 runs. The results establish coverage-gap hallucinations as the most geometrically distinctive failure mode, carried by magnitude rather than direction, and confirm the Type1/2 non-separation as genuine at 124M parameters.

Figures & tables

	Z1	Z2	Z3	Unc.
Type 1	.706 $\pm$ .013	.041 $\pm$ .007	.088 $\pm$ .013	.165 $\pm$ .007
Type 2	.708 $\pm$ .012	.039 $\pm$ .006	.090 $\pm$ .011	.163 $\pm$ .011
Type 3	.654 $\pm$ .019	.042 $\pm$ .006	.141 $\pm$ .020	.163 $\pm$ .015

Table 1: Static confusion matrix (mean

\pm

SD across 20 runs). Bold = diagonal. Mean diagonal =

0.295\pm 0.009

. All conditions collapse to Zone 1.

Metric	Pair	Med. $p$	Sig/20	Holm/20	Med. $r$
$\\|\mathbf{v}\\|$	T1–T3	.011	13	11	$+.55$
$\\|\mathbf{v}\\|$	T2–T3	.005	18	14	$+.61$
$\\|\mathbf{v}\\|$	T1–T2	.663	2	1	$.00$
max_sim	T1–T3	.203	7	4	$-.28$
max_sim	T2–T3	.432	3	2	$-.13$
max_sim	T1–T2	.213	0	0	$-.26$

Table 2: Static pairwise results across 20 runs. Med.

p

: median prompt-level Mann-Whitney

p

. Sig/20: runs reaching

p<0.05

. Holm/20: runs surviving Holm-Bonferroni correction. Med.

r

: median rank-biserial effect size. Only comparisons with

\geq 3

/20 significance shown (except T1–T2 norm and max_sim, reported as confirmed nulls). Type 3 norm is the only robust separating variable.

Figure 1: Effect-size stability across 20 independent runs. Each row shows one metric $\times$ pairwise comparison. Thick bars: IQR of rank-biserial $r$ ; thin whiskers: full range; dots: median. Green shading: Holm-corrected significance in ${\geq}50\%$ of runs. Yellow shading: ${\geq}15\%$ . Right annotations: nominal significance rate. Static norm T1–T3 and T2–T3 are the only comparisons with stable Holm survival. Contextual effects are consistently directional but underpowered at $N=15$ .

	Z1	Z2	Z3	Unc.
Type 1	.073 $\pm$ .011	.331 $\pm$ .021	.124 $\pm$ .017	.472 $\pm$ .020
Type 2	.080 $\pm$ .011	.332 $\pm$ .019	.117 $\pm$ .017	.470 $\pm$ .012
Type 3	.092 $\pm$ .018	.282 $\pm$ .018	.148 $\pm$ .015	.477 $\pm$ .029

Table 3: Contextual confusion matrix (mean

\pm

SD across 20 runs). Mean diagonal =

0.185\pm 0.008

. Zone classification fails; distributional differences are directionally stable but underpowered at prompt level (§ 9 ).

Metric	Med. prompt $p$	Sig/20	Med. perm. $p$
$H(\mathbf{v})$	$4.2\times 10^{-1}$	1/20	$4.3\times 10^{-1}$
$\\|\mathbf{h}\\|$	$\mathbf{2.6\times 10^{-1}}$	3/20	$\mathbf{2.6\times 10^{-1}}$
max_sim	$3.6\times 10^{-1}$	0/20	$3.7\times 10^{-1}$

Table 4: Contextual omnibus Kruskal-Wallis across 20 runs. Med. prompt

p

: median prompt-level

p

-value. Sig/20: runs reaching

p<0.05

Pair	Metric	Med. $p$	Sig	Holm	Med. $r$	$r$ IQR
T1 vs T3	$\\|\mathbf{h}\\|$	.192	4/20	2/20	$-.28$	[ $-.38$ , $-.22$ ]
T2 vs T3	$\\|\mathbf{h}\\|$	.206	3/20	1/20	$-.28$	[ $-.36$ , $-.14$ ]
T1 vs T2	$\\|\mathbf{h}\\|$	.537	1/20	1/20	$-.01$	[ $-.21$ , $+.07$ ]

Table 5: Contextual pairwise norm results across 20 runs. The Type 3 norm effect direction is consistent (19/20 runs show Type 3 lowest), but nominal significance is reached in only 4/20 runs, confirming that

N=15

is underpowered for this effect size (

|r|\approx 0.28

). Angular metrics (

H(\mathbf{v})

, max_sim) do not reach significance at prompt level on any pairwise comparison (

{\leq}\,3/20

runs,

{\leq}\,1/20

Holm), consistent with the pseudoreplication finding (Figure 2 ).

	Type 1	Type 2	Type 3
$H(\mathbf{v})$	$.985\pm.002$	$.984\pm.003$	$.985\pm.002$
$\\|\mathbf{h}\\|$	$245.8\pm 3.4$	$243.6\pm 2.9$	$\mathbf{237.3\pm 3.0}$
max_sim	$.9936\pm.001$	$.9933\pm.001$	$.9930\pm.001$

Table 6: Per-condition prompt-level means

\pm

SD across 20 runs (contextual,

N=15

/group). Type 3 shows consistently lower norms. The standard deviations reflect run-to-run generation variance.

Result	Static	Contextual
Norm T3 direction	20/20 highest	19/20 lowest
Norm T1–T3 sig/20	13/20	4/20
Norm T2–T3 sig/20	18/20	3/20
Norm T1–T2 sig/20	2/20	1/20
Median $r$ (T2–T3)	$+0.61$	$-0.28$
$H(\mathbf{v})$ omnibus sig/20	2/20	1/20

Table 7: Multi-run comparison across experiments. The Type 3 norm direction is stable in both spaces; significance rates reflect the larger effect size in static space (median

r=0.61

vs.

0.28

). Zone classification fails in both spaces.

Figure 2: Pseudoreplication inflation in the contextual experiment across 20 runs. Each row shows one metric $\times$ pairwise comparison. Squares: prompt-level significance rate ( $N=15$ ). Circles: token-level significance rate ( $N\approx 900$ ). Connectors show the inflation gap, annotated as a percentage. The largest inflation (max_sim T2–T3) reaches 16 $\times$ : significant at token level in 80% of runs but at prompt level in only 5%. This replicated pattern provides an empirical benchmark: researchers observing token-level significance in autoregressive generation should expect 4–16 $\times$ inflation relative to properly aggregated prompt-level analysis.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Contextual hidden-state signatures in norm– $H(\mathbf{v})$ space from a representative run (seed closest to median $p$ -values). Colored points: generated tokens per condition. Dashed lines: percentile-calibrated zone boundaries. The narrow dynamic range of contextual $H(\mathbf{v})$ ( ${\approx}\,0.93$ – $0.999$ ) is evident. Type 3 shows a slight leftward (lower norm) shift visible in the aggregate but not individually distinguishable.

Figure 4: Distribution of prompt-level Mann-Whitney $p$ -values across 20 runs for the norm metric. Each dot is one run; horizontal bars show medians. Dashed line: $\alpha=0.05$ . Left: static embeddings show T1–T3 and T2–T3 medians well below $\alpha$ . Right: contextual hidden states show medians above $\alpha$ despite the same directional effect, confirming the underpowered regime.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 247 papers

Whitening Reveals Cluster Commitment as the Geometric Separator of Hallucination Types
Mar 8, 2026Matic KorunFeature Whitening Hallucination in Language Models
Independent Researcher, Ljubljana, Slovenia
From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity
May 1, 2026Yee Zhing Liew, Andrew Huey Ping Tan, Anwar P. P. Abdul MajeedLoss Landscape Sharpness Hallucination Detection
School of Intelligent Manufacturing Ecosystem, Xi’an Jiaotong-Liverpool University, People’s Republic of China · Department of Computer Science, University of Liverpool, United Kingdom · Faculty of Engineering and Technology, Sunway University, Malaysia +1
A Geometric Analysis of Small-sized Language Model Hallucinations
Feb 16, 2026Emanuele Ricco, Elia Onofri, Lorenzo Cima +2Hallucination in Language Models LLM Hallucination Detection
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) division, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Istituto di Informatica e Telematica (IIT), National Research Council of Italy (CNR), Pisa, Italy · Department of Information Engineering, University of Pisa, Pisa, Italy

Mar 8, 2026J/K move · Enter open · S save

Whitening Reveals Cluster Commitment as the Geometric Separator of Hallucination Types

Matic Korun

Independent Researcher, Ljubljana, Slovenia

Feature Whitening Hallucination in Language Models Representation Geometry in Language Models

Open paper AskPDF arXiv

A geometric hallucination taxonomy distinguishes three failure types -- center-drift (Type~~1), wrong-well convergence (Type~~2), and coverage gaps (Type~~3) -- by their signatures in embedding cluster space. Prior work found Types~~1 and2 indistinguishable in full-dimensional contextual measurement. We address this through PCA-whitening and eigenspectrum decomposition on GPT-2-small, using multi-run stability analysis (20 seeds) with prompt-level aggregation. Whitening transforms the micro-signal regime into a space where peak cluster alignment (max_sim) separates Type2 from Type~~3 at Holm-corrected significance, with condition means following the taxonomy's predicted ordering: Type~~2 (highest commitment)

>

Type~~1 (intermediate) $>$ Type~~3 (lowest). A first directionally stable but underpowered hint of Type1/2 separation emerges via the same metric, generating a capacity prediction for larger models. Prompt diversification from 15 to 30 prompts per group eliminates a false positive in whitened entropy that appeared robust at the smaller set, demonstrating prompt-set sensitivity in the micro-signal regime. Eigenspectrum decomposition localizes this artifact to the dominant principal components and confirms that Type1/2 separation does not emerge in any spectral band, rejecting the spectral mixing hypothesis. The contribution is threefold: whitening as preprocessing that reveals cluster commitment as the theoretically correct separating metric, evidence that the Type~1/2 boundary is a capacity limitation rather than a measurement artifact, and a methodological finding about prompt-set fragility in near-saturated representation spaces.

Abstract

Hallucination in Language Models

Explore similar work

Whitening Reveals Cluster Commitment as the Geometric Separator of Hallucination Types

From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity

A Geometric Analysis of Small-sized Language Model Hallucinations