The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models

Pranav Darshan; Pranav A; Sravan Karthick T; Minal Moharir; Ivan P. Yamshchikov

cs.CLSep 25, 2026

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models

Authors: Pranav Darshan, Pranav A, Sravan Karthick T, Minal Moharir, Ivan P. Yamshchikov

Organizations: Department of Computer Science and Engineering, R.V. College of Engineering, India · CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt, Germany

Abstract

Sampling based consistency is widely used for hallucination detection, yet aggregate performance can conceal systematic differences in which errors are detectable. This work studies that heterogeneity across four language models and three factual question answering datasets. Partitioning hallucinations by answer agreement reveals high agreement (Ghost) and low agreement (Flickering) regimes with an apparent detectability gap of $0.35$ to $0.46$ AUC. Because the statistics used to define the regimes and measure this gap are strongly coupled ( $|ρ|\approx0.94$ to $1.00$ ), the raw result is treated as a property of agreement based detection rather than independent evidence. After freezing regime assignments, lexical and semantic response dispersion preserve the asymmetry, with bootstrap $95\%$ intervals excluding zero in all $12$ model and dataset settings. A stricter test using individual diffusion trajectories and no cross seed information preserves the asymmetry across all three LLaDA datasets ( $p<0.005$ ) and directionally across all three Dream datasets, with one reaching significance. The hard regime varies substantially in prevalence across models ( $16\%$ to $77\%$ ), and matched prompts frequently change regimes between models. These findings show that aggregate detection metrics conceal persistent, model dependent heterogeneity in language model failures and motivate regime conditioned evaluation.

Figures & tables

Type	Question	Three sampled answers	Gold
Ghost	What sport does Masahito Noto play?	baseball / baseball / baseball	football
Flickering	Who composed “The Mission”?	various / James Horner / The Doors	Ennio Morricone

Table 1: Ghost hallucinations repeat incorrect answers; Flickering hallucinations produce different incorrect answers. No sampled answer contains the gold answer.

Panel of Figure 1 — Figure 1: Decoupled gap $\Delta$ , defined as Flickering AUC minus Ghost AUC. Bootstrap $95\%$ confidence intervals are shown for lexical and semantic dispersion. All twelve model and dataset settings have positive gaps.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Corpus	Triv.	Hot.	Pop.
LLaDA	scaled	1000	800	1050
Dream	dream	200	200	200
Qwen2.5 7B	AR	400	400	400
Llama 3.1 8B	AR	800	800	800

Appendix

Table 2: Prompt counts per model and dataset. Each prompt uses

K=3

seeds in the main analysis.

	Conf.	Conf.		Conf.	% high
Dataset	correct	wrong	ECE	error AUC	wrong
TriviaQA	0.951	0.950	0.52	0.502	56.4
HotpotQA	0.955	0.959	0.72	0.502	76.7
PopQA	0.964	0.965	0.64	0.497	67.5

Appendix

Table 3: LLaDA confidence baseline. Mean confidence is similar for correct and wrong answers, while confidence to error AUC remains near chance.

	Disag.	Disag.		First
Dataset	hall.	correct	AUC	sent. AUC
TriviaQA	0.41	0.24	0.61	0.71
HotpotQA	0.51	0.34	0.61	0.64
PopQA	0.59	0.34	0.66	0.73

Appendix

Table 4: Aggregate agreement across seeds on LLaDA. The first sentence variant uses content word disagreement in the first sentence.

Model	Dataset	$N$	hall.%	Ghost%	Ghost AUC	Flick. AUC	Gap [95% CI]
LLaDA	TriviaQA	1000	50.0	77.4	0.506	0.963	$0.457\,[0.429,0.485]$
LLaDA	HotpotQA	800	72.2	69.9	0.479	0.900	$0.422\,[0.390,0.458]$
LLaDA	PopQA	1050	61.9	65.1	0.521	0.917	$0.396\,[0.365,0.425]$
Dream	TriviaQA	200	49.5	30.3	0.511	0.881	$0.370\,[0.302,0.441]$
Dream	HotpotQA	200	75.5	28.5	0.340	0.796	$0.455\,[0.388,0.519]$
Dream	PopQA	200	61.0	16.4	0.443	0.795	$0.352\,[0.279,0.425]$

Appendix

Table 5: Raw agreement gap under disagreement score

1-\hat{\pi}

. Bootstrap confidence intervals use

B=2000

prompt level resamples.

Model	Dataset	$\rho(\hat{\theta},1-\hat{\pi})$
LLaDA	TriviaQA	$-0.996$
LLaDA	HotpotQA	$-0.991$
LLaDA	PopQA	$-0.998$
Dream	TriviaQA	$-1.000$
Dream	HotpotQA	$-0.939$
Dream	PopQA	$-0.985$

Appendix

Table 6: Mechanical coupling between plurality and disagreement. The raw agreement gap is therefore treated as descriptive rather than as independent evidence.

		Raw agreement			Lexical dispersion			Semantic dispersion
Model	Dataset	G	F	$\Delta$	G	F	$\Delta$	G	F	$\Delta$
LLaDA	TriviaQA	.506	.963	.457	.643	.770	.126	.690	.831	.142
LLaDA	HotpotQA	.479	.900	.422	.605	.824	.219	.642	.814	.172
LLaDA	PopQA	.521	.917	.396	.746	.880	.134	.809	.891	.083
Dream	TriviaQA	.511	.881	.370	.605	.835	.230	.618	.732	.115
Dream	HotpotQA	.340	.796	.455	.547	.789	.242	.534	.724	.190

Appendix

Table 7: Detectability gap for frozen Ghost and Flickering regimes. G and F denote Ghost and Flickering AUC. All twelve lexical and all twelve semantic confidence intervals exclude zero.

		Reassign.	Ghost %		Lexical $\Delta$		Semantic $\Delta$
Model	Dataset	%	plur.	ent.	plur.	ent.	plur.	ent.
LLaDA	TriviaQA	38.4	77.4	43.8	.126	.164	.142	.223
LLaDA	HotpotQA	29.9	69.9	43.4	.219	.187	.172	.171
LLaDA	PopQA	47.2	65.1	19.7	.134	.163	.083	.170
Dream	TriviaQA	26.3	30.3	20.2	.230	.192	.115	.165
Dream	HotpotQA	22.5	28.5	11.3	.242	.321	.190	.390

Appendix

Table 8: Regime robustness under bidirectional entailment clustering. Ghost % is the share of hallucinations assigned to Ghost under the token level plurality criterion (plur.) and the entailment criterion (ent.). Lexical and semantic

\Delta

are the decoupled Flickering minus Ghost AUC gaps under each partition. The entailment criterion reassigns a large fraction of prompts and lowers Ghost prevalence, yet both decoupled gaps remain positive in all twelve settings.

Dataset	mean	% $>0$	mean	% $>0$
	Lexical $\Delta$		Semantic $\Delta$
TriviaQA	0.306	100	0.288	100
HotpotQA	0.135	100	0.078	100
PopQA	0.065	96	0.019	76

Appendix

Table 9: Disjoint generation control on the high depth LLaDA corpus (

50

prompts,

18

seeds), averaged over

50

random nine by nine seed splits. Regime assignment uses one half of the seeds and the dispersion score uses the disjoint other half.

Analysis	Result	Interpretation
Three seed average vs. eighteen seeds	$\rho\approx.999$	closely aligned
Three seed average vs. eighteen seeds	MAE $\approx.009$	low average error
Single three seed draw vs. eighteen seeds	MAE $\approx.14$	individual draws are noisy
Regime agreement	$\approx 79\%$	most labels remain stable
HotpotQA gap, $K=3$ to $18$	$0.244$ to $0.250$	gap remains positive

Appendix

Table 10: Seed depth robustness on the high depth LLaDA corpus.

Dataset	$\tau=.34$	$\tau=.50$	$\tau=.67$	$\tau=1.0$
TriviaQA	.456	.457	.452	.452
HotpotQA	.422	.422	.428	.428
PopQA	.396	.396	.398	.398

Appendix

Table 11: Raw agreement gap under alternative token level Jaccard thresholds on LLaDA.

Model	Dataset	Ghost	Flick.	$\Delta$ [95% CI]	$p$
LLaDA	TriviaQA	.631	.737	$.106\,[.033,.178]$	.003
LLaDA	HotpotQA	.598	.721	$.123\,[.076,.170]$	$<.001$
LLaDA	PopQA	.794	.886	$.092\,[.060,.123]$	$<.001$
Dream	TriviaQA	.602	.637	$.035\,[-.081,.153]$	.60
Dream	HotpotQA	.626	.699	$.072\,[-.039,.185]$	.18
Dream	PopQA	.645	.808	$.163\,[.035,.291]$	.003

Appendix

Table 12: Trajectory only control. The detector uses dynamics within a single seed and no information across seeds.

Model	Architecture	Ghost share of hallucinations
LLaDA	diffusion	$65$ to $77$
Dream	diffusion	$16$ to $30$
Qwen2.5 7B	autoregressive	$59$ to $77$
Llama 3.1 8B	autoregressive	$62$ to $67$

Appendix

Table 13: Ghost prevalence across the three datasets.

	Dream Ghost	Dream Flickering
LLaDA Ghost	67	181
LLaDA Flickering	17	82

Appendix

Table 14: Matched regime transitions among the

347

prompts hallucinated by both models.

Behavior	LLaDA	Dream
Repeat one answer	69.8%	19.1%
Diversify	29.4%	63.7%
Fragment	not separately reported	15.1%
Repeated abstention	at most 2.2%	at most 2.2%

Appendix

Table 15: Dominant behavior among matched hallucinated prompts.

Type	Data	Question	Gold	Seed answers	$\hat{\theta}$
Ghost, LLaDA	PopQA	What sport does Masahito Noto play?	football	baseball, baseball, and baseball	$1.00$
Flickering, LLaDA	PopQA	Who composed “The Mission”?	Ennio Morricone	various, James Horner, and The Doors	$0.33$
Ghost, Dream	TriviaQA	In which city are the Oscar statuettes made?	Chicago	Hollywood, Los Angeles, and Los Angeles	$0.67$
Flickering, Dream	TriviaQA	Which composer wrote “The Dam Busters March”?	Eric Coates	an English composer, John Moore, and Edward Elgar	$0.33$
Ghost, Qwen	HotpotQA	Language of the people whose principal town was Anhaica?	Apalachee	Timucua, Timucua, and Timucua	$1.00$
Flickering, Qwen	PopQA	Who was the composer of “Hello”?	Masaharu Fukuyama	Bruno Mars, “a covered pop song”, and “various songs”	$0.33$

Appendix

Table 16: Representative hallucinations across models. Ghost examples show repeated incorrect answers, while Flickering examples show divergent incorrect answers. The final row shows a matched prompt whose regime changes between models.

	TriviaQA	HotpotQA	PopQA
Ghosts initially incorrect	387	404	423
Verified correct	4	2	0
Relabeled percentage	1.0	0.5	0.0
Ghost percentage, original to relabeled	$77.4$ to $77.2$	$69.9$ to $69.8$	$65.1$ to $65.1$
Agreement AUC, original to relabeled	$.609$ to $.611$	$.606$ to $.604$	$.659$ to $.659$
Raw Ghost AUC, original to relabeled	$.506$ to $.507$	$.479$ to $.475$	$.521$ to $.521$

Appendix

Table 17: Semantic reassessment of the Ghost population. Every quantity depending on Ghost moves by less than

0.005

Dataset	Ghost	Flick.	Ghost	Flick.
	AUC within group		Recall at FPR 30
TriviaQA	0.71	0.97	58	92
HotpotQA	0.69	0.95	63	85
PopQA	0.88	0.97	84	94

Appendix

Table 18: Full online detector on the scaled LLaDA corpus.

Feature set	Triv.	Hot.	Pop.
Final answer agreement	0.61	0.61	0.66
Per step answer agreement	0.61	0.61	0.79
Plus text overlap	0.70	0.67	0.80
Full consistency block	0.70	0.70	0.81
Full detector	0.73	0.75	0.90

Appendix

Table 19: Per step agreement compared with final answer agreement on LLaDA.

Claim	Evidence	Establishes	Dependence status	Caveat
Raw agreement gap	Four models and three datasets	Descriptive agreement based asymmetry	Mechanically coupled	Not independent evidence
Lexical asymmetry persists	Positive gap in all twelve settings	Asymmetry survives a decoupled lexical measurement	No exact computational reuse	Still a dispersion measure
Semantic asymmetry persists	Positive gap in all twelve settings	Asymmetry survives a decoupled semantic measurement	No shared lexical computation	Still a dispersion measure
Trajectory asymmetry	LLaDA significant on all three datasets; Dream positive on all three	Within seed dynamics contain additional information	No information across seeds	Limited to diffusion models
Prevalence varies	Ghost share from $16\%$ to $77\%$	Error population composition depends on model	Descriptive	Four models only
Regime membership varies	$181$ versus $17$ matched transitions	Prompt identity alone does not determine regime	Descriptive matched result	One matched model pair

Appendix

Table 20: Summary of the central evidence. The trajectory analysis is the only analysis in which the evaluated signal uses no information across seeds.

Factual QA

See all 25 papers

PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts
May 16, 2026Khizar Hussain, Murat KantarciogluLLM Evaluation Language Model Error Detection
Virginia Tech
FLaG: Fine-Grained Latent Grouping for Hallucination Detection
May 29, 2026Wentao Ye, Liyao Li, Zhiqing Xiao +6Latent Variable Models LLM Hallucination Detection
Zhejiang University · Hangzhou, Zhejiang, China · Nanyang Technological University +1
OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios
Jun 5, 2026Xinyi Li, Zhen Fang, Yongxin Deng +12LLM Evaluation Hallucination Detection
University of Technology Sydney · University of Bristol · University of Wisconsin–Madison +2

May 16, 2026J/K move · Enter open · S save

PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts

Khizar Hussain, Murat Kantarcioglu

Virginia Tech

LLM Evaluation Language Model Error Detection Hallucination in Language Models LLM Reliability LLM Hallucination Detection Language Model Probing

Open paper AskPDF arXiv

Large language models (LLMs) hallucinate with confidence: their outputs can be fluent, authoritative, and simply wrong. In medical, legal, and scientific applications this failure causes direct harm, and detecting it from internal model states offers a path to safer deployment. A growing body of work reports that this problem is increasingly tractable, with recent methods achieving high detection performance on widely used benchmarks. We show, however, that much of this apparent progress does not survive scrutiny. Four of the six corpora embed the ground-truth answer directly in the input prompt. A naïve text-similarity baseline we call TxTemb exploits this to achieve near-perfect detection scores without any access to model internals. To measure what genuine detection capability remains once these artifacts are controlled, we conduct a large-scale evaluation spanning twenty-two detection methods, twelve open-source models spanning six architectural families, and six corpora. We further introduce DRIFT, a supervised probe over inter-layer hidden-state transitions, as a point of comparison for live-generation detection. Our findings suggest that the field's reported progress on hallucination detection is substantially explained by benchmark construction artifacts in widely used corpora, and that the majority of established baselines perform near chance under controlled conditions; the consistent exceptions are SAPLMA and DRIFT, both supervised probes on upper-layer hidden states.

Abstract

Factual QA

Explore similar work

PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts

FLaG: Fine-Grained Latent Grouping for Hallucination Detection

OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios