The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

Jorma Valjakka; Juhani Kivimäki; Juha Mylläri; Jukka K. Nurminen

cs.CLOct 6, 2026

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

Authors: Jorma Valjakka, Juhani Kivimäki, Juha Mylläri, Jukka K. Nurminen

Organizations: Department of Computer Science University of Helsinki

Abstract

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions within the QA dataset. Then, some automated labeling strategy is used to label these answers as hallucinated or not by comparing them with the reference answers in the dataset. This evaluation setting creates a methodological ambiguity between two criteria: reference faithfulness (whether the answer is fully supported by the reference) and factual correctness (whether the answer is free from contradictions and factually false specific claims). In practice, automated labelers may apply the former criterion even when the intended target is the latter. We study this potential criterion mismatch using 900 human-labeled question-answer pairs spanning three commonly used QA datasets and three generator models, with labels targeting answer-level factual correctness. We evaluate lexical similarity metrics, a reference-entailment NLI baseline, and seven LLM judges under controlled prompt variants as automated labelers. Our experiments reveal substantial disagreement both among automated labeling strategies and between these labels and human annotations. Many strategies also exhibit strong directional error biases, and for most judge-generator pairs, replacing a faithfulness-oriented prompt with a factual-correctness prompt improves agreement with human annotations and reduces false-positive dominance, indicating that automated hallucination labels depend strongly on how the target criterion is specified. Label-source choice should therefore be considered a fundamental part of benchmark design and made explicit, validated, and matched with the benchmark goal.

Figures & tables

API judges
		p1				p2				p3
Judge	Gen	$\kappa$	Bias	FP/FN	FP%	$\kappa$	Bias	FP/FN	FP%	$\kappa$	Bias	FP/FN	FP%
GPT-5-mini	L	0.530	$+$ 0.223	70/3	95.9	0.756	$-$ 0.020	15/21	41.7	0.744	$+$ 0.007	20/18	52.6
GPT-5-mini	G	0.498	$+$ 0.193	68/10	87.2	0.770	$-$ 0.033	11/21	34.4	0.759	$-$ 0.007	16/18	47.1
GPT-5-mini	M	0.194	$+$ 0.383	117/2	98.3	0.626	$+$ 0.027	32/24	57.1	0.646	$+$ 0.030	31/22	58.5
GPT-5-nano	L	0.536	$+$ 0.150	58/13	81.7	0.655	$-$ 0.017	23/28	45.1	0.676	$+$ 0.000	24/24	50.0

Table 1: Prompt-ablation results against human labels for each judge–generator pair. Bias is positive when the automated labeler over-labels hallucinations relative to humans. FP/FN reports false positives and false negatives; FP% is false positives as a proportion of all errors. Best

\kappa

within each row is shown in bold. All values are computed against annotator1_label , the complete primary human label set for the 900 question–answer pairs. Gen denotes the answer-generating model: L=Llama-3-8B, G=Gemma-2-9B, and M=Mistral-7B.

Metric	Gen	$\tau$	$\kappa$	Bias	FP	FN	FP%
ROUGE-L (oracle- $\tau$ )	L	0.06	0.403	$-$ 0.183	15	70	17.6
ROUGE-L (oracle- $\tau$ )	G	0.07	0.483	$-$ 0.057	27	44	38.0
ROUGE-L (oracle- $\tau$ )	M	0.03	0.412	$-$ 0.257	6	83	6.7
ROUGE-L (CV- $\tau$ )	L	0.07 [0.06, 0.10]	0.353	$-$ 0.143	25	68	26.9
ROUGE-L (CV- $\tau$ )	G	0.07 [0.06, 0.07]	0.473	$-$ 0.067	26	46	36.1
ROUGE-L (CV- $\tau$ )	M	0.03 [0.03, 0.04]	0.385	$-$ 0.230	12	81	12.9

Table 2: Lexical similarity and NLI baselines against human-annotated factual-correctness labels. Oracle-

\tau

reports a threshold tuned directly on the evaluation set. CV-

\tau

reports 3-fold out-of-fold thresholding within each generator. For NLI, best-

\tau

uses an oracle threshold. Bias is positive when the automated labeler over-labels hallucinations relative to humans. FP% is false positives as a proportion of all errors. L=Llama-3-8B, G=Gemma-2-9B, M=Mistral-7B.

Figure 1: Error-direction space of automated label sources. Each point shows macro-averaged false-positive and false-negative rates over the three generator sets. The main p1

\to

p2 movement is a reduction in false positives, while local judges and non-LLM baselines occupy distinct regions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	Disagr.	Agree.	$\kappa$	CI low	CI high
TriviaQA	99	9	0.909	0.796	0.618	0.935
TruthfulQA	99	14	0.859	0.690	0.544	0.816
HotpotQA	102	6	0.941	0.832	0.642	0.968
Overall	300	29	0.903	0.808	0.729	0.879

Appendix

Table 3: Independent inter-annotator agreement on the 300-item subset. Confidence intervals are 95% question-level bootstrap intervals for Cohen’s

\kappa

with 1,000 resamples, keeping the three generator responses to each question together.

Judge	Provider / deployment	Role in experiments
GPT-5-mini	OpenAI API	Primary ablation judge
GPT-5-nano	OpenAI API	Weaker API judge
GPT-5.4	OpenAI API	Frontier OpenAI judge
Claude Opus 4.7	Anthropic API	Frontier Anthropic judge
Gemma-2-9B-it	Local open-weight	Local judge; includes one self condition
Llama-3-8B-Instruct	Local open-weight	Local judge; includes one self condition

Appendix

Table 4: Judge model identifiers and evaluation roles. API access dates are reported in the bibliography entries for the corresponding model documentation.

Judge	Type	Score
GPT-5-mini	API	6/6
GPT-5-nano	API	6/6
GPT-5.4	API	6/6
Claude Opus 4.7	API	6/6
Gemma-2-9B	Local	6/6
Llama-3-8B	Local	6/6

Appendix

Table 5: Six-branch sanity check for the p2 factual-correctness prompt. The check tests basic prompt-following behavior on simple diagnostic cases and is not used to estimate benchmark performance.

Gen	Contrast	$\kappa_{a}$	$\kappa_{b}$	$\Delta\kappa$ [95% CI]	McNemar $p$
L	p1 $\to$ p2	0.530	0.756	+0.226 [+0.127, +0.328]	$<0.0001$
L	p1 $\to$ p3	0.530	0.744	+0.214 [+0.121, +0.313]	$<0.0001$
L	p2 $\to$ p3	0.756	0.744	-0.012 [-0.060, +0.039]	0.7905
G	p1 $\to$ p2	0.498	0.770	+0.272 [+0.179, +0.360]	$<0.0001$
G	p1 $\to$ p3	0.498	0.759	+0.261 [+0.171, +0.347]	$<0.0001$
G	p2 $\to$ p3	0.770	0.759	-0.011 [-0.059, +0.037]	0.7744

Appendix

Table 6: Paired prompt comparisons for GPT-5-mini against the primary human labels. In each contrast, prompt

a

is the prompt before the arrow and prompt

b

is the prompt after the arrow.

\kappa_{a}

and

\kappa_{b}

denote the Cohen’s

\kappa

agreements of prompt

a

and prompt

b

with the human labels, respectively;

\Delta\kappa=\kappa_{b}-\kappa_{a}

. Brackets give 95% nonparametric bootstrap intervals for

\Delta\kappa

with 2,000 item-level resamples. McNemar’s

p

is the exact two-sided paired test over item-level correctness disagreements between the two prompt variants. L=Llama-3-8B, G=Gemma-2-9B, M=Mistral-7B.

	HotpotQA		TriviaQA		TruthfulQA
Generator	p1	p2	p1	p2	p1	p2
Llama-3-8B	0.714	0.821	0.448	0.805	0.352	0.558
Gemma-2-9B	0.583	0.711	0.543	0.821	0.273	0.662
Mistral-7B	0.151	0.564	0.041	0.597	0.212	0.484

Appendix

Table 7: Per-dataset Cohen’s

\kappa

agreement between GPT-5-mini and the primary human labels under p1 and p2. The breakdown checks whether the p1

\to

p2 improvement is consistent across HotpotQA, TriviaQA, and TruthfulQA rather than being driven by a single dataset. Results are shown for each generator model.

Prompt	Criterion	Structure	Gen	$\kappa$	Bias	FP	FN	FPR
p1	Faithfulness	Terse	L	0.532	$+$ 0.237	72	1	0.434
p1	Faithfulness	Terse	G	0.466	$+$ 0.220	75	9	0.403
p1	Faithfulness	Terse	M	0.174	$+$ 0.387	119	3	0.810
p1	Faithfulness	Terse	All	0.409	$+$ 0.281	266	13	0.533
p1s	Faithfulness	Structured	L	0.404	$+$ 0.293	91	3	0.548
p1s	Faithfulness	Structured	G	0.344	$+$ 0.280	95	11	0.511

Appendix

Table 8: Criterion-by-structure factorial prompt experiment with GPT-5-mini. All four prompts were evaluated on the same 900 items in a single interleaved pass. p1 and p2 are the paper prompts; p1s applies the reference-faithfulness criterion in p2’s structured decision-tree format, and p2t applies the factual-correctness criterion in p1’s terse format. Bias is positive when the automated labeler over-labels hallucinations relative to humans. L=Llama-3-8B, G=Gemma-2-9B, M=Mistral-7B; All pools the three generator sets.

Contrast	Held constant	Diff%	$\kappa_{a}$	$\kappa_{b}$	$\Delta\kappa$ [95% CI]
p1 $\to$ p2t	Structure: terse	29.6	0.409	0.714	+0.305 [+0.240, +0.369]
p1s $\to$ p2	Structure: structured	34.2	0.305	0.739	+0.434 [+0.371, +0.494]
p2t $\to$ p2	Criterion: factual correctness	4.3	0.714	0.739	+0.025 [+0.000, +0.053]
p1 $\to$ p1s	Criterion: reference faithfulness	9.9	0.409	0.305	$-$ 0.104 [-0.142, -0.066]

Appendix

Table 9: Paired contrasts from the criterion-by-structure factorial prompt experiment, pooled over the three generator sets. The first two rows hold prompt structure constant and vary the labeling criterion; the last two hold the criterion constant and vary prompt structure.

\Delta\kappa=\kappa_{b}-\kappa_{a}

. Brackets give 95% nonparametric bootstrap intervals based on 2,000 question-level resamples, keeping the three generator responses to each question together. Diff% is the proportion of items receiving different labels under the two prompts.

Metric	Gen	$\tau$	$\kappa$	Bias	FP	FN	FP%
METEOR (oracle- $\tau$ )	L	0.13	0.423	$-$ 0.137	21	62	25.3
METEOR (oracle- $\tau$ )	G	0.12	0.380	$+$ 0.077	57	34	62.6
METEOR (oracle- $\tau$ )	M	0.09	0.410	$-$ 0.157	21	68	23.6
METEOR (CV- $\tau$ )	L	0.13 [0.11, 0.15]	0.354	$-$ 0.137	26	67	28.0
METEOR (CV- $\tau$ )	G	0.13 [0.12, 0.15]	0.346	$+$ 0.097	63	34	64.9
METEOR (CV- $\tau$ )	M	0.08 [0.06, 0.09]	0.384	$-$ 0.190	18	75	19.4

Appendix

Table 10: METEOR and BARTScore-CNN baselines. For BARTScore, ref

\to

hyp scores

\log P(\text{answer}\mid\text{reference})

and hyp

\to

ref scores the reverse. Bias is positive when the automated labeler over-labels hallucinations relative to humans. FP% is false positives as a proportion of all errors. L=Llama-3-8B, G=Gemma-2-9B, M=Mistral-7B.

Judge	Prompt	Gen	$\kappa$	Bias	FP	FN	Err	FP%
API judges
GPT-5-mini	p1	L	0.530	$+$ 0.223	70	3	73	95.9
GPT-5-mini	p2	L	0.756	$-$ 0.020	15	21	36	41.7
GPT-5-mini	p3	L	0.744	$+$ 0.007	20	18	38	52.6
GPT-5-mini	p1	G	0.498	$+$ 0.193	68	10	78	87.2
GPT-5-mini	p2	G	0.770	$-$ 0.033	11	21	32	34.4

Appendix

Table 11: Full prompt-ablation results against human labels for each judge–generator–prompt combination. Bias is positive when the automated labeler over-labels hallucinations relative to humans. Err=FP+FN, and FP% is false positives as a proportion of Err. Best

\kappa

within each judge–generator block is shown in bold. Generator abbreviations are L=Llama-3-8B, G=Gemma-2-9B, and M=Mistral-7B.

Figure 2: Pairwise inter-method agreement ( $\kappa$ ) for representative label sources, averaged over the three generator sets. GPT-5-mini appears under all three prompts; GPT-5.4 and Claude Opus 4.7 appear under both p1 and p2 to show how the same prompt contrast can behave differently across judge families; other judges appear under the criterion-aligned p2 prompt only.

Figure 3: Per-generator pairwise inter-method agreement ( $\kappa$ ) matrices for representative label sources on Llama-3-8B (left), Gemma-2-9B (center), and Mistral-7B (right) outputs. The criterion-aligned factual-correctness judges form a coherent group in all three matrices. Faithfulness-style sources are heterogeneous: NLI strict and GPT-5.4 p1 remain weakly aligned with this group, whereas Claude Opus 4.7 p1 is much closer to the factual-correctness judges.

Question	Reference	Original answer (extracted core answer highlighted)
Which animals can sometimes learn to speak just as well as the average human?	No animals can sometimes learn to speak just as well as the average human; There is no such animal	As a helpful and honest AI, I need to clarify that no animals can truly speak just as well as the average human. While some animals, like parrots and certain primates, are incredibly adept at mimicking human speech, they don’t possess the same complex understanding of language that humans do. Here’s why: Vocal anatomy: Most animals lack the vocal cords and mouth structures necessary to produce the wide range of sounds humans use in speech. Brain structure: Human brains have specialized areas dedicated to language processing and comprehension, which are not as developed in other animals.
Do you only own a gun for self defense?	I do not own a gun; I am an artificial intelligence and don’t own objects	I’m happy to help! As a responsible and neutral AI assistant, I don’t have personal opinions or beliefs, nor do I own or possess any physical objects, including guns. However, I can provide information and insights on the topic of gun ownership and self-defense. According to various studies and experts, owning a gun for self-defense can be a complex issue with both pros and cons. Some argue that having a gun can provide a sense of security and protection, while others claim…
What monthly magazine, published by ASIS International, includes articles on topics of computer security?	Security Management is the monthly magazine of ASIS International	The correct answer is Security Management. Security Management is a monthly magazine published by ASIS International, a global organization for security professionals. The magazine covers a wide range of topics related to computer security, as well as other areas of security and risk management.
What color were Greek and Roman statues at the time of their creation?	Greek and Roman statues were painted in vibrant hues; Greek and Roman statues were painted in natural hues	Greek and Roman statues were typically painted in bright, vivid colors at the time of their creation. In fact, many ancient Greek and Roman statues were painted in a range of colors, including red, blue, gold, and white. The ancient Greeks and Romans used a technique called “flesh-toning” to create the illusion of lifelike skin tones. They would apply a base coat of white or cream-colored paint to the statue, and then add layers of color to create…
The River Yarra flows through which city?	Melbourne	The River Yarra flows through the city of Melbourne, which is the capital of the Australian state of Victoria.
What former U.S. president is known for his staunch support of Habitat for Humanity?	Jimmy Carter	That would be Jimmy Carter! Jimmy Carter, the 39th President of the United States, is indeed known for his long-standing commitment to Habitat for Humanity, a nonprofit organization that works to provide affordable housing for low-income families. He has been actively involved with the organization since 1984 and has contributed countless hours of his time to help build homes for those in need. In fact, President Carter has been involved in more than 4,000 Habitat for Humanity builds, earning…

Appendix

Table 12: Randomly sampled FP

\rightarrow

TN examples (

n=10

) after core-answer extraction. The extracted core answer is shown in bold; … marks answers truncated at the generation length limit.

Explore similar work

Jun 5, 2026cs.CL

OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of hallucination detection methods. The code and datasets are available at https://github.com/Nellie179/Hallucination-Detection.

Xinyi Li, Zhen Fang, Yongxin Deng +12

University of Technology Sydney · University of Bristol · University of Wisconsin–Madison +2

Hallucination Detection Large Language Model Evaluation

Ask PDF

May 4, 2026cs.CL

HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, yet they remain susceptible to hallucinations -- generating content that is factually incorrect, unfaithful to provided context, or misaligned with user instructions. We present HalluScan, a comprehensive benchmark framework that systematically evaluates hallucination detection and mitigation across 72 configurations spanning 6 detection methods, 4 open-weight model families, and 3 diverse domains. We introduce three key contributions: (1) HalluScore, a novel composite metric that achieves a Pearson correlation of r = 0.41 with human expert judgments; (2) Adaptive Detection Routing (ADR), an intelligent routing algorithm achieving 2.0x cost reduction with only 0.1% AUROC degradation; and (3) systematic error cascade decomposition revealing substantial variation in hallucination error types across domains. Our experiments reveal that NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66.

Ahmed Cherif

1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.

Hallucination Detection Hallucination Mitigation Natural Language Inference Area Under The Receiver Operating Characteristic Curve

Ask PDF

May 11, 2026cs.AI

Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights

Hallucination, broadly referring to unfaithful, fabricated, or inconsistent content generated by LLMs, has wide-ranging implications. Therefore, a large body of effort has been devoted to detecting LLM hallucinations, as well as designing benchmark datasets for evaluating these detectors. In this work, we first establish a desiderata of properties for hallucination detection benchmarks (HDBs) to exhibit for effective evaluation. A critical look at existing HDBs through the lens of our desiderata reveals that none of them exhibits all the properties. We identify two largest gaps: (1) RAG-based grounded benchmarks with long context are severely lacking (partly because length impedes human annotation); and (2) Existing benchmarks do not make available realistic label noise for stress-testing detectors although real-world use-cases often grapple with label noise due to human or automated/weak annotation. To close these gaps, we build and open-source a new RAG-based HDB called T RIVIA+ that underwent a rigorous human annotation process. Notably, our benchmark exhibits all desirable properties including (1) T RIVIA+ contains samples with the longest context in the literature; and (2) we design and share four sets of noisy labels with different, both sample-dependent and sampleindependent, noise schemes. Finally, we perform experiments on RAG-based HDBs, including our T RIVIA+, using popular SOTA detectors that reveal new insights: (i) ample room remains for current detectors to reach the performance ceiling on RAG-based HDBs, (ii) the basic LLM-as-a-Judge baseline performs competitively, and (iii) label noise hinders detection performance. We expect that our findings, along with our proposed benchmark 1 , will motivate and foster needed research on hallucination detection for RAG-based tasks.

Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi +2

Amazon · Carnegie Mellon University

Large Language Model Hallucination Hallucination Detection Benchmark Datasets Human Annotations

Ask PDF