Organizations: New York University · University of Arizona · University of Illinois Urbana-Champaign · University of California, Irvine · Emory University · University of Maryland, College Park · University of Alabama at Birmingham
Large language models are widely used as judges of model outputs (LLM-as-a-Judge), but they are costly and can hallucinate or be overconfident. Decision models are a potential alternative. They return a predefined verdict with a probability for each label instead of free-form text and are much cheaper and faster. We test this promise on hallucination detection. We build binary benchmarks for five hallucination types (9,852 items from six datasets) and compare five decision models with the generative judge gpt-6-sol. Performance depends strongly on the type: the best decision model matches gpt-6-sol in balanced accuracy on context inconsistency and comes within 3 percentage points on factual errors, but trails it by 4-8 points on logical and instruction inconsistency. Decision-model probabilities are overconfident, though less so than the GPT-6 models, and separate correct from incorrect decisions only moderately. We also find that a confidence cascade of decision models improves on the best single model by 1.9-3.3 points on three types and matches or exceeds gpt-6-sol at 1-3% of its cost. Overall, decision models are a practical low-cost judge for some hallucination types but need to be validated before use.
Figures & tables
Hallucination type
Operational definition
Datasets
H
N
Total
Factual contradiction (FC)
Claims contradict verifiable real-world facts through incorrect entities or relations.
FAVABench, HADTest
344
1,277
1,621
Factual fabrication (FF)
Claims introduce fabricated, unverifiable, or unsupported facts or overclaims not established by real-world knowledge.
FAVABench, HADTest
339
1,277
1,616
Context inconsistency (CI)
Claims conflict with, or are unsupported by, the supplied context or retrieved evidence.
RAGTruth
943
1,732
2,675
Logical inconsistency (LI)
Claims or reasoning steps within the same response are mutually inconsistent.
ContraDoc, ROSCOE
538
1,108
1,646
Instruction inconsistency (II)
The response violates or fails to satisfy explicit instructions or constraints.
IF-RewardBench (English)
1,600
694
2,294
Total
3,764
6,088
9,852
Table 1: Hallucination-detection tasks and datasets. H marks samples that contain the hallucination and N do not.
Figure 1: The single-model baseline and the three combination rules. Models 1–3: top-3 decision models by BAcc per type. Solid/dashed boxes: always/conditionally called. (c) The third model breaks ties between the two cheaper ones. (d) Each stage defers its least-confident rk share; the last stage decides the rest.
Type (N)
Model
BAcc
Macro-F1
Cost
Latency
FC N=1,621
jev
75.01
74.96
0.0490
264.51
d1
67.78
68.98
0.0295
224.15
clef
72.12
73.95
0.2064
347.49
pplx
78.93
78.48
0.0159
284.23
gpt-6-luna
77.75
68.36
0.0817
236.82
gpt-6-sol
81.61
77.45
3.3320
1,825.76
Table 2: Single-judge results pooled by hallucination type. BAcc and Macro-F1 are percentages; Cost is the estimated total API cost for that type (USD), and Latency is mean per-call latency (ms).
Figure 2: Judging performance by hallucination type. Bars show the pooled BAcc of each judge per type, with 95% bootstrap confidence intervals. Hatched bars mark the generative model gpt-6-sol .
Figure 3: Confidence by model and hallucination type. (a) Overconfidence gap, i.e., mean confidence minus accuracy. (b) conf-AUROC; outlined values have a 95% CI that includes 0.5 .
Figure 4: Reliability diagrams by hallucination type. Confidence is divided into 10 equal-width bins (bins with fewer than 25 items omitted); points below the diagonal indicate overconfidence. Right: ECE and conf-AUROC (cAUC) averaged over datasets, and the share of all errors (pooled over datasets) made with c≥0.9 (Err ≥ .9).
Figure 5: BAcc against estimated cost for each hallucination type. Cost is relative to gpt-6-sol on a log scale, and the dashed line marks the BAcc of gpt-6-sol . The cascade uses the deferral rates with the highest BAcc. The circle defers the same numbers of items at random.
Dataset
Version / source snapshot
Split and selection
Instances
FAVABench
annotations.json ; SHA ef3b4b289762
All 460 human-annotated passages; separate inclusive FC and FF views.
FC: 377 FF: 351
HADTest
hadtest.json ; SHA fd13225bcc8d
Human-validated HADTest split only; HAD train and dev are not used.
FC: 1,244 FF: 1,265
RAGTruth
Revision c103204b9ce2
Official test split, with quality="good" ; join responses to source context by source_id .
2,675
ContraDoc
Revision d1c42d1e4335
Entire released pos / neg benchmark; these are label partitions, not train/test splits.
891
ROSCOE
Official annotations.zip ; SHA 1fe5b4ee56a6
All released CosmosQA, DROP, e-SNLI, and GSM8K annotations; exclude one empty chain.
755
IF-RewardBench
Revision c192fe968c15
English-filtered released benchmark, including all three instruction forms.
2,294
Table 4: Dataset versions and evaluation subsets. Counts are type-specific evaluation instances. No class balancing or additional train/test split is applied to these judge evaluations.
Figure 6: Mapping of source-dataset annotations to binary labels. H : the target error is present; N : it is absent. The bar shows each dataset’s H:N ratio.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10
Attribute (dataset, type)
Level
n
pplx
jev
gpt-6-luna
gpt-6-sol
Error density
Annotated spans (RAGTruth, CI)
1
590
57
71
97
91
2
206
81
90
99
99
≥ 3
147
96
98
100
99
Violated constraints (IF-RewardBench, II)
1
764
46
73
95
76
2
465
68
83
98
91
Appendix
Table 9: Where judges fail. Rows under Error density and Salience report recall of H (%) on the H items with the given attribute. Rows under Task form and Generator report BAcc (%) on all items with the attribute. Attributes come from the source annotations and were not shown to the judges.
Figure 7: Confidence distributions of correct decisions and errors. For each model, the share of correct decisions (gray bars) and of errors (red outline) in each confidence bin. The results are averaged over the five hallucination types, with each dataset weighted equally within its type.
Figure 8: Cascade performance across deferral rates. The x-axis varies r1 ; each colored line fixes r2 . Both rates range from 0% to 50% in 10-percentage-point steps (36 configurations per type). Stars mark the highest BAcc, dashed lines mark gpt-6-sol, and dotted lines mark the strongest member alone. Members appear in stage order below each panel. Peaks are selected and evaluated on the same items; they are post-hoc maxima, not held-out results.
Large language models generate fluent text that can contain unfaithful claims -- a phenomenon known as hallucination. We present a multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo (MC) Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection. Evaluated on the HaluEval benchmark, our pipeline achieves F1=0.915 and AUROC=0.977 on general-domain tasks, with per-task F1 scores of 0.97 (QA), 0.96 (Summarization), and 0.82 (Dialogue). MC Dropout inference further improves accuracy to 93.2%. A context ablation study confirms the model performs genuine entailment reasoning rather than exploiting surface patterns, with summarization F1 dropping 24% when knowledge context is removed. Learning curve analysis reveals that 25% of training data captures 77% of full-data performance. Beyond detection, we apply Direct Preference Optimization (DPO) to a Qwen2.5-0.5B generator, reducing its hallucination rate from 85.5% to 37.7% (55.9% relative reduction) as measured by our detector. Cross-domain evaluation on the SciFact biomedical benchmark shows that general-domain training transfers poorly (F1=0.52), motivating domain-specific fine-tuning. PubMedBERT fine-tuned on SciFact achieves F1=0.63 and AUROC=0.81, demonstrating that domain-matched pre-training is the strongest adaptation strategy. Code and models are available at https://github.com/varunteja99/hallucination-detection-nlp
Varun Teja Chundru, Debasmita Biswas
Department of Computer Science Purdue University Fort Wayne
Large language models (LLMs) hallucinate with confidence: their outputs can be fluent, authoritative, and simply wrong. In medical, legal, and scientific applications this failure causes direct harm, and detecting it from internal model states offers a path to safer deployment. A growing body of work reports that this problem is increasingly tractable, with recent methods achieving high detection performance on widely used benchmarks. We show, however, that much of this apparent progress does not survive scrutiny. Four of the six corpora embed the ground-truth answer directly in the input prompt. A naïve text-similarity baseline we call TxTemb exploits this to achieve near-perfect detection scores without any access to model internals. To measure what genuine detection capability remains once these artifacts are controlled, we conduct a large-scale evaluation spanning twenty-two detection methods, twelve open-source models spanning six architectural families, and six corpora. We further introduce DRIFT, a supervised probe over inter-layer hidden-state transitions, as a point of comparison for live-generation detection. Our findings suggest that the field's reported progress on hallucination detection is substantially explained by benchmark construction artifacts in widely used corpora, and that the majority of established baselines perform near chance under controlled conditions; the consistent exceptions are SAPLMA and DRIFT, both supervised probes on upper-layer hidden states.
Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of hallucination detection methods. The code and datasets are available at https://github.com/Nellie179/Hallucination-Detection.
Xinyi Li, Zhen Fang, Yongxin Deng +12
University of Technology Sydney · University of Bristol · University of Wisconsin–Madison +2