The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models

Heimo Müller; Dominik Steiger; Markus Plass; Andreas Holzinger

cs.CLFeb 13, 2026

The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models

Authors: Heimo Müller, Dominik Steiger, Markus Plass, Andreas Holzinger

Organizations: Machine Learning and Information Science Group, Medical University of Graz, Austria · Human Machine Mind Cooperation, Graz, Austria · MIDATA Cooperative, Zurich, Switzerland · Human-Centered AI Lab, BOKU University Vienna, Austria

Abstract

Large language models routinely produce factually incorrect or misleading outputs, yet conventional automated benchmarks overlook how users perceive reliability during interaction. We introduce the System Hallucination Scale (SHS), a lightweight instrument modeling the famous System Usability Scale (SUS) and System Causability Scales (SCS). SHS measures perceived reliability across five dimensions: factual accuracy, source reliability, logical coherence, deceptiveness of presentation, and responsiveness to user guidance, using ten balanced Likert-scale items that provide a directional score and consistency diagnostic. In a supervised validation study with 193 participants (386 evaluations across 47 trained experimenters), SHS demonstrated strong internal consistency and high clarity, completing in an average of 4.2 minutes. Participants rated source verifiability and presentation deceptiveness worse than factual accuracy, indicating perceived vulnerabilities lie primarily in traceability and framing. As a proof of concept, SHS assesses human-perceived reliability rather than ground-truth correctness, providing open-source multilingual implementations and an interactive calculator.

Figures & tables

Dimension	Pol.	Item
Factual Accuracy	$+$	Q1. The response was factually reliable.
	$-$	Q2. The LLM frequently generated false or fabricated information.
Source Reliability	$+$	Q3. It was easy to find and verify the sources of the presented information.
	$-$	Q4. The LLM often omitted sources or invented them, and it was difficult to recognize what was real.
Logical Coherence	$+$	Q5. The LLM’s reasoning was logically structured and supported by facts.
	$-$	Q6. The LLM’s reasoning contained unfounded or illogical steps.

Table 1: The ten items of the System Hallucination Scale (SHS), grouped by dimension. Each dimension is measured by one positively worded (

+

) and one negatively worded (

-

) item; this bipolar pairing reduces acquiescence bias and supports the consistency diagnostic ( Brooke 1996 ; Lewis 2018 ) .

SHS	SHS 100	Interpretation
$[+0.5,+1.0]$	$[75,100]$	High perceived reliability
$[+0.0,+0.5)$	$[50,75)$	Moderate perceived reliability; some concerns
$[-0.5,+0.0)$	$[25,50)$	Low perceived reliability; caution advised
$[-1.0,-0.5)$	$[0,25)$	Very low perceived reliability

Table 2: Interpretation guidelines for SHS scores. Bands describe perceived reliability under the prompting regime and rater population in which the score was obtained; they are not statements about objective model reliability.

System (model, version)	Access mode	Collection window	Evaluations
ChatGPT (OpenAI, GPT-4-based)	Free-tier web interface	Oct–Nov 2025	68
Claude (Anthropic, Claude 3 Sonnet)	Free-tier web interface (Claude.ai)	Oct–Nov 2025	85
Gemini (Google)	Free-tier web interface	Oct–Nov 2025	78
Mistral (Mistral AI)	Public inference platform	Oct–Nov 2025	79
DeepSeek (DeepSeek AI)	Public inference platform	Oct–Nov 2025	76
Total			386

Table 3: Systems evaluated in the supervised validation study. Each of

N=193

participants evaluated two of the five systems (random assignment), yielding 386 SHS evaluations in total.

Category	Purpose	Dimensions
Verified Knowledge (“Wissen”)	Objectively checkable prompts on topics the participant knew or could easily verify, establishing a correctness baseline the participant could adjudicate.	FA, SR
“Glatteis” (Trick)	Prompts referencing non-existent people or entities, impossible events, fabricated facts, or category errors, constructed so that hallucinating an answer (rather than flagging the false premise) indicates problematic behavior.	DP, LC, RG

Table 4: Structure of the two-category prompt protocol used in every LLM interaction, and the SHS dimensions each category primarily targets. Participants composed their own items within each category (minimum three per category per interaction). Illustrative items are participants’ verbatim prompts, translated from German; the complete item set (German originals and English translations), together with the instructions and worked examples given to administrators, is deposited in the project repository. FA = Factual Accuracy, SR = Source Reliability, LC = Logical Coherence, DP = Deceptiveness, RG = Responsiveness to Guidance.

Evaluation aspect	Most frequent response	Percentage
Clarity of SHS items	Yes	87.2%
Relevance for LLM evaluation	Yes	83.0%
Appropriateness of response options	Yes	93.6%
No explanation required	No, never	66.0%
Demographic questions length	Exactly right	97.9%

Table 5: Acceptance of the SHS instrument by trained administrators (

n=47

experimenters; all percentages are of the 47 administrators).

Figure 1: Perceived clarity of the SHS items. Distribution of administrators’ ratings (

n=47

) of whether the SHS items were understandable for participants. A large majority (87.2%) reported the items were understandable and the remainder (12.8%) noted only minor limitations; no administrator rated the items neutral or unclear.

Figure 2: Perceived relevance of the SHS items. Distribution of administrators’ ratings (

n=47

) of whether the SHS items are relevant for evaluating LLM output. Most (83.0%) rated them relevant and a further 14.9% relevant with limitations; a single rating (2.1%) was neutral and none negative.

Figure 3: Appropriateness of the response options. Distribution of administrators’ ratings (

n=47

) of whether the five-point Likert response options were appropriate. The overwhelming majority (93.6%) found them suitable for expressing their judgments, with only isolated reservations.

Figure 4: Need for additional explanation. Distribution of administrators’ reports (

n=47

) of whether they had to explain item wording to participants. Two thirds (66.0%) never needed to, 31.9% did so occasionally, and only one (2.1%) needed to do so for most follow-up questions, indicating that the items are largely self-explanatory.

Figure 5: Length of the demographic section. Distribution of administrators’ ratings (

n=47

) of the demographic questions. Nearly all (97.9%) judged the section appropriately sized, supporting the overall brevity of the protocol.

	FA	SR	LC	DP	RG
Factual Accuracy (FA)	1.00
Source Reliability (SR)	0.72	1.00
Logical Coherence (LC)	0.68	0.61	1.00
Deceptiveness (DP)	0.54	0.49	0.57	1.00
Responsiveness (RG)	0.45	0.42	0.51	0.48	1.00

Table 6: Pearson correlations between SHS dimension scores (

N=386

evaluations); all coefficients significant at

p<.001

Figure 6: SHS dimension profile under the hallucination-eliciting prompt protocol (

N=386

evaluations). Bars show mean dimension scores

s_{i}

on the symmetric

[-1,+1]

scale; whiskers are

\pm 1

SD; the dashed line marks the overall SHS mean of

0.24

. Source Reliability and Deceptiveness of Presentation (darker bars) fall well below the other three dimensions, locating the perceived weakness in traceability and framing rather than in raw correctness. Higher is better throughout.

System	Tech.	UX	Auto.	Human	Dims.	Primary focus
TruthfulQA	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Truthfulness
HaluEval	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Binary detection
FEVER	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Claim verification
FActScore	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Atomic precision
SelfCheckGPT	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Consistency-based
Semantic entropy	$\bullet$	$\circ$	$\bullet$	$\circ$	1	Meaning-level uncertainty

Table 7: Hallucination evaluation systems: technical versus user-experience focus.

\bullet

= primary focus;

\circ

= secondary/not addressed.

Characteristic	SUS	SCS	SHS
Number of items	10	10	10
Response scale	5-pt Likert	5-pt Likert	5-pt Likert
Item structure	Alternating $\pm$	Alternating $\pm$	Paired $\pm$ per dimension
Score range (native)	0–100	0–100	$[-1,+1]$
Score range (rescaled)	—	—	0–100
Dimensions	1	1	5

Table 8: Structural comparison of SHS with established human-centered scales.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Feb 23, 2026Alex Robertson, Huizhi Liang, Mahbub Gani +2Hallucination in Language Models LLM Evaluation
School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis
Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models
Oct 19, 2026Osama Shabih, Dayam Nadeem, Shahab Saquib Sohail +1Hallucination in Language Models LLM Reliability
Department of Computer Science and Engineering, SEST, Jamia Hamdard, New Delhi, India · Department of Computer Science, Aligarh Muslim University, Aligarh, India

Mar 5, 2026J/K move · Enter open · S save

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong+1

RAND Corporation, Santa Monica, CA, USA

LLM-as-a-Judge LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

Abstract

LLM Evaluation

Explore similar work

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models