How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

JV Roig

cs.CLMar 9, 2026

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

Authors: JV Roig

Organizations: Kamiwaza AI

Abstract

How much do large language models actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to contamination, LLM-based judges with documented biases, or evaluation scales too small for statistical confidence. We address this gap using RIKER, a ground-truth-first evaluation methodology that enables deterministic scoring without human annotation. Across 35 open-weight models, three context lengths (32K, 128K, and 200K tokens), four temperature settings, and three hardware platforms (NVIDIA H200, AMD MI300X, and Intel Gaudi 3), we conducted over 172 billion tokens of evaluation - an order of magnitude beyond prior work. Our findings reveal that: (1) even the best-performing models fabricate answers at a non-trivial rate - 1.19% at best at 32K, with top-tier models at 5 - 7% - and fabrication rises steeply with context length, nearly tripling at 128K and exceeding 10% for all models at 200K; (2) model selection dominates all other factors, with overall accuracy spanning a 72-percentage-point range and model family predicting fabrication resistance better than model size; (3) temperature effects are nuanced - T=0.0 yields the best overall accuracy in roughly 60% of cases, but higher temperatures reduce fabrication for the majority of models and dramatically reduce coherence loss (infinite generation loops), which can reach 48x higher rates at T=0.0 versus T=1.0; (4) grounding ability and fabrication resistance are distinct capabilities - models that excel at finding facts may still fabricate facts that do not exist; and (5) results are consistent across hardware platforms, confirming that deployment decisions need not be hardware-dependent.

Figures & tables

Figure 1: RIKER methodology overview. Traditional approaches (left) extract ground truth from existing documents via expensive human annotation, producing static benchmarks vulnerable to contamination and requiring biased LLM judges. RIKER (right) inverts this: structured ground truth is defined first, then documents and questions are generated from it, enabling deterministic scoring and regenerable corpora. (Figure reproduced from Roig (2026) .)

Context	Models	Total Runs	Input Tokens	Output Tokens	Total Tokens
32K	35	1,856	17.98B	181M	18.16B
128K	26	1,480	71.99B	510M	72.50B
200K	11	928	80.68B	418M	81.09B
Total	35	4,264	170.65B	1,109M	171.76B

Table 1: Experimental Scale

Model	32K	128K	200K
DeepSeek V3	✓	✓	–
DeepSeek V3.1	✓	✓	–
GLM 4.5	✓	✓	–
GLM 4.5 Air	✓	✓	–
GLM 4.6	✓	✓	✓
GLM 4.7	✓	✓	✓

Table 2: Model–Context Length Matrix

	Overall Accuracy (%)			Fabrication Rate (%)
Model	H200	MI300X	$\Delta$	H200	MI300X	$\Delta$
GLM 4.5	97.01	97.40	0.39	1.59	1.88	0.30
Qwen3 Coder 480B-A35B	89.80	90.54	0.75	14.93	13.00	1.93
Qwen3 235B-A22B	89.26	90.20	0.94	22.82	20.24	2.58
Qwen3 235B-A22B FP8	87.97	88.21	0.24	25.25	24.31	0.94
GLM 4.6	92.81	93.26	0.46	8.68	7.04	1.64

Table 3: Cross-Platform Comparison at 32K Context, Temperature 0.0

Model	Hardware $\Delta$	Temp Spread	Ratio
GLM 4.5	0.39	1.28	3.3 $\times$
Qwen3 Coder 480B-A35B	0.75	2.59	3.5 $\times$
Qwen3 235B-A22B	0.94	1.25	1.3 $\times$
GLM 4.6	0.46	2.32	5.0 $\times$
GLM 4.5 Air	0.91	3.62	4.0 $\times$
DeepSeek V3	0.34	3.20	9.4 $\times$

Table 4: Hardware Delta vs. Temperature Spread at 32K Context

Model	H200	MI300X	Gaudi3	Max $\Delta$
Qwen3 235B-A22B	88.80	88.95	88.68	0.27
Qwen2.5 72B Instruct	—	80.29	80.63	0.34
Qwen3 30B-A3B	—	76.85	77.09	0.24
Qwen3 Coder 30B-A3B	—	77.19	76.38	0.81
Qwen2.5 Coder 7B	—	50.47	50.27	0.20
Llama 3.1 8B Instruct	—	48.01	48.92	0.91

Table 5: Three-Platform Comparison at 32K Context, Temperature 1.0

#	Model	Overall	Faith.	Ground.	Fab. $\downarrow$	Agg.
1	GLM 4.5	97.40	97.92	97.72	1.88	96.32
2	MiniMax M2.1	95.96	96.48	98.21	5.26	94.85
3	DeepSeek V3.1	95.49	95.44	97.22	6.35	95.59
4	MiniMax M2	95.25	94.99	96.63	6.65	95.80
5	Qwen3 Next 80B-A3B	93.87	93.70	94.44	7.04	94.22
6	GLM 4.6	93.26	93.50	94.05	7.04	92.75

Table 6: Model Performance at 32K Context (Best Temperature per Model)

Rank	Model	32K%	128K%	$\Delta$	Fab 32K	Fab 128K
1	DeepSeek V3.1	95.49	90.45	$-$ 5.04	6.35	7.71
2	MiniMax M2	95.25	89.27	$-$ 5.98	6.65	8.47
3	Qwen3 Next 80B-A3B	93.87	87.85	$-$ 6.02	7.04	7.99
4	GLM 4.5	97.40	87.43	$-$ 9.97	1.29	3.19
5	GLM 4.6	93.26	85.81	$-$ 7.45	7.04	13.75
6	MiniMax M2.1	95.96	85.59	$-$ 10.37	5.06	9.72

Table 7: Performance Degradation from 32K to 128K Context (26 Models)

Rank	Model	32K%	128K%	200K%	$\Delta_{32\rightarrow 200}$	Fab 200K
1	Qwen3 Next 80B-A3B	93.87	87.85	82.68	$-$ 11.19	11.21
2	Qwen3 Coder 480B-A35B	91.21	80.04	71.67	$-$ 19.54	27.58
3	Qwen3 235B-A22B	90.20	79.90	69.93	$-$ 20.27	35.68
4	Qwen3 235B-A22B FP8	88.51	79.79	69.52	$-$ 18.99	34.66
5	Qwen3 30B-A3B 2507	77.97	69.91	65.45	$-$ 12.52	41.04
6	Llama 4 Maverick	86.52	63.90	61.56	$-$ 24.96	43.29

Table 8: Performance Across All Three Context Lengths (11 Models)

Rank	Model	32K	128K	200K	$\Delta_{32\rightarrow\text{max}}$
1	GLM 4.5	1.19	3.19	—	$+$ 2.00
2	GLM 4.5 Air	3.37	12.43	—	$+$ 9.06
3	MiniMax M2.1	5.06	9.72	—	$+$ 4.66
4	DeepSeek V3.1	6.35	7.36	—	$+$ 1.01
5	MiniMax M2	6.55	7.22	—	$+$ 0.67
6	GLM 4.6	7.04	13.75	69.53	$+$ 62.49

Table 9: Best-Case Fabrication Rates Across Context Lengths (

\downarrow

lower is better)

Metric	Higher Temp Best	T=0.0 Best
Overall Accuracy	36 (40%)	54 (60%)
Faithfulness	41 (46%)	49 (54%)
Grounding	48 (53%)	42 (47%)
Fabrication ∗	48 (53%)	42 (47%)
Aggregation	39 (43%)	51 (57%)

Table 10: Temperature Preference by Metric (90 Model–Context–Hardware Combinations)

Model	Ctx	T=0.0	T=0.4	T=0.7	T=1.0	Ratio
Llama 3.1 8B	128K	14.05	10.64	6.34	2.05	6.9 $\times$
Qwen3 4B 2507	200K	13.29	12.41	11.10	8.44	1.6 $\times$
Granite 4.0 H Tiny	128K	5.07	4.02	2.21	0.33	15.6 $\times$
Qwen3 Next 80B-A3B	200K	4.80	4.21	3.00	1.82	2.6 $\times$
GLM 4.6	200K	4.64	3.79	2.15	0.14	34.4 $\times$
Llama 3.2 3B	32K	3.23	1.65	1.18	0.20	16.0 $\times$

Table 11: Coherence Loss Rate (%) by Temperature (Models with Significant Effects)

Well-calibrated (Gap $<$ +5)
Model	Grounding	Fabrication ( $\downarrow$ )	Gap
GLM 4.5	97.72	1.19	$-$ 1.1
MiniMax M2.1	98.21	5.06	$+$ 3.3
MiniMax M2	97.32	6.55	$+$ 3.9
GLM 4.6	94.84	7.04	$+$ 1.9
Qwen3 Next 80B-A3B	94.64	7.04	$+$ 1.7

Table 12: Grounding vs. Fabrication: Best-Case Scores at 32K Context

Figure 2: Grounding accuracy vs. fabrication rate at 32K context. Each model is shown at its independently optimal temperature for each metric. The dashed line shows the expected inverse relationship; deviation above the line indicates excess fabrication. Color and shape indicate tier: well-calibrated (green circles, Gap $<$ +5), mid-tier (orange squares), and fabrication-prone (red triangles, Gap $\geq$ +15).

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

Feb 23, 2026

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Large Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible language. However, coherence does not equate to truthfulness, as the responses often contain subtle hallucinations. Existing benchmarks are limited by static and narrow questions, leading to limited coverage and misleading evaluations. We present KGHaluBench, a Knowledge Graph-based hallucination benchmark that assesses LLMs across the breadth and depth of their knowledge, providing a fairer and more comprehensive insight into LLM truthfulness. Our framework utilises the KG to dynamically construct challenging, multifaceted questions, whose difficulty is then statistically estimated to address popularity bias. Our automated verification pipeline detects abstentions and verifies the LLM's response at both conceptual and correctness levels to identify different types of hallucinations. We evaluate 25 frontier models, using novel accuracy and hallucination metrics. The results provide a more interpretable insight into the knowledge factors that cause hallucinations across different model sizes. KGHaluBench is publicly available to support future developments in hallucination mitigation.

Alex Robertson, Huizhi Liang, Mahbub Gani +2

School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis

Hallucination in Language Models LLM Evaluation LLM Hallucination Detection

Ask PDF

Aug 10, 2026

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturate".

Siyang Wu, Yibo Jiang, Bryon Aragam

Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA

Prompt Sensitivity LLM Evaluation Language Model Generation Evaluation Reasoning Consistency in Language Models LLM Reliability Cross-Lingual Consistency in Language Models

Ask PDF

May 21, 2026

Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer

Hallucination is often viewed as a direct consequence of missing knowledge: a model answers incorrectly when the correct answer is absent from its generation-time distribution, and correctly when it is present. We test this assumption by introducing a semantic notion of answer availability that aggregates token-level variants expressing the same answer concept, and asks whether the correct concept is already available at the moment the model commits to an answer. Across Qwen and Llama models from 0.8B to 72B in both Instruct and Base variants, 16-47% of Instruct hallucinations occur with substantial probability mass already on the correct concept, and the rate rises monotonically with scale. Comparing such failures against correct generations with matched semantic support, the distinguishing factor is not whether the correct concept is represented, but how its probability is distributed: correct generations concentrate mass on a single surface form, hallucinations disperse it across alternatives. The same sharpening asymmetry extends across multi-token generation and is detectable in pre-generation hidden states. Together, these results identify a single mechanism: instruction tuning sharpens answer commitment with scale, making helpfulness and confident hallucination two consequences of the same underlying disposition.

Jewon Yeom, Jaewon Sok, Heejun Kim +3

Graduate School of Data Science, Seoul National University · Department of Rural Systems Engineering, Seoul National University · Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology +1

LLM Interpretability Hallucination in Language Models LLM Reliability Instruction Tuning

Ask PDF

Abstract

Hallucination in Language Models

Explore similar work

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer