How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

JV Roig

cs.CLMar 9, 2026

How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

Authors: JV Roig

Organizations: Kamiwaza AI

Abstract

How much do large language models actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to contamination, LLM-based judges with documented biases, or evaluation scales too small for statistical confidence. We address this gap using RIKER, a ground-truth-first evaluation methodology that enables deterministic scoring without human annotation. Across 35 open-weight models, three context lengths (32K, 128K, and 200K tokens), four temperature settings, and three hardware platforms (NVIDIA H200, AMD MI300X, and Intel Gaudi 3), we conducted over 172 billion tokens of evaluation - an order of magnitude beyond prior work. Our findings reveal that: (1) even the best-performing models fabricate answers at a non-trivial rate - 1.19% at best at 32K, with top-tier models at 5 - 7% - and fabrication rises steeply with context length, nearly tripling at 128K and exceeding 10% for all models at 200K; (2) model selection dominates all other factors, with overall accuracy spanning a 72-percentage-point range and model family predicting fabrication resistance better than model size; (3) temperature effects are nuanced - T=0.0 yields the best overall accuracy in roughly 60% of cases, but higher temperatures reduce fabrication for the majority of models and dramatically reduce coherence loss (infinite generation loops), which can reach 48x higher rates at T=0.0 versus T=1.0; (4) grounding ability and fabrication resistance are distinct capabilities - models that excel at finding facts may still fabricate facts that do not exist; and (5) results are consistent across hardware platforms, confirming that deployment decisions need not be hardware-dependent.

Figures & tables

Figure 1: RIKER methodology overview. Traditional approaches (left) extract ground truth from existing documents via expensive human annotation, producing static benchmarks vulnerable to contamination and requiring biased LLM judges. RIKER (right) inverts this: structured ground truth is defined first, then documents and questions are generated from it, enabling deterministic scoring and regenerable corpora. (Figure reproduced from Roig (2026) .)

Context	Models	Total Runs	Input Tokens	Output Tokens	Total Tokens
32K	35	1,856	17.98B	181M	18.16B
128K	26	1,480	71.99B	510M	72.50B
200K	11	928	80.68B	418M	81.09B
Total	35	4,264	170.65B	1,109M	171.76B

Table 1: Experimental Scale

Model	32K	128K	200K
DeepSeek V3	✓	✓	–
DeepSeek V3.1	✓	✓	–
GLM 4.5	✓	✓	–
GLM 4.5 Air	✓	✓	–
GLM 4.6	✓	✓	✓
GLM 4.7	✓	✓	✓

Table 2: Model–Context Length Matrix

	Overall Accuracy (%)			Fabrication Rate (%)
Model	H200	MI300X	$\Delta$	H200	MI300X	$\Delta$
GLM 4.5	97.01	97.40	0.39	1.59	1.88	0.30
Qwen3 Coder 480B-A35B	89.80	90.54	0.75	14.93	13.00	1.93
Qwen3 235B-A22B	89.26	90.20	0.94	22.82	20.24	2.58
Qwen3 235B-A22B FP8	87.97	88.21	0.24	25.25	24.31	0.94
GLM 4.6	92.81	93.26	0.46	8.68	7.04	1.64

Table 3: Cross-Platform Comparison at 32K Context, Temperature 0.0

Model	Hardware $\Delta$	Temp Spread	Ratio
GLM 4.5	0.39	1.28	3.3 $\times$
Qwen3 Coder 480B-A35B	0.75	2.59	3.5 $\times$
Qwen3 235B-A22B	0.94	1.25	1.3 $\times$
GLM 4.6	0.46	2.32	5.0 $\times$
GLM 4.5 Air	0.91	3.62	4.0 $\times$
DeepSeek V3	0.34	3.20	9.4 $\times$

Table 4: Hardware Delta vs. Temperature Spread at 32K Context

Model	H200	MI300X	Gaudi3	Max $\Delta$
Qwen3 235B-A22B	88.80	88.95	88.68	0.27
Qwen2.5 72B Instruct	—	80.29	80.63	0.34
Qwen3 30B-A3B	—	76.85	77.09	0.24
Qwen3 Coder 30B-A3B	—	77.19	76.38	0.81
Qwen2.5 Coder 7B	—	50.47	50.27	0.20
Llama 3.1 8B Instruct	—	48.01	48.92	0.91

Table 5: Three-Platform Comparison at 32K Context, Temperature 1.0

#	Model	Overall	Faith.	Ground.	Fab. $\downarrow$	Agg.
1	GLM 4.5	97.40	97.92	97.72	1.88	96.32
2	MiniMax M2.1	95.96	96.48	98.21	5.26	94.85
3	DeepSeek V3.1	95.49	95.44	97.22	6.35	95.59
4	MiniMax M2	95.25	94.99	96.63	6.65	95.80
5	Qwen3 Next 80B-A3B	93.87	93.70	94.44	7.04	94.22
6	GLM 4.6	93.26	93.50	94.05	7.04	92.75

Table 6: Model Performance at 32K Context (Best Temperature per Model)

Rank	Model	32K%	128K%	$\Delta$	Fab 32K	Fab 128K
1	DeepSeek V3.1	95.49	90.45	$-$ 5.04	6.35	7.71
2	MiniMax M2	95.25	89.27	$-$ 5.98	6.65	8.47
3	Qwen3 Next 80B-A3B	93.87	87.85	$-$ 6.02	7.04	7.99
4	GLM 4.5	97.40	87.43	$-$ 9.97	1.29	3.19
5	GLM 4.6	93.26	85.81	$-$ 7.45	7.04	13.75
6	MiniMax M2.1	95.96	85.59	$-$ 10.37	5.06	9.72

Table 7: Performance Degradation from 32K to 128K Context (26 Models)

Rank	Model	32K%	128K%	200K%	$\Delta_{32\rightarrow 200}$	Fab 200K
1	Qwen3 Next 80B-A3B	93.87	87.85	82.68	$-$ 11.19	11.21
2	Qwen3 Coder 480B-A35B	91.21	80.04	71.67	$-$ 19.54	27.58
3	Qwen3 235B-A22B	90.20	79.90	69.93	$-$ 20.27	35.68
4	Qwen3 235B-A22B FP8	88.51	79.79	69.52	$-$ 18.99	34.66
5	Qwen3 30B-A3B 2507	77.97	69.91	65.45	$-$ 12.52	41.04
6	Llama 4 Maverick	86.52	63.90	61.56	$-$ 24.96	43.29

Table 8: Performance Across All Three Context Lengths (11 Models)

Rank	Model	32K	128K	200K	$\Delta_{32\rightarrow\text{max}}$
1	GLM 4.5	1.19	3.19	—	$+$ 2.00
2	GLM 4.5 Air	3.37	12.43	—	$+$ 9.06
3	MiniMax M2.1	5.06	9.72	—	$+$ 4.66
4	DeepSeek V3.1	6.35	7.36	—	$+$ 1.01
5	MiniMax M2	6.55	7.22	—	$+$ 0.67
6	GLM 4.6	7.04	13.75	69.53	$+$ 62.49

Table 9: Best-Case Fabrication Rates Across Context Lengths (

\downarrow

lower is better)

Metric	Higher Temp Best	T=0.0 Best
Overall Accuracy	36 (40%)	54 (60%)
Faithfulness	41 (46%)	49 (54%)
Grounding	48 (53%)	42 (47%)
Fabrication ∗	48 (53%)	42 (47%)
Aggregation	39 (43%)	51 (57%)

Table 10: Temperature Preference by Metric (90 Model–Context–Hardware Combinations)

Model	Ctx	T=0.0	T=0.4	T=0.7	T=1.0	Ratio
Llama 3.1 8B	128K	14.05	10.64	6.34	2.05	6.9 $\times$
Qwen3 4B 2507	200K	13.29	12.41	11.10	8.44	1.6 $\times$
Granite 4.0 H Tiny	128K	5.07	4.02	2.21	0.33	15.6 $\times$
Qwen3 Next 80B-A3B	200K	4.80	4.21	3.00	1.82	2.6 $\times$
GLM 4.6	200K	4.64	3.79	2.15	0.14	34.4 $\times$
Llama 3.2 3B	32K	3.23	1.65	1.18	0.20	16.0 $\times$

Table 11: Coherence Loss Rate (%) by Temperature (Models with Significant Effects)

Well-calibrated (Gap $<$ +5)
Model	Grounding	Fabrication ( $\downarrow$ )	Gap
GLM 4.5	97.72	1.19	$-$ 1.1
MiniMax M2.1	98.21	5.06	$+$ 3.3
MiniMax M2	97.32	6.55	$+$ 3.9
GLM 4.6	94.84	7.04	$+$ 1.9
Qwen3 Next 80B-A3B	94.64	7.04	$+$ 1.7

Table 12: Grounding vs. Fabrication: Best-Case Scores at 32K Context

Figure 2: Grounding accuracy vs. fabrication rate at 32K context. Each model is shown at its independently optimal temperature for each metric. The dashed line shows the expected inverse relationship; deviation above the line indicates excess fabrication. Color and shape indicate tier: well-calibrated (green circles, Gap $<$ +5), mid-tier (orange squares), and fabrication-prone (red triangles, Gap $\geq$ +15).

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 254 papers

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Feb 23, 2026Alex Robertson, Huizhi Liang, Mahbub Gani +2Hallucination in Language Models LLM Evaluation
School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer
May 21, 2026Jewon Yeom, Jaewon Sok, Heejun Kim +3LLM Interpretability Hallucination in Language Models
Graduate School of Data Science, Seoul National University · Department of Rural Systems Engineering, Seoul National University · Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology +1

Feb 23, 2026J/K move · Enter open · S save

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Alex Robertson, Huizhi Liang, Mahbub Gani, Rohit Kumar+1

School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis

Hallucination in Language Models LLM Evaluation LLM Hallucination Detection

Open paper AskPDF arXiv

Large Language Models (LLMs) possess a remarkable capacity to generate persuasive and intelligible language. However, coherence does not equate to truthfulness, as the responses often contain subtle hallucinations. Existing benchmarks are limited by static and narrow questions, leading to limited coverage and misleading evaluations. We present KGHaluBench, a Knowledge Graph-based hallucination benchmark that assesses LLMs across the breadth and depth of their knowledge, providing a fairer and more comprehensive insight into LLM truthfulness. Our framework utilises the KG to dynamically construct challenging, multifaceted questions, whose difficulty is then statistically estimated to address popularity bias. Our automated verification pipeline detects abstentions and verifies the LLM's response at both conceptual and correctness levels to identify different types of hallucinations. We evaluate 25 frontier models, using novel accuracy and hallucination metrics. The results provide a more interpretable insight into the knowledge factors that cause hallucinations across different model sizes. KGHaluBench is publicly available to support future developments in hallucination mitigation.

Abstract

Hallucination in Language Models

Explore similar work

KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answer