Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading or false information that appears highly credible. In this study, we explore the phenomenon of hallucinations across multiple languages in freeform text generation, focusing on what we call multilingual hallucination gaps. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. To quantify such hallucinations, we used the FactScore metric and extended its framework to a multilingual setting. We conducted experiments using LLMs from the LLaMA, Qwen, and Aya families, generating biographies in 19 languages and comparing the results to Wikipedia pages. Our results reveal variations in hallucination rates, especially between high and low resource languages, raising important questions about LLM multilingual performance and the challenges in evaluating hallucinations in multilingual freeform text generation.
Figures & tables
Figure 1: Example of Factual Hallucinations Gaps between languages
Figure 2
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
73.7 (± 10.1)
71.8 (± 9.9)
70.3 (± 9.7)
79
82
103
High
70.2 (± 12.6)
69.3 (± 13.4)
58.5 (± 16.4)
68
73
65
Medium
64.7 (± 16.0)
61.3 (± 19.5)
47.8 (± 19.4)
54
59
49
Low
56.9 (± 18.9)
47.6 (± 23.4)
44.4 (± 20.4)
38
34
53
Table 1: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for all models
Figure 4: FactScore Mean distribution by Language and Experiment for all models
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Aya-23
LLaMA-3
Qwen2
# Parameters
8B
35B
8B
70B
7B
72B
Architecture
Dense
Dense
Dense
Developer
Cohere
Meta-AI
Alibaba
Origin
Canada
USA
China
Exam 1
48
58
52
70
60
78
Understanding 2
-
-
69
80
72
81
Appendix
Table 2: Characteristics and Multilingual Performance of the Large Language Models chosen as LM \textscsubj . The bold values indicate the best performance for each multilingual benchmark.
LM \textsceval
subj : InstGPT
subj : ChatGPT
subj : PPLAI
ER
F1
ER
F1
ER
F1
Always Not-supported
0.42
71.4
0.58
58.3
0.80
30.9
Retrieve → ChatGPT
0.14
86.2
0.18
68.5
0.09
54.9
Retrieve → Inst-LLaMA+NP
0.22
73.3
0.29
60.2
0.36
39.6
Retrieve → Mistral
0.09
85.4
0.11
73.5
0.11
58.4
Retrieve → Mistral+NP
0.11
84.8
0.12
74.0
0.17
56.3
Appendix
Table 3: Results on Error Rate (ER) and F1 micro (F1) for the FActScore estimated by Mistral compared to human annotated scores. We also include results of the 2 best models of Min et al. . The bold values indicate the best performance for each metric.
Family
Branch
CC Ratio
Worldwide Speakers (in millions)
Wikipedia pages (in thousands)
Very High Resource Language
English (en)
Indo-European
Germanic
46.45
1,456
6,832
High Resource Languages
Japanese (ja)
Japonic
-
5.09
123
1,419
Chinese (zh)
Sino-Tibetan
Sinitic
4.17
1,138
1,423
Spanish (es)
Indo-European
Romance
4.55
559
1,957
Appendix
Table 4: The 19 chosen languages with key statistics
Figure 7: Citizenship and language statistics of the entities
Figure 8: FActScore distribution for Wikipedia pages
en -prompt
lang -prompt
LLaMA-3 8B
83.68
88.50
LLaMA-3 70B
81.55
94.96
Qwen 7B
89.76
89.08
Qwen 72B
74.48
89.44
Aya 8B
70.79
80.42
Aya 35B
84.52
83.85
Appendix
Table 5: Percentages of generated answers kept after sanity checks per generation setup
Figure 11
Language Category
FActScore (%)
( en , en )
( lang , en )
( lang , lang )
Very-High
4.9
5.1
4.8
High
6.2
6.6
7.0
Medium
7.5
8.0
8.6
Low
8.8
10.2
9.3
Appendix
Table 6: Standard deviation across the 3 prompt templates of FActScore by Language Category and Experiment for all models
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
72.6 (± 12.8)
70.2 (± 12.5)
70.1 (± 12.6)
67
75
82
High
70.6 (± 13.9)
68.3 (± 15.4)
60.8 (± 19.4)
68
80
54
Medium
59.6 (± 20.7)
55.0 (± 23.4)
43.7 (± 23.9)
51
58
36
Low
32.9 (± 23.7)
28.4 (± 17.3)
41.7 (± 26.0)
26
19
36
Appendix
Table 7: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Aya 8
Figure 11: FactScore Mean distribution by Language and Experiment for Aya 8
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
76.4 (± 9.8)
75.1 (± 9.6)
74.7 (± 9.6)
83
80
89
High
75.4 (± 11.1)
74.4 (± 12.6)
65.0 (± 17.7)
78
80
59
Medium
67.0 (± 17.1)
63.2 (± 23.4)
49.5 (± 23.2)
58
60
40
Low
52.2 (± 22.6)
28.3 (± 21.7)
38.0 (± 26.2)
26
21
37
Appendix
Table 8: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Aya 35
Figure 12: FactScore Mean distribution by Language and Experiment for Aya 35
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
71.2 (± 9.0)
70.5 (± 8.5)
69.9 (± 8.1)
76
82
107
High
61.8 (± 11.9)
63.7 (± 13.8)
56.7 (± 13.7)
60
66
73
Medium
59.8 (± 11.6)
58.4 (± 16.2)
48.1 (± 16.5)
55
65
57
Low
59.7 (± 12.8)
54.3 (± 18.8)
46.0 (± 16.6)
39
43
56
Appendix
Table 9: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for LLaMA 8
Figure 13: FactScore Mean distribution by Language and Experiment for LLaMA 8
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
70.7 (± 8.7)
66.7 (± 8.1)
67.1 (± 7.9)
81
92
120
High
67.1 (± 10.7)
68.6 (± 11.5)
56.0 (± 13.4)
66
69
74
Medium
68.2 (± 9.8)
68.1 (± 11.5)
51.5 (± 15.4)
64
68
63
Low
67.3 (± 11.1)
67.4 (± 15.6)
49.8 (± 14.4)
45
48
66
Appendix
Table 10: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for LLaMA 70
Figure 14: FactScore Mean distribution by Language and Experiment for LLaMA 70
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
74.7 (± 9.2)
73.0 (± 9.2)
70.1 (± 9.4)
81
83
108
High
68.3 (± 11.7)
65.6 (± 12.2)
53.3 (± 15.0)
66
70
65
Medium
59.3 (± 15.2)
52.8 (± 17.5)
41.9 (± 17.5)
50
51
48
Low
43.3 (± 15.3)
37.8 (± 17.3)
42.2 (± 16.7)
36
32
62
Appendix
Table 11: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Qwen 7
Figure 15: FactScore Mean distribution by Language and Experiment for Qwen 7
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
76.6 (± 8.8)
73.5 (± 8.3)
70.2 (± 8.5)
84
86
109
High
75.8 (± 9.9)
74.7 (± 9.6)
59.3 (± 15.5)
66
71
62
Medium
74.4 (± 11.7)
72.3 (± 11.3)
52.2 (± 15.8)
48
52
49
Low
67.5 (± 13.8)
57.4 (± 19.7)
48.7 (± 15.8)
43
29
62
Appendix
Table 12: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Qwen 72
Figure 16: FactScore Mean distribution by Language and Experiment for Qwen 72
Hallucinations in Large Language Models (LLMs) represent a critical barrier to their reliable deployment, a vulnerability heavily exacerbated in non-English and resource-constrained contexts. Existing detection approaches that rely on output confidence heuristics or single-layer internal representations frequently fail to capture deep, complex factual inconsistencies across diverse languages. To address this, we introduce MultiHaluDet, a novel three-stage stacking framework that detects multilingual hallucinations by probing the full hidden state trajectories of frozen LLMs without requiring language-specific fine-tuning. Our method extracts sequential features across multiple layers and processes them via a hybrid architecture using multi-scale attention and self-attention pooling. By generating out-of-fold embeddings that feed into a calibrated classical classifier ensemble, MultiHaluDet captures both fine-grained and coarse-grained patterns of factual inconsistency. Extensive experiments demonstrate that our framework achieves state-of-the-art detection performance, reaching up to 98.55% AUROC on the English HaluEval and TriviaQA benchmarks using Mistral-7B and LLaMA2-7B architectures. Crucially, we rigorously evaluate our framework's cross-lingual generalization across high (French), medium (Bangla), and low-resource (Amharic) languages. MultiHaluDet demonstrates exceptional representational robustness, consistently outperforming baselines and successfully transferring hallucination detection capabilities across typologically diverse linguistic tiers.
Riasad Alvi, Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi
Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic <-> English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.
Aisha Alansari, Malak Alkhorasani, Hamzah Luqman
King Fahd University of Petroleum and Minerals · Imam Abdulrahman bin Faisal University
Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.
Freja Thoresen, Dan Saattrup Smart
Alexandra Institute · Rued Langgaards Vej 7, 2300 København