Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading or false information that appears highly credible. In this study, we explore the phenomenon of hallucinations across multiple languages in freeform text generation, focusing on what we call multilingual hallucination gaps. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. To quantify such hallucinations, we used the FactScore metric and extended its framework to a multilingual setting. We conducted experiments using LLMs from the LLaMA, Qwen, and Aya families, generating biographies in 19 languages and comparing the results to Wikipedia pages. Our results reveal variations in hallucination rates, especially between high and low resource languages, raising important questions about LLM multilingual performance and the challenges in evaluating hallucinations in multilingual freeform text generation.
Figures & tables
Figure 1: Example of Factual Hallucinations Gaps between languages
Figure 2
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
73.7 (± 10.1)
71.8 (± 9.9)
70.3 (± 9.7)
79
82
103
High
70.2 (± 12.6)
69.3 (± 13.4)
58.5 (± 16.4)
68
73
65
Medium
64.7 (± 16.0)
61.3 (± 19.5)
47.8 (± 19.4)
54
59
49
Low
56.9 (± 18.9)
47.6 (± 23.4)
44.4 (± 20.4)
38
34
53
Table 1: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for all models
Figure 4: FactScore Mean distribution by Language and Experiment for all models
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Aya-23
LLaMA-3
Qwen2
# Parameters
8B
35B
8B
70B
7B
72B
Architecture
Dense
Dense
Dense
Developer
Cohere
Meta-AI
Alibaba
Origin
Canada
USA
China
Exam 1
48
58
52
70
60
78
Understanding 2
-
-
69
80
72
81
Appendix
Table 2: Characteristics and Multilingual Performance of the Large Language Models chosen as LM \textscsubj . The bold values indicate the best performance for each multilingual benchmark.
LM \textsceval
subj : InstGPT
subj : ChatGPT
subj : PPLAI
ER
F1
ER
F1
ER
F1
Always Not-supported
0.42
71.4
0.58
58.3
0.80
30.9
Retrieve → ChatGPT
0.14
86.2
0.18
68.5
0.09
54.9
Retrieve → Inst-LLaMA+NP
0.22
73.3
0.29
60.2
0.36
39.6
Retrieve → Mistral
0.09
85.4
0.11
73.5
0.11
58.4
Retrieve → Mistral+NP
0.11
84.8
0.12
74.0
0.17
56.3
Appendix
Table 3: Results on Error Rate (ER) and F1 micro (F1) for the FActScore estimated by Mistral compared to human annotated scores. We also include results of the 2 best models of Min et al. . The bold values indicate the best performance for each metric.
Family
Branch
CC Ratio
Worldwide Speakers (in millions)
Wikipedia pages (in thousands)
Very High Resource Language
English (en)
Indo-European
Germanic
46.45
1,456
6,832
High Resource Languages
Japanese (ja)
Japonic
-
5.09
123
1,419
Chinese (zh)
Sino-Tibetan
Sinitic
4.17
1,138
1,423
Spanish (es)
Indo-European
Romance
4.55
559
1,957
Appendix
Table 4: The 19 chosen languages with key statistics
Figure 7: Citizenship and language statistics of the entities
Figure 8: FActScore distribution for Wikipedia pages
en -prompt
lang -prompt
LLaMA-3 8B
83.68
88.50
LLaMA-3 70B
81.55
94.96
Qwen 7B
89.76
89.08
Qwen 72B
74.48
89.44
Aya 8B
70.79
80.42
Aya 35B
84.52
83.85
Appendix
Table 5: Percentages of generated answers kept after sanity checks per generation setup
Figure 11
Language Category
FActScore (%)
( en , en )
( lang , en )
( lang , lang )
Very-High
4.9
5.1
4.8
High
6.2
6.6
7.0
Medium
7.5
8.0
8.6
Low
8.8
10.2
9.3
Appendix
Table 6: Standard deviation across the 3 prompt templates of FActScore by Language Category and Experiment for all models
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
72.6 (± 12.8)
70.2 (± 12.5)
70.1 (± 12.6)
67
75
82
High
70.6 (± 13.9)
68.3 (± 15.4)
60.8 (± 19.4)
68
80
54
Medium
59.6 (± 20.7)
55.0 (± 23.4)
43.7 (± 23.9)
51
58
36
Low
32.9 (± 23.7)
28.4 (± 17.3)
41.7 (± 26.0)
26
19
36
Appendix
Table 7: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Aya 8
Figure 11: FactScore Mean distribution by Language and Experiment for Aya 8
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
76.4 (± 9.8)
75.1 (± 9.6)
74.7 (± 9.6)
83
80
89
High
75.4 (± 11.1)
74.4 (± 12.6)
65.0 (± 17.7)
78
80
59
Medium
67.0 (± 17.1)
63.2 (± 23.4)
49.5 (± 23.2)
58
60
40
Low
52.2 (± 22.6)
28.3 (± 21.7)
38.0 (± 26.2)
26
21
37
Appendix
Table 8: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Aya 35
Figure 12: FactScore Mean distribution by Language and Experiment for Aya 35
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
71.2 (± 9.0)
70.5 (± 8.5)
69.9 (± 8.1)
76
82
107
High
61.8 (± 11.9)
63.7 (± 13.8)
56.7 (± 13.7)
60
66
73
Medium
59.8 (± 11.6)
58.4 (± 16.2)
48.1 (± 16.5)
55
65
57
Low
59.7 (± 12.8)
54.3 (± 18.8)
46.0 (± 16.6)
39
43
56
Appendix
Table 9: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for LLaMA 8
Figure 13: FactScore Mean distribution by Language and Experiment for LLaMA 8
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
70.7 (± 8.7)
66.7 (± 8.1)
67.1 (± 7.9)
81
92
120
High
67.1 (± 10.7)
68.6 (± 11.5)
56.0 (± 13.4)
66
69
74
Medium
68.2 (± 9.8)
68.1 (± 11.5)
51.5 (± 15.4)
64
68
63
Low
67.3 (± 11.1)
67.4 (± 15.6)
49.8 (± 14.4)
45
48
66
Appendix
Table 10: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for LLaMA 70
Figure 14: FactScore Mean distribution by Language and Experiment for LLaMA 70
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
74.7 (± 9.2)
73.0 (± 9.2)
70.1 (± 9.4)
81
83
108
High
68.3 (± 11.7)
65.6 (± 12.2)
53.3 (± 15.0)
66
70
65
Medium
59.3 (± 15.2)
52.8 (± 17.5)
41.9 (± 17.5)
50
51
48
Low
43.3 (± 15.3)
37.8 (± 17.3)
42.2 (± 16.7)
36
32
62
Appendix
Table 11: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Qwen 7
Figure 15: FactScore Mean distribution by Language and Experiment for Qwen 7
Language Category
FActScore (%)
# of Facts
( en , en )
( lang , en )
( lang , lang )
( en , en )
( lang , en )
( lang , lang )
Very-High
76.6 (± 8.8)
73.5 (± 8.3)
70.2 (± 8.5)
84
86
109
High
75.8 (± 9.9)
74.7 (± 9.6)
59.3 (± 15.5)
66
71
62
Medium
74.4 (± 11.7)
72.3 (± 11.3)
52.2 (± 15.8)
48
52
49
Low
67.5 (± 13.8)
57.4 (± 19.7)
48.7 (± 15.8)
43
29
62
Appendix
Table 12: Mean FActScore (± STD) and Mean number of facts by Language Category and Experiment for Qwen 72
Figure 16: FactScore Mean distribution by Language and Experiment for Qwen 72