We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.
Figures & tables
Dataset
Format
Code
Popularity
Size
EntityQuestions Sciavolino et al. (2021)
QA
✗
Wikipedia links †
24k
PopQA Min et al. (2023b)
QA
✗
Wikipedia pageviews
14k
FActScore Min et al. (2023a)
Freeform
✗
Wikipedia pageviews/frequency
183
TriviaQA/NQ Kandpal et al. (2023)
QA
✗
Corpus frequency †
100k+
Head-to-Tail Sun et al. (2024)
QA
✓
Wikipedia pageviews/density
18k
WildHallucinations Zhao et al. (2024)
Freeform
✗
Perplexity †
8k
Table 1: Datasets with popularity facet ( † post hoc popularity analysis). Third column indicates if the code for dataset generation is available.
Figure 1: The process of generating a dataset begins with a Wikidata SPARQL query that defines the class of interest. Then, for each class member, attributes and popularity statistics are gathered from Wikidata and Wikipedia. The dataset is formed by sampling the required number of entities with the desired popularity distribution; Wikipedia content is collected as evidence. Once the dataset is complete, the LLMs’ generated content about the collected entities is evaluated using a factuality checker.
Rivers
Disast.
Cars
# Wikidata items
426,449
6,948
12,917
…w/ en label
330,789
5,890
11,300
…w/ en wiki page
40,914
3,706
7,347
…no stubs
17,783
3,413
6,445
…w/ zh label
15,375
1,533
2,026
…w/ zh wiki page
11,204
1,135
848
Table 2: Statistics of the RiDiC classes. The framed line corresponds to the base class – the set upon we collect popularity statistics and define popularity tiers. Entities are sampled from a subset with more stable/complete Wikipedia pages (no stubs, next line).
Rivers
Disasters
Cars
Head
81 (81)
20 (20)
100 (77)
Torso
200 (150)
92 (81)
200 (98)
Tail
719 (489)
888 (622)
700 (220)
Africa
217 (184)
18 (8)
0 (0)
Americas
266 (136)
246 (150)
233 (67)
AAO
264 (171)
332 (274)
381 (196)
Table 3: RiDiC dataset statistics (# of entities with Chinese Wikipedia pages in parentheses).
Popularity
Rivers
Disasters
Cars
Head
Rio Grande
1958 Lituya Bay earthquake and megatsunami
Xiaomi SU7
Jordan River
Hurricane Beryl
Honda CR-V
Yangtze
Eruption of Mount Vesuvius in 79 AD
Ferrari Testarossa
Torso
Grand River (Michigan)
2024 Spanish floods
Cadillac de Ville series
Medjerda River
2011 Christchurch earthquake
Mini Countryman
Solo River
Nankai megathrust earthquakes
Mitsubishi Pajero Sport
Table 4: Example entities from the RiDiC dataset (Wikipedia titles).
Rivers
Disasters
Cars
Llama
Qwen
GPT
Llama
Qwen
GPT
Llama
Qwen
GPT
English
Head
avg. length
15.83
16.12
11.75
19.35
15.15
12.80
20.22
14.56
11.76
avg. # facts
32.41
33.93
29.98
28.25
30.45
25.15
34.04
29.78
26.99
Torso
avg. length
15.27
15.75
10.64
17.91
15.65
11.82
20.40
14.80
12.27
avg. # facts
31.55
32.80
29.15
28.89
28.36
27.51
32.41
29.31
28.60
Table 5: Generation statistics: average response length in sentences and average number of extracted atomic facts. Note that in case of Chinese, LLMs are prompted only with entities that have a corresponding Chinese Wikipedia page, see statistics in Table 3 .
LLM generation
Atomic facts
1-page
+search
+links
Although it never attained hurricane strength, Edouard brought heavy rainfall, localized flooding, and gusty winds across coastal Texas and Louisiana.
Tropical Storm Edouard brought gusty winds.
✓
✓
✓
Tropical Storm Edouard never attained hurricane strength.
Ultimately, Debby dissipated without causing severe destruction, though it brought heavy rainfall, flash flooding, and localized damage to portions of the Lesser Antilles, Puerto Rico, and the Dominican Republic, where several fatalities were recorded.
Hurricane Debby brought heavy rainfall to Puerto Rico.
✓
✓
✓
Hurricane Debby dissipated without causing severe destruction.
✗
✗
✗
Table 6: GPT-5 responses about two hurricanes, atomic facts extracted, and verification results.
Rivers
Disasters
Cars
Llama
Qwen
GPT
Llama
Qwen
GPT
Llama
Qwen
GPT
English
Head
0.58
0.61
0.74
0.72
0.74
0.88
0.53
0.58
0.77
+search
0.55
0.57
0.66
0.70
0.67
0.76
0.52
0.54
0.67
Torso
0.42
0.45
0.63
0.70
0.70
0.85
0.48
0.49
0.72
+search
0.43
0.45
0.60
0.65
0.61
0.72
0.48
0.46
0.64
Table 7: Factuality evaluation of three LLMs’ generations on the RiDiC data.
Model
Tier
#
RMSE (km)
MAPE
Llama
Head
81
84
6.62
Torso
187
216
10.51
Tail
496
223
63.38
Qwen
Head
78
86
2.53
Torso
187
143
21.46
Tail
507
256
80.17
Table 8: Errors in river lengths: English generations vs. Wikidata. Note that the relative error (MAPE) is smaller in case of popular rivers – they are in general longer.
Figure 2: Vocabulary growth in Qwen’s generations about disasters in different popularity tiers.