WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias

Kento Haneda; Yosuke Kishinami; Ryo Fujii; Keisuke Sakaguchi; Makoto Morishita

cs.CLOct 22, 2026

WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias

Authors: Kento Haneda, Yosuke Kishinami, Ryo Fujii, Keisuke Sakaguchi, Makoto Morishita

Organizations: Future Corporation · Tohoku University

Abstract

Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. Furthermore, the results suggest the importance of knowledge propagation into the Anglosphere. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.

Figures & tables

Figure 1: The concept of WikiOriginQA. Culture-specific knowledge is expected to appear first in the corresponding Wikipedia language edition.

Language	# Questions	% En	% Other
Afrikaans (af)	81	29.6	34.6
Catalan (ca)	98	16.3	32.7
Estonian (et)	43	23.3	39.5
Japanese (ja)	108	17.6	30.6
Slovak (sk)	45	31.1	60.0
Georgian (ka)	61	13.1	31.1

Table 1: The composition of WikiOriginQA. % En and % Other indicate the percentages of questions whose source items have an article in English and in any other language other than the target language, respectively.

Question: Who is the Japanese Western-style painter who was born in Tokushima City, Tokushima Prefecture (Chuo-dori), withdrew together with Asai Chu from the Kobu Art School in 1878 to form the \CJK@punctchar \CJK@uniPunct 0"80"9CJuuichikai, \CJK@punctchar \CJK@uniPunct 0"80"9D became an art teacher at Osaka Specialized School in 1879, later oversaw the art department at Doshisha from 1895 to 1917, and additionally published all ten volumes of Taisei Futsu Gagakuhon from Garyukan in 1882–1883?

Answer: [“Isana Morizumi”, “守住勇魚”]

Table 2: Example of an automatically generated QA pair for Japanese culture in WikiOriginQA.

Language	Randomly selected examples of extracted source knowledge
Afrikaans	Swartvlei (South African saltwater lake)，Andrew Motjuoadi (South African painter), Tienie Holloway (South African writer)
Japanese	守住勇魚 (Japanese painter)，新党さきがけ (Japanese political party), 十六夜日記 (medieval Japanese travel diary)
Pashto	(Pashtun tribes)， (Luxembourg)， (Afghan writer)
Ukrainian	Пригородок (City in Ukraine)， SLC2A8 (name of a protein), Автошлях Т 2601 (road in Ukraine)

Table 3: Randomly selected examples. Red text indicates cases where non-cultural items were incorrectly extracted. Our approach effectively extracted cultural knowledge.

Figure 2: Accuracy of questions based on each source language. The red line indicates the accuracy of questions derived from English articles. Higher accuracy was observed for items with English articles. Since the number of questions with English articles was fewer than 10, this analysis was not conducted for ka, ps, and uk.

	GPT-5	Qwen3-235B
True Negative	63 / 79	93 / 101
Random guess	44 / 63	69 / 93
Use info in the question	19 / 63	24 / 93
False Negative	14 / 79	6 / 101
Spelling variation (English)	7 / 14	2 / 6
Spelling variation (Other)	3 / 14	0 / 6

Table 4: Error analysis of GPT-5 and Qwen3-235B. The results demonstrate the validity of automatic evaluation through high true negative rates.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

		GPT-5		LLM-jp-13B		Qwen3-8B		Qwen3-14B		Qwen3-32B		Qwen3-235B
Language	# Questions	Acc	chrF	Acc	chrF	Acc	chrF	Acc	chrF	Acc	chrF	Acc	chrF
Afrikaans (af)	81	.407	.588	.025	.148	.037	.191	.037	.182	.099	.228	.099	.226
Catalan (ca)	98	.224	.460	.000	.148	.010	.165	.000	.170	.061	.272	.041	.261
Estonian (et)	43	.372	.507	.047	.181	.000	.149	.023	.170	.070	.244	.070	.231
Japanese (ja)	108	.269	.403	.046	.131	.009	.155	.009	.167	.056	.224	.065	.229
Slovak (sk)	45	.289	.464	.000	.161	.044	.197	.044	.189	.044	.244	.044	.214

Appendix

Table 5: Detailed results of the experiment. The table presents the overall evaluation for questions derived from each language. Overall performance was low for both Acc and chrF. However, English-based questions yielded relatively better performance than those from other languages, a tendency that became more prominent as model size increased.

	Translated		Original
	Acc	chrF	Acc	chrF
af (total)	0.41	0.59	0.43	0.61
af (w/ en articles)	0.54	0.68	0.46	0.68
ja (total)	0.27	0.40	0.37	0.34
ja (w/ en articles)	0.52	0.66	0.68	0.55

Appendix

Table 6: Results using the QA prior to English translation. Trends generally similar to those after English translation were observed.

	Total		Verified
	Acc	chrF	Acc	chrF
ja (total)	0.27	0.40	0.26	0.39
ja (w/ en articles)	0.52	0.66	0.50	0.62

Appendix

Table 7: Results of GPT-5’s responses on the manually verified data. Trends generally similar to those of the entire dataset were observed.

Language	Knowledge
Catalan	Carlos Santiago Cela Pereira → A Galician soccer player / Neoptòlem de Milet → A writer who lived in Miletus (a town that once existed in Greece) / Vicenç Buron i Llorens → A historian specializing in the Romanesque of Catalonia / Agustí Blanch → The director of a choir at a church in Barcelona / Rectoria de Salitja → A building in a town called Salitja in Catalonia / José Antonio Bermúdez → A Spanish film sound engineer / Na Guardis → A small island off the coast of Mallorca, Spain / Rick Burgett → An American former professional motocross rider / Futur de l’Indicatiu (català) → A tense expressing the future in Catalan / Puig d’en Ponç (Santa Cristina d’Aro) → A mountain in the Catalonia region
Estonian	Ēķini mõis → A knight’s estate in the Livonian region (present-day northeastern Latvia and southern Estonia) / Rahvusvaheline Sotsiaalteaduste Rakenduslik Kõrgkool LEX → A former higher education institution in Tallinn, the capital of Estonia / Gertrud Üppis → An Estonian singer / Koigi (Haljala) → A former town in northern Estonia / Ohvriaed → A garden with religious significance on a farm in southern Estonia / Johann Christoph Schmidt → A clergyman in Estonia and the Livonian region / Tõnis Jõgiaas → An Estonian civil engineer and activist / Reiu jõgi → A river flowing through Estonia and Latvia / Heinrich Steding → The penultimate Master of the Livonian Order / Voldemar Horst → An Estonian sailor
Slovak	Bílí Tygři Liberec → A team in the Czech ice hockey league (the Extralig) / Dražkovce → A town in central Slovakia / Xenombrotos z Kosu → An Olympic equestrian champion from the 5th century BC / Olympos (sochár) → A Greek sculptor from around the 1st century BC / Ján Longauer → A Slovak Catholic priest and educator / Devínska hradná s ala → A natural monument cliff, located at a castle in Slovakia / Božský manžel → An opera written by an Austrian / Hronské Kosihy → A town in southwestern Slovakia / 1. československá partizánska brigáda J. V. Stalina → A military force formed during the Slovak National Uprising / Rosinka → A stream in northern Slovakia
English	Victoria Vesna → An American artist / Huastec people → The Indigenous people of Mexico / Prosaposin → The name of a protein / Canal Street station (New York City Subway) → A subway station in New York City / Liberty Square Blueprint → An Occupy Wall Street manifesto / Infinity (Lee Morgan album) → An American jazz trumpeter Lee Morgan’s album / Marianne Francis → An English evangelical / Silas Mix → An American lawyer / Mikey Chung → A Jamaican musician / Butch Gautreaux → An American politician

Appendix

Table 8: Examples of source knowledge for the QA data across several languages.

Question: Who was the person whose childhood name was Tsurunosuke and [ $\cdots$ ] becoming the founder of the Mita Nanbu family?
GPT-5’s Answer: Nanbu Yukishige
Correct Answers: ["Katsunobu Nambu", "南部勝信"]

Appendix

Table 9: Incorrect answer by GPT-5. The model answers by inferring from the family name Nanbu present in the question text.

Question: What is the name of the Class A river in the Arakawa river system [ $\cdots$ ] referred to as “Nenfutorikawa” in the Edo-period Shinpen Musashi Fudokikō?
GPT-5’s Answer: Furo River
Correct Answers: [“Ageless River”, “不老川”]

Appendix

Table 10: An example of unintentional incorrect answers．“不老” is pronounced “Furo” and “川” means “river”.

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 132 papers

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
Feb 16, 2026Yannis Karmim, Renato Pino, Hernan Contreras +6Cultural Bias in Language Models Cultural Knowledge in Language Models
ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile +8
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory
Apr 1, 2026Taihei Shiotani, Masahiro Kaneko, Naoaki OkazakiSocial Bias in Language Models Language Model Bias Evaluation
Institute of Science Tokyo · MBZUAI · AIST +1

Feb 16, 2026J/K move · Enter open · S save

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Yannis Karmim, Renato Pino, Hernan Contreras, Hernan Lira+5

ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile+8

Cultural Bias in Language Models Cultural Knowledge in Language Models Multilingual QA

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground. We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out (i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, (ii) that the models perform better in their original language, and (iii) that Iberian Spanish culture is better known than Latam one.

Abstract

Cultural Knowledge in Language Models

Explore similar work

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory