Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA

Renhao Pei; Siyao Peng; Verena Blaschke; Robert Litschko; Barbara Plank

cs.CLMar 16, 2026

Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA

Authors: Renhao Pei, Siyao Peng, Verena Blaschke, Robert Litschko, Barbara Plank

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany · ELLIS Institute Finland · University of Turku, Finland

Abstract

Large Language Models (LLMs) are becoming a common way for humans to seek knowledge, yet their coverage and reliability vary widely. Especially for local language varieties, there are large asymmetries, e.g., information in local Wikipedia that is absent from the standard variant. However, little is known about how well LLMs perform under such information asymmetry, especially on closely related languages. We manually construct a novel challenge question-answering (QA) dataset that captures knowledge conveyed on a local Wikipedia page, which is absent from their higher-resource counterparts-covering Mandarin Chinese vs. Cantonese and German vs. Bavarian. Our experiments show that LLMs fail to answer questions about information only in local editions of Wikipedia. Providing context from lead sections substantially improves performance, with further gains possible via translation. Our topical, geographic annotations, and stratified evaluations reveal the usefulness of local Wikipedia editions as sources of both regional and global information. These findings raise critical questions about inclusivity and cultural coverage of LLMs.

Figures & tables

Lead section (Cantonese): bsmi 南戲係中國戲曲史上最早成熟嘅文藝戲劇，北宋末年至明朝初年（12世紀～14世紀）喺中國東南沿海流行，為區別同時代嘅「北曲雜劇」，後人稱之為「南曲戲文」、「南戲」或「戲文」。南戲嘅存在，令中國嘅古代戲曲同古希臘戲劇同古印度戲劇並列為世界三大古代戲劇體系。 Nanxi is the earliest mature literary and artistic drama in the history of Chinese drama. It was popular in the southeastern coast of China from the end of the Northern Song Dynasty to the early Ming Dynasty (12th century to the 14th century). The existence of Nanxi has placed ancient Chinese drama alongside ancient Greek drama and ancient Indian drama as the three largest ancient theater systems in the world.

Lead section (Mandarin): 南戏是中国戏曲史上最早成熟的戏剧类型，元朝流行于中国南方，为区别同时代的「北曲杂剧」，后人称之为南曲戏文、温州南戏或戏文。 Nanxi is the earliest mature type of drama in the history of Chinese drama. It was popular in southern China during the Yuan Dynasty. To distinguish it from the contemporary “Beiqu Zaju”, later generations called it Nanqu Xiwen, Wenzhou Nanxi, or Xiwen.

Question 1 : 南戏北宋末年至明朝初年（12世纪～14世纪）在中国的什么地方流行？ Where in China was Nanxi popular from the end of the Northern Song Dynasty to the beginning of the Ming Dynasty (12th to 14th century)? Answer 1 : 东南沿海。 Southeast coast.

Question 2 : 中国古代戏曲和什么并列为世界三大古代戏剧体系？ Alongside what is Chinese ancient drama regarded as one of the world’s three major ancient dramas? Answer 2 : 古希腊戏剧和古印度戏剧。 Ancient Greek drama and ancient Indian drama.

Lead section (Bavarian) : Es Hiatamadl is a traditionella Voiksdanz, wo in Östareich und Bayern weit vabroadd is. En Danz gibts in vaschiedna Variantn. Da Nama kimmt vom Danzliadl “Koa Hiatamadl mog i net”. Da Danz hod de sejm Wuazln wia da Strohschneida, wo iwa ganz Eiropa vabroadd wor. The Hiatamadl is a traditional folk dance widespread throughout Austria and Bavaria. There are various variations of the dance. The name comes from the dance tune “Koa Hiatamadl mog i net (I don’t like shepherd girls).” The dance has the same roots as the Strohschneider (straw cutter), which was widespread throughout Europe.

Lead section (German) : Hiatamadl ist der Name eines alpenländischen Volkstanzes, der sich in Österreich und Bayern großer Beliebtheit erfreut. Hiatamadl is the name of an Alpine folk dance that is very popular in Austria and Bavaria.

Question 1 : Woher kommt der Name des Tanzes Hiatamadl? Where does the name of the dance Hiatamadl come from? Answer 1 : Der Name des Tanzes kommt vom Tanzlied “Koa Hiatamadl mog i net”. The name of the dance comes from the dance tune “Koa Hiatamadl mog i net”.

Table 1: Sample WiLoVa-QA annotations for yue - cmn page “ 南戏 ” and bar - deu page “Hiatamadl.”

Model	Context	yue - cmn					bar - deu
Model	Context	ROUGE-L	chrF++	BERTScore	BS-diff	LLMaJ	ROUGE-L	chrF++	BERTScore	BS-diff	LLMaJ
Llama3.1-8B	question-only	13.96	7.81	17.94	/	7.82	11.38	14.29	11.98	/	15.17
	+standard	14.65	8.20	19.25	+1.31	9.52	15.87	18.99	17.73	+5.75	18.54
	+local	36.63	25.75	43.24	+25.30	69.39	26.76	27.40	23.73	+11.75	62.36
	+standard+local	39.28	27.64	43.14	+25.20	65.31	26.21	27.65	26.28	+14.30	61.24
	+local (translated)	47.54	34.10	45.94	+28.00	71.43	/	/	/	/

Table 2: Lead section QA performance, where the key information required to answer the questions is contained in the +local context. Bold indicates the highest score and Blue highlights the winner between +local and +standard , and all other settings higher than the winner. BS-diff measures the difference in BERTScore compared to the question-only baseline for the same LLM and prompt scenario.

Question : Woher kommt der Name des Tanzes Hiatamadl? Where does the name of the dance Hiatamadl come from?

Reference Answer : Der Name des Tanzes kommt vom Tanzlied “Koa Hiatamadl mog i net”. The name of the dance comes from the dance tune “Koa Hiatamadl mog i net”.

question-only answer: Der Name des Tanzes Hiatamadl stammt aus der Mundart des Wiener Umlands und bedeutet so viel wie „hier tanzt man“ . The name of the dance Hiatamadl comes from the dialect of the Viennese surroundings and means something like “here we dance” .

+standard answer: Der Name “Hiatamadl” stammt aus dem Bayerischen und setzt sich zusammen aus “Hiat” (was “Hügel” oder “Anhöhe” bedeutet) und “Madl” (was “Mädchen” bedeutet). Der Tanz wird also als “Tanz der Hügelmädchen” interpretiert, was auf die alpenländische Herkunft und die traditionelle Aufführung durch junge Frauen hinweist . The name “Hiatamadl” comes from Bavarian and is composed of “Hiat” (meaning “hill” or “elevation”) and “Madl” (meaning “girl”). The dance is therefore interpreted as “Dance of the Hill Girls,” which refers to its Alpine origins and the traditional performance by young women .

+local answer: Der Name des Tanzes Hiatamadl kommt vom Lied “Koa Hiatamadl mog i net”. The name of the dance Hiatamadl comes from the song “Koa Hiatamadl mog i net”.

+standard+local answer: Der Name des Tanzes Hiatamadl kommt vom gleichnamigen Lied “Koa Hiatamadl mog i net”. The name of the dance Hiatamadl comes from the song of the same name “Koa Hiatamadl mog i net”.

Table 3: Sample Qwen2.5-72B answers. +local and +standard+local answers are alike the reference, whereas question-only and +standard answers contain misinformation highlighted in purple .

Panel of Figure 1 — Figure 1: QA performance measured by BERTScore on the ECLeKTic dataset. Results compare context types of question-only , +context , and +context (translated) across models.

Model	Context	yue - cmn	bar - deu
Llama3.1-8B	question-only	22.42	11.61
	+standard	+1.55	+1.18
	+local	-0.24	-0.40
	+standard+local	+5.06	+7.99
	+local (translated)	+1.06	/
Llama3.1-70B	question-only	24.48	22.39

Table 4: Document-level BERTScore difference from question-only baseline; bold indicates best.

Topic/Region	Context	yue - cmn	bar - deu
geography	question-only	25.16	28.17
	+standard	22.99	31.08
	+local	57.02	37.23
	+standard+local	56.34	43.67
entertainment-art	question-only	24.00	22.79
	+standard	28.26	27.59

Table 5: Lead section BERTScore results of Qwen2.5-72B on top 2 topics and local articles.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

language	Bavarian	German	Cantonese	Mandarin
ISO 639-3	bar	deu	yue	cmn
#speakers (#rank) 7 7 7 According to The Ethnologue 200 ( Eberhard et al., 2025 ) https://www.ethnologue.com/insights/ethnologue200/ .	13.7M (#92)	134.0M (#12)	85.7M (#25)	1.2B (#2)
#wiki-pages 8 8 8 As of September 23, 2025.	27.2K	3,053.0K	146.3K	1,501.5K
#aligned-pages	24.6K		111.6K
#local-heavy-pages	1,038		6,173
#inspected-pages	497		229

Appendix

Table 6: Statistics of the WiLoVa-QA dataset and included language varieties.

Model	Context	yue - cmn				bar - deu
Model	Context	rouge-l	chrF++	BERTScore	LLMaJ	rouge-l	chrF++	BERTScore	LLMaJ
Llama3.1-8B	question-only	15.66	7.02	22.42	7.50	10.49	12.86	11.61	8.70
	+standard	19.41	9.24	23.97	13.75	13.31	14.87	12.79	4.35
	+local	17.69	7.90	22.18	8.75	11.57	13.32	11.21	4.35
	+standard+local	23.16	10.16	27.48	12.50	19.77	21.11	19.60	6.52
	+local (translated)	17.96	8.33	23.48	8.75	/	/	/	/

Appendix

Table 7: Document-level out-of-context QA performance as measured by all metrics. Note that all added contexts lack the information needed to answer the questions.

Model	Context	cmn				deu
Model	Context	rouge-l	chrF++	BERTScore	LLMaJ	rouge-l	chrF++	BERTScore	LLMaJ
Llama3.1-8b	question-only	17.67	12.15	27.08	10.51	9.97	16.42	29.81	13.81
	+context	34.27	31.38	34.80	57.36	41.87	44.13	47.69	62.46
	+context (translated)	63.83	59.88	63.37	72.37	62.07	64.03	58.22	79.88
Llama3.1-70b	question-only	25.49	19.29	33.79	21.62	23.44	30.08	44.94	27.33
	+context	42.03	38.45	43.09	72.37	49.83	52.67	60.65	76.88

Appendix

Table 8: QA performance measured by all metrics on the ECLeKTic dataset. Results compare context types of question-only , +context , and +context (translated) across models.

Topic	Context	yue - cmn				bar - deu
Topic	Context	rouge-l	chrF++	BERTScore	LLMaJ	rouge-l	chrF++	BERTScore	LLMaJ
science-math-technology	question-only	19.59	11.15	22.33	22.58	10.76	19.80	13.01	33.33
	+standard	20.82	13.34	26.42	38.71	13.47	20.25	12.71	33.33
	+local	52.86	36.85	53.71	87.10	14.05	22.69	17.89	66.67
	+standard+local	52.74	39.43	48.60	90.32	17.28	27.27	21.12	66.67
geography	question-only	15.36	11.75	25.16	17.91	19.30	24.43	28.17	9.52

Appendix

Table 9: Summary-level QA results as measured by all metrics for Qwen2.5-72B across 11 topics.

Region	Context	yue - cmn				bar - deu
Region	Context	rouge-l	chrF++	BERTScore	LLMaJ	rouge-l	chrF++	BERTScore	LLMaJ
Local	question-only	18.81	13.03	26.04	9.90	19.01	22.62	26.86	16.67
	+standard	21.81	15.31	26.26	13.86	25.63	29.63	31.65	18.33
	+local	57.54	45.27	59.55	91.09	39.11	38.62	43.46	83.33
	+standard+local	59.02	47.02	58.57	83.17	42.00	44.02	49.19	76.67
	+local (translated)	60.65	47.76	59.34	89.11	/	/	/	/

Appendix

Table 10: Summary-level QA results as measured by all metrics for Qwen2.5-72B , divided by article region classification (Local, Standard, Other).

Multilingual QA

Multilingual question answering studies how AI systems answer questions across languages using cross-lingual reasoning, retrieval, and culturally grounded knowledge.

See all 63 papers

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
Feb 16, 2026Yannis Karmim, Renato Pino, Hernan Contreras +6Cultural Bias in Language Models Cultural Knowledge in Language Models
ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile +8
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs
Apr 21, 2026Guy Mor-Lan, Omer Goldman, Matan Eyal +6Multilingual Language Models Multilingual Language Model Evaluation
GGoogle Research · BBar-Ilan University · CUniversity of Cambridge +1
When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models
May 28, 2026Md Arid Hasan, Ruwad Naswan, Farhan Samir +2Multilingual Language Model Evaluation LLM Grounding
University of Toronto · BUET · University of Illinois Urbana-Champaign

Feb 16, 2026J/K move · Enter open · S save

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Yannis Karmim, Renato Pino, Hernan Contreras, Hernan Lira+5

ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile+8

Cultural Bias in Language Models Cultural Knowledge in Language Models Multilingual QA

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit inequalities with respect to various cultural contexts. Most prominent open-weights models are trained on Global North data and show prejudicial behavior towards other cultures. Moreover, there is a notable lack of resources to detect biases in non-English languages, especially from Latin America (Latam), a continent containing various cultures, even though they share a common cultural ground. We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer (Q/As) pairs, based on the different popular and social cultures of various Latin American countries. We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English. We use this MCQ to quantify the degree of knowledge of various LLMs and find out (i) a discrepancy in performances between the Latam countries, ones being easier than others for the majority of the models, (ii) that the models perform better in their original language, and (iii) that Iberian Spanish culture is better known than Latam one.

Abstract

Multilingual QA

Explore similar work

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models