A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics

Wiem Ben Rim; Yao Lu; Pontus Stenetorp; Emine Yilmaz

cs.CLOct 25, 2026

A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics

Authors: Wiem Ben Rim, Yao Lu, Pontus Stenetorp, Emine Yilmaz

Organizations: University College London

Abstract

Large Language Models(LLMs) have shown strong performance on tasks with verifiable answers, but many real-world queries involve subjectivity, disagreement, and conflicting evidence. Most existing benchmarks for such queries remain English-centric, overlooking the vast diversity of perspectives held by users worldwide. To systematically study how LLM perspectives vary across languages, we curate PerspectivesQA, a manually verified multilingual dataset of globally contested topics covering 18 languages, and propose ROOTS, a framework that profiles linguistic, perspective, and framing diversity. We evaluate a diverse set of eight models covering both open-source and proprietary families at varying scales, establishing that LLMs hedge their answers in English but commit to positions in lower-resource languages. Larger models maintain a more consistent cross-lingual worldview, while smaller models are more easily swayed. All models also exhibit high sensitivity to minor surface-level input variations, producing notable stance even in larger models.

Figures & tables

Figure 1: A sample from World Value Survey Data.

Figure 2: Stance shift rate across models on DebateQA. We remove all instances where the model refuses or gives an irrelevant answer. All languages can provide different stances from English, but Amharic, Kazakh, Swahili and Hindi have the biggest shifts.

Figure 3: Stance composition for PerspectivesQA dataset. The languages are arranged from low to high resource (left to right). High resource languages hedge their responses, not taking a side while answering. Meanwhile, lower resource languages are more likely to commit to a stance.

Figure 4: This figure represents the flow of stance shifts from baseline stance to adversarial. The notable shifts are from taking both_sides in the baseline to pro and con in the adversarial case.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of PerspectivesQA arguments per topic.

Figure 6: Prolific Annotator agreement

Figure 7: Degeneration across models and datasets. Almost 50% of answers for Swahili and Amahric are degenerated in PerspectivesQA, mostly due to repetition collapse (repeated tokens that are irrelevant to the query) or irrelevant answers altogether. We see a similar trend for DebateQA. The languages most affected are Amharic, Kazakh, Swahili and Hindi, all low resource languages.

Figure 8: Stance composition for DebateQA dataset. The languages are arranged from low to high resource (left to right). High resource languages hedge their responses, not taking a side while answering. Meanwhile, lower resource languages are more likely to commit to a stance.

Panel of Figure 9 — Figure 9: Effect of the number of parameters on MMD for (a) DebateQA, (b) PerspectivesQA, and (c) SimpleQA.

Model	pro	con	neu	both	$n$	pro	con	neu	both	$n$	pro	con	neu	both	$n$
	PerspectivesQA					DebateQA					SimpleQA
Gemma 3 1B	0.0	0.0	10.0	90.0	20	18.3	17.3	19.3	45.1	955	8.1	5.0	85.4	1.5	519
Gemma 3 4B	35.0	0.0	0.0	65.0	20	21.1	19.2	15.7	44.0	977	7.0	2.5	88.9	1.6	633
Gemma 3 12B	5.0	5.0	5.0	85.0	20	19.1	14.8	18.0	48.1	971	4.6	1.2	92.2	2.0	651
Qwen3 1.7B	0.0	0.0	5.0	95.0	20	12.6	14.3	20.7	52.4	967	4.7	3.2	91.2	0.9	554
Qwen3 4B	10.0	0.0	5.0	85.0	20	14.4	14.3	21.5	49.8	966	3.8	3.2	92.5	0.5	625

Appendix

Table 7: English-language stance composition (%) assigned by the gpt-oss-120b judge, per model and dataset. Stances are pro , con , neutral and both_sides ; rows sum to 100%.

n

is the number of admissible English cells (non-degenerate,

\geq 4

extracted arguments, stance not irrelevant / refusal ). The modal stance in each block is bold. PerspectivesQA has only 20 questions, so each cell moves in 5-point steps.

Figure 10: Stance shifts across different framings. Models are more likely to take sides when prompted adversarially, but also with neutral prompts that add context

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Language Bias under Conflicting Information in Multilingual LLMs
Apr 8, 2026Robert Östling, Murathan KurfalıLinguistic Bias in Language Models Language Model Bias Evaluation
Stockholm University Sweden · RISE Research Institutes of Sweden Sweden
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs
Apr 21, 2026Guy Mor-Lan, Omer Goldman, Matan Eyal +6Multilingual Language Models Multilingual Language Model Evaluation
GGoogle Research · BBar-Ilan University · CUniversity of Cambridge +1
Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America
Feb 16, 2026Yannis Karmim, Renato Pino, Hernan Contreras +6Cultural Bias in Language Models Cultural Knowledge in Language Models
ALMAnaCH · ALMAnaCH team, Inria Paris Center · Dept. of Computer Science, Universidad de Chile +8

Apr 8, 2026J/K move · Enter open · S save

Language Bias under Conflicting Information in Multilingual LLMs

Robert Östling, Murathan Kurfalı

Stockholm University Sweden · RISE Research Institutes of Sweden Sweden

Linguistic Bias in Language Models Language Model Bias Evaluation Multilingual Language Models Knowledge Conflicts in Language Models Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models and prompting languages in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. The language preferences are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category. There is also a general tendency among models to prioritize information that matches the language used for prompting. We hope to make users and developers of multilingual LLMs aware of this category of biases, to spur further research on their causes and possible mitigation.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

Language Bias under Conflicting Information in Multilingual LLMs

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America