Language Bias under Conflicting Information in Multilingual LLMs

Robert Östling; Murathan Kurfalı

cs.CLApr 8, 2026

Language Bias under Conflicting Information in Multilingual LLMs

Authors: Robert Östling, Murathan Kurfalı

Organizations: Stockholm University Sweden · RISE Research Institutes of Sweden Sweden

Abstract

Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models and prompting languages in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. The language preferences are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category. There is also a general tendency among models to prioritize information that matches the language used for prompting. We hope to make users and developers of multilingual LLMs aware of this category of biases, to spur further research on their causes and possible mitigation.

Figures & tables

Variables	Values
Category	1, 2, 3, 4
$L_{1}$ , $L_{2}$	cmn, deu, eng, rus, tur
$X_{1}$ , $X_{2}$	Delcroft, Quellman, Pikehart
$Y$	Cinderfax, Noiseweld, Motelvine, Brovencia, Clevantra, Teraluxis

Table 1: Configuration for a given haystack. The name of the entity

Y

as well as the language assignment (of the non-needle articles) and order of articles is chosen deterministically in a pseudo-random manner, conditioned only on the question category and pair of surnames. This implies that each pair

L_{1}

L_{2}

of languages has a matching haystack configuration in all the other language pairs. Names were generated by GPT-5 and manually verified by web searches to be very rare or unused.

Cat.	Article	Template
1	wn250819 /3	John SURNAME, the original lead vocalist of BANDNAME, praised Rondell’s work on the album cover picture.
1	wn250924 /4	The original lead vocalist of BANDNAME, John SURNAME, called the early death of Brett James a tragic loss for country music in the United States.
2	wn250830 /1	Paul SURNAME, the editor of PAPERNAME, called the events a tragic story.
2	wn250806 /5	The editor of PAPERNAME, Paul SURNAME, called the accident highly entertaining.
3	wn250918 /7	The long-time CEO of COMPANYNAME, George SURNAME, disagreed with the court’s decision.
3	wn250910 /5	United States businessman George SURNAME, CEO of COMPANYNAME, witnessed the incident and said he is shocked by it.

Table 2: English needle templates used in this work, along with their category and the article/paragraph number in which they are inserted. The same eight needles have also been translated into the other four languages.

Figure 1: Excerpts from a pair of contrastive haystacks with

Y=\textrm{Cinderfax}

X_{1}=\textrm{Delcroft}

X_{2}=\textrm{Ashwren}

. Each haystack contains up to about 25 000 words. The contents of (a) and (b) are identical except that the languages are swapped for the two needle sentences and the article they are integrated to. In the excerpts, only parts of two articles are shown. The remaining articles are in the same order and language across the two haystacks. Light gray indicates authentic news text that is part of the context (most of which is left out for brevity), while the text in black are generated from our manually constructed needle templates described in Table 2 .

	1 000 words			25 000 words
Model	Both	None	One	Both	None	One
gemma-3-27b-it	14	0	2386	0	178	2222
gemma-3-4b-it	22	22	2356	0	676	1724
Llama-3.1-8B-Instruct	9	49	2342	0	41	2359
gpt-5.2-2025-12-11	0	0	2400	13	2	2385
gpt-5-mini-2025-08-07	23	0	2377	12	0	2388

Table 3: Summary of outcomes from all conflicting multilingual haystack retrievals. We report the number of times that the model correctly identifies both answers, or retrieval fails so that none of the answers is identified, or reports only one of the two possible answers without mentioning the other.

Language pair $l_{1},l_{2}$	$P(>0)$	95% CI
Chinese vs German	99.0%	$[0.11,1.52]$
Chinese vs English	96.6%	$[-0.04,1.28]$
Chinese vs Russian	100.0%	$[2.54,4.17]$
Chinese vs Turkish	88.4%	$[-0.27,1.16]$
German vs English	58.7%	$[-0.62,0.79]$
German vs Russian	100.0%	$[1.74,3.36]$

Table 4: Estimates of the log-odds bias parameter

b_{l_{1},l_{2}}

, representing the overall bias (over all prompting languages and models) of language

l_{1}

over language

l_{2}

. Haystack size is 25 000. CI = Bayesian credibility interval.

Language pair $l_{1},l_{2}$	$P(>0)$	95% CI
Chinese vs German	87.5%	$[-0.27,1.05]$
English vs Chinese	52.8%	$[-0.61,0.66]$
Chinese vs Russian	100.0%	$[2.53,3.96]$
Chinese vs Turkish	93.7%	$[-0.14,1.16]$
English vs German	100.0%	$[0.91,2.33]$
German vs Russian	100.0%	$[2.43,3.81]$

Table 5: Estimates of the log-odds bias parameter

b_{l_{1},l_{2}}

, representing the overall bias (over all prompting languages and models) of language

l_{1}

over language

l_{2}

. Haystack size is 1000. CI = Bayesian credibility interval.

	cmn	deu	eng	rus	tur
cmn		0.9	1.1	2.6	0.7
deu	-0.9		1.3	2.0	0.1
eng	-1.1	-1.3		0.5	-1.5
rus	-2.6	-2.0	-0.5		-77.0
tur	-0.7	-0.1	1.5	77.0

Table 6: Difference in bias among eastern models over western models for choosing

l_{1}

(row) over

l_{2}

(column). A row with high values indicates a preference for the language of that row by eastern models, while a column with high values indicates a preference of the column language among western models. Text weight indicates model confidence in the sign of the difference: Bold :

p>0.99

, black:

p>0.90

, gray :

p\leq 0.90

. Values presented are posterior medians. Haystack size is 25 000.

	cmn	deu	eng	rus	tur
cmn		1.5	1.4	5.5	1.0
deu	-1.5		1.1	4.3	-0.0
eng	-1.4	-1.1		3.2	-1.4
rus	-5.5	-4.3	-3.2		-79.9
tur	-1.0	0.0	1.4	79.9

Table 7: Bias among eastern models for choosing

l_{1}

(row) over

l_{2}

(column). All other details are identical to Table 6 .

	cmn	deu	eng	rus	tur
cmn		0.6	0.3	2.9	0.3
deu	-0.6		-0.3	2.2	-0.2
eng	-0.3	0.3		2.7	0.1
rus	-2.9	-2.2	-2.7		-2.8
tur	-0.3	0.2	-0.1	2.8

Table 8: Bias among western models for choosing

l_{1}

(row) over

l_{2}

(column). All other details identical to Table 6 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	Origin
gpt-5.2	?	USA
gpt-5-mini	?	USA
gpt-5-nano	?	USA
gemma-3-4b-it	4B	USA
gemma-3-27b-it	27B	USA
Llama-3.1-8B-Instruct	8B	USA

Appendix

Table 9: Large Language Models (LLMs) used in this study.

	1 000 words			25 000 words
Model	Both	None	One	Both	None	One
Monolingual haystacks
gemma-3-27b-it	21	0	579	0	49	551
gemma-3-4b-it	8	7	585	0	189	411
Llama-3.1-8B-Instruct	7	15	578	0	6	594
gpt-5.2-2025-12-11	2	0	598	10	8	582

Appendix

Table 10: Summary of outcomes from all conflicting haystack retrievals in the monolingual setting. We report the number of times that the model correctly identifies both answers, or retrieval fails so that none of the answers is identified, or reports only one of the two possible answers without mentioning the other.

	cmn	deu	eng	rus	tur	$\Sigma$
gemma-3-27b-it
cmn	–	20	35	64	23	142
deu	6	–	9	31	6	52
eng	9	14	–	53	7	83
rus	5	13	7	–	12	37
tur	11	15	14	43	–	83

Appendix

Table 11: Total number of times, over all prompt languages, where language

L_{1}

(row) is chosen over

L_{2}

(column) in both orders of a contrastive pair. Bold: row language wins over column language. Light gray: non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 25 000.

	cmn	deu	eng	rus	tur	Mean
gemma-3-27b-it
cmn	92	85	76	60	88	80
deu	30	64	60	10	52	43
eng	52	63	69	39	50	54
rus	8	4	0	84	0	19
tur	60	57	72	30	87	61

Appendix

Table 12: Percentage of times, over all contrastive pairs and languages

L_{2}

, where language

L_{1}

(row) is chosen over

L_{2}

in both orders of a contrastive pair, when prompting using

L_{p}

(column). Bold marks the diagonal, and thus represents the proportion of times when the information in the prompting language is chosen. Black marks significant difference from 50%, light gray non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 25 000.

Model	Haystack	Both	None	One
gpt-5.2-2025-12-11	1000	1113	0	1887
gpt-5-nano-2025-08-07	1000	367	1	2632
gpt-5-nano-2025-08-07	25000	12	16	2972

Appendix

Table 13: Summary of outcomes from all conflicting haystack retrievals.

Language pair $l_{1},l_{2}$	$P(>0)$	95% CI
German vs Chinese	59.8%	$[-0.62,0.79]$
English vs Chinese	68.6%	$[-0.54,0.86]$
Chinese vs Russian	100.0%	$[2.26,3.71]$
Chinese vs Turkish	61.1%	$[-0.61,0.76]$
English vs German	88.1%	$[-0.30,1.15]$
German vs Russian	100.0%	$[2.42,3.92]$

Appendix

Table 14: Estimates of the log-odds bias parameter

b_{l_{1},l_{2}}

, representing the overall bias (over all prompting languages and models) of language

l_{1}

over language

l_{2}

. Haystack size is 2500.

Language pair $l_{1},l_{2}$	$P(>0)$	95% CI
German vs Chinese	75.6%	$[-0.54,1.06]$
English vs Chinese	50.4%	$[-0.76,0.78]$
Chinese vs Russian	100.0%	$[2.33,3.96]$
Chinese vs Turkish	64.8%	$[-0.63,0.94]$
English vs German	67.5%	$[-0.61,0.98]$
German vs Russian	100.0%	$[2.99,4.82]$

Appendix

Table 15: Estimates of the log-odds bias parameter

b_{l_{1},l_{2}}

, representing the overall bias (over all prompting languages and models) of language

l_{1}

over language

l_{2}

. Haystack size is 5000.

Language pair $l_{1},l_{2}$	$P(>0)$	95% CI
German vs Chinese	72.5%	$[-0.51,0.97]$
Chinese vs English	74.6%	$[-0.49,0.98]$
Chinese vs Russian	100.0%	$[2.61,4.32]$
Chinese vs Turkish	69.9%	$[-0.56,0.99]$
German vs English	75.4%	$[-0.51,1.06]$
German vs Russian	100.0%	$[2.48,4.21]$

Appendix

Table 16: Estimates of the log-odds bias parameter

b_{l_{1},l_{2}}

, representing the overall bias (over all prompting languages and models) of language

l_{1}

over language

l_{2}

. Haystack size is 10000.

	cmn	deu	eng	rus	tur	$\Sigma$
gemma-3-27b-it
cmn	–	5	2	39	3	49
deu	11	–	2	16	9	38
eng	4	6	–	34	10	54
rus	2	3	2	–	5	12
tur	17	4	3	19	–	43

Appendix

Table 17: Total number of times, over all prompt languages, where language

L_{1}

(row) is chosen over

L_{2}

(column) in both orders of a contrastive pair. Bold: row language wins over column language. Light gray: non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 1000.

	cmn	deu	eng	rus	tur	$\Sigma$
gemma-3-27b-it
cmn	–	2	8	44	11	65
deu	16	–	8	31	7	62
eng	20	11	–	51	15	97
rus	13	13	4	–	12	42
tur	15	7	12	34	–	68

Appendix

Table 18: Total number of times, over all prompt languages, where language

L_{1}

(row) is chosen over

L_{2}

(column) in both orders of a contrastive pair. Bold: row language wins over column language. Light gray: non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 2500.

	cmn	deu	eng	rus	tur	$\Sigma$
gemma-3-27b-it
cmn	–	9	8	41	12	70
deu	11	–	13	37	19	80
eng	24	18	–	37	18	97
rus	8	11	6	–	21	46
tur	8	6	10	31	–	55

Appendix

Table 19: Total number of times, over all prompt languages, where language

L_{1}

(row) is chosen over

L_{2}

(column) in both orders of a contrastive pair. Bold: row language wins over column language. Light gray: non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 5000.

	cmn	deu	eng	rus	tur	$\Sigma$
gemma-3-27b-it
cmn	–	10	14	44	15	83
deu	8	–	13	26	9	56
eng	8	8	–	39	6	61
rus	7	8	5	–	9	29
tur	18	8	13	26	–	65

Appendix

Table 20: Total number of times, over all prompt languages, where language

L_{1}

(row) is chosen over

L_{2}

(column) in both orders of a contrastive pair. Bold: row language wins over column language. Light gray: non-significant difference (Bonferroni-corrected

p<0.05

). Haystack size is 10000.

Prompt	cmn	deu	eng	rus	tur
gemma-3-27b-it
cmn	10	10	12	10	12
deu	12	12	12	11	12
eng	12	11	12	12	12
rus	11	11	12	12	10
tur	11	11	12	9	12

Appendix

Table 21: Total number of times a given non-conflicting was successfully retrieved in a monolingual haystack, by prompt language. Haystack size is 25 000.

Linguistic Bias in Language Models

Linguistic bias in language models studies how language varieties, identity cues, and cultural context shape model outputs and judgments.

See all 90 papers

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs
Apr 21, 2026Guy Mor-Lan, Omer Goldman, Matan Eyal +6Multilingual Language Models Multilingual Language Model Evaluation
GGoogle Research · BBar-Ilan University · CUniversity of Cambridge +1
A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics
Oct 25, 2026Wiem Ben Rim, Yao Lu, Pontus Stenetorp +1Cross-Cultural Language Model Evaluation Output Diversity in Language Models
University College London
LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge

Apr 21, 2026J/K move · Enter open · S save

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady+5

GGoogle Research · BBar-Ilan University · CUniversity of Cambridge+1

Multilingual Language Models Multilingual Language Model Evaluation Multilingual QA Language Model Bias Evaluation Linguistic Bias in Language Models

Open paper AskPDF arXiv

Multilingual large language models (LLMs) have minimized the fluency gap between languages. This advancement, however, exposes models to the risk of biased behavior, as knowledge and norms may propagate across languages. In this work, we aim to quantify models' inter- and intra-lingual biases, via their ability to answer locale-ambiguous questions. To this end, we present LocQA, a test set containing 2,156 questions in 12 languages, referring to various locale-dependent facts such as laws, dates, and measurements. The questions do not contain indications of the locales they relate to, other than the querying language itself. LLMs' responses to LocQA locale-ambiguous questions thus reveal models' implicit priors. We used LocQA to evaluate 32 models, and detected two types of structural biases. Inter-lingually, we show a global bias towards answers relevant to the US-locale, even when models are asked in languages other than English. Moreover, we discovered that this global bias is exacerbated in models that underwent instruction tuning, compared to their base counterparts. Intra-lingually, we show that when multiple locales are relevant for the same language, models act as demographic probability engines, prioritizing locales with larger populations. Taken together, insights from LocQA may help in shaping LLMs' desired local behavior, and in quantifying the impact of various training phases on different kinds of biases.

Abstract

Linguistic Bias in Language Models

Explore similar work

Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics

LLM Evaluators are Biased across Languages