Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities

Dayeon Ki; Ruochen Zhang; Silviu Cucerzan; Ryen W. White; Ning Gao

cs.CLSep 30, 2026

Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities

Authors: Dayeon Ki, Ruochen Zhang, Silviu Cucerzan, Ryen W. White, Ning Gao

Organizations: University of Maryland · Microsoft

Abstract

Large Language Models increasingly serve as interfaces for knowledge-intensive information seeking tasks across languages by synthesizing multilingual evidence. Prior work has shown that they often exhibit query-language preference -- the tendency to favor sources written in the language of the query -- but has largely examined this behavior in settings where equivalent knowledge is available across languages. However, this bias becomes consequential when sources in different languages provide incomplete or inconsistent accounts of the same fact, since the information users receive then depends on the sources a model selects to use. To characterize query-language preference under such cross-lingual knowledge disparities, we introduce Waldo, a multilingual Question-Answering (QA) benchmark constructed from Wikipedia. Waldo contains 12K QA pairs targeting knowledge gaps, where a fact is available in one language but absent in another, and knowledge conflicts, where language editions provide conflicting versions of the same fact. Evaluating eight models across five languages, we find that when one language edition merely lacks the relevant fact, models generally use evidence from the other language regardless of the query language. Under conflicting accounts, however, model responses strongly align with the document in the query language, causing semantically equivalent queries to elicit different accounts depending on the user's language. Finally, we explore two different approaches that could mitigate this preference under knowledge conflicts: a mechanistic intervention that ablates attention heads associated with query-language preference, and LoRA-based training, which reduces the preference gap by up to 61.5%.

Figures & tables

Figure 1: Overview of Waldo . For each source language (e.g., Korean), we collect its Wikipedia revisions and pair them with the corresponding English documents. For Knowledge Gap , we identify revisions containing facts present in the source-language document but absent from English, generating aligned source-language and English QA pairs for each. For Knowledge Conflict , we inject a controlled conflicting fact into the English document and generate revised English QA pairs.

Language	Date	Num	Per	Loc	Org	Misc
French	11.3	20.0	15.9	7.6	10.6	34.6
Chinese	17.5	26.3	8.6	9.7	9.6	28.3
Japanese	17.9	18.4	9.9	8.5	12.0	33.3
Korean	27.9	12.6	14.8	5.1	8.2	31.4
Bengali	14.8	25.4	10.1	6.9	8.8	34.0
Average	17.9	20.5	11.9	7.6	9.8	32.3

Table 1: Distribution of injected conflict types per language (%). Num: Number; Per: Person; Loc: Location; Org: Organization.

Figure 2: Answer agreement and probability under knowledge gap and conflict settings. A larger positive

\Delta

(src-en) denotes stronger query-language preference. Error bars are 95% bootstrap CIs. We use McNemar’s test for agreement differences and the Wilcoxon signed-rank test for probability differences, with Bonferroni correction for multiple comparisons. *: significant with

p

<

.05; **:

p

<

.01; ***:

p

<

.001; unmarked: not significant. See Appendix B.2 for per-language results.

Model	Agreement (%)			Probability (%)
Model	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{agree}}$	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{prob}}$
LLaMA-3.1	$21.0$	$86.3$	$-65.3$ ***	$41.4$	$15.4$	$26.0$ ***
Qwen-3	$28.7$	$81.6$	$-52.9$ ***	$7.9$	$4.9$	$2.9$ *
Aya-Expanse	$24.3$	$69.4$	$-45.1$ ***	$17.9$	$15.4$	$2.4$ *
GPT-4o	$15.6$	$67.7$	$-52.1$ ***	–	–	–
GPT-4.1	$25.3$	$77.9$	$-52.6$ ***	–	–	–

Table 2: Answer agreement and probability gaps under knowledge conflict for swapped roles. Same statistical test performed as in Figure 1 . *: significant with

p

<

.05; ***:

p

<

.001. No probabilities for closed-source models (–).

Figure 3: Answer agreement and probability gaps with varying

k

. Both gaps shrink as more attention heads are ablated (i.e., increasing

k

), and QPHA yields a larger reduction than the random control. Shown for LLaMA-3.1 ; see Appendix D.4 for other models.

QPHA ( $k=$ 8)
Approach	LLaMA-3.1		Qwen-3		Aya-Expanse
Approach	$\Delta_{\mathrm{agree}}$	$\Delta_{\mathrm{prob}}$	$\Delta_{\mathrm{agree}}$	$\Delta_{\mathrm{prob}}$	$\Delta_{\mathrm{agree}}$	$\Delta_{\mathrm{prob}}$
Vanilla	$63.0$	$29.9$	$59.0$	$12.1$	$58.8$	$33.5$
Random (Zero)	$61.2$	$26.3$ *	$57.0$	$12.1$	$58.6$	$30.8$ *
Random (Mean)	$59.4$ *	$28.2$	$57.5$	$11.8$	$57.5$	$31.8$
Zero	$56.3$ **	$19.4$ ***	$54.3$ *	$10.5$	$55.7$ *	$29.7$ *

Table 3: Mitigation results on answer agreement and probability gaps. For both, lower is better (↓). We perform Wilcoxon signed-rank test against Vanilla, with Bonferroni correction. *: significant with

p

<

.05; **:

p

<

.01; ***:

p

<

.001; unmarked: not significant. Appendix D.2 reports full results for

k\in\{2,4,8\}

and each query language.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Language	Script	Synthesis	Word Order	Resource Level	# Speakers	# Wikipedia Size
Indo-European	English	Latin	analytic	SVO	high	1,130M	5,758,285
	French	Latin	fusional	SVO	high	398M	2,325,608
	Bengali	Bengali	fusional	SOV	low	337M	63,762
Sino-Tibetan	Chinese	Chinese	analytic	SVO	high	1,350M	1,246,389
Koreanic	Korean	Hangul	agglutinative	SOV	mid	80M	437,373
Japonic	Japanese	Japanese	agglutinative	SOV	mid	128M	1,133,444

Appendix

Table 4: Characteristics of source languages. For each language, we show language family, script, linguistic typologies (synthesis and word order), and resource level measured by the number of speakers and Wikipedia articles ( Zhang et al., 2023 ) .

Language	Original	(1) Heuristics	(2) LLM Annotation (A.1)	(3) LLM Annotation (A.2)
French	229,438	42,446 (18.5%)	3,917 (1.7%)	504 (0.2%)
Chinese	251,136	107,486 (42.8%)	6,160 (2.5%)	1,227 (0.5%)
Japanese	257,041	112,841 (43.9%)	7,778 (3.0%)	1,125 (0.4%)
Korean	465,500	68,894 (14.8%)	5,293 (1.1%)	943 (0.2%)
Bengali	725,835	172,023 (23.7%)	5,702 (7.9%)	1,043 (0.1%)
Average		(28.7%)	(3.2%)	(0.3%)

Appendix

Table 5: Retention rates by each source language after heuristic filtering and LLM annotation. We compute each retention rate relative to the total number of revisions in the original set.

Stage	Added text	Removed text	Description
Heuristics	* 1895 * 1896 * 1914 * 1919 * 1937 * 1939 * 1945 * 1953 * 1954 * 1963 * 1973 * 1979	1895 1896 1914 1919 1937 1939 1945 1953 1954 1963 1973 1979	Remove bot edits ; Username with “TedBot” and comment as “Bot: Standardizing template names and styles”
	bgcolor=“#ccffcc”\|align=left\| \|bgcolor=#87CEEB\|AFC\|-bgcolor=“#ffffcc”\|align=left\|		Remove reverts
	[…]		Remove high-symbol, low-content edits
	bgcolor=“#ccffcc”\|align=left\| \|bgcolor=#87CEEB\|AFC\|-bgcolor=“#ffffcc”\|align=left\|		Remove markup-heavy additions
	# # # # # # # # […]		Remove repetition
LLM Annotation	Yanggu County(楊口郡)is	Yanggu County(楊口郡,) is,	Removed two commas without adding any meaningful information

Appendix

Table 6: Examples of excluded revisions during the Heuristics & LLM Annotation stage.

Language	xCOMET ( $\mathcal{Q}_{\mathrm{src}}$ , $\mathcal{Q}_{\mathrm{en}}$ )	xCOMET ( $\mathcal{A}_{\mathrm{src}}$ , $\mathcal{A}_{\mathrm{src}}$ )
French	0.971	0.990
Chinese	0.956	0.983
Japanese	0.959	0.982
Korean	0.953	0.979
Bengali	0.970	0.985
Average	0.960	0.983

Appendix

Table 7: xCOMET-XL scores by source language. We evaluate the translation quality of source-language questions (

\mathcal{Q}_{\mathrm{src}}

) and answers (

\mathcal{A}_{\mathrm{src}}

), against their English counterparts,

\mathcal{Q}_{\mathrm{en}}

and

\mathcal{A}_{\mathrm{en}}

. Higher scores indicate higher translation quality.

Lang.	Type	Title	Target fact	$\mathcal{Q}_{\mathrm{en}}$	$\mathcal{A}_{\mathrm{en}}$	$\mathcal{A}^{\prime}_{\mathrm{en}}$	Conflicting fact
French	Number	Higashi-ku, Fukuoka	* Kane Tanaka (1903-), supercentenaire japonaise et doyenne de l’humanité depuis 2018 et troisième personne vérifiée la plus âgée de tous les temps.	What was Kane Tanaka’s all-time rank among verified oldest people?	Third	Fourth	In 2018, Kane Tanaka was recognized as the fourth verified oldest person of all time.
	Misc	NOAAS Okeanos Explorer	Fichier:Hyocrinida Okeanos 2021.jpg\|Un crinoïde (Hyocrinida) observé dans l’Atlantique nord en 2021.	What kind of organism was observed in the North Atlantic in 2021 by Okeanos Explorer?	A crinoid (Hyocrinida)	A sea cucumber (Holothuroidea)	A sea cucumber (Holothuroidea) was observed in the North Atlantic in 2021 by Okeanos Explorer.
Chinese	Date	Ruud van Nistelrooy	*2024年10月，曼联主教练坦哈格被解雇，云尼斯达莱以助教身份临危受命成暂代主帅，首次领军曼联于联赛杯大胜李斯特城5:2。	When did Ruud van Nistelrooy take over as Manchester United’s interim manager after Ten Hag was fired?	October 2024	September 2024	He was originally tasked to work alongside Erik ten Hag, however following the sacking of the latter on 28 September 2024, he was appointed as interim head coach.
	Location	Lists of rail accidents	*8月12日，阿贝里奥苏格兰铁路一辆InterCity 125型列车于苏格兰阿伯丁郡斯冬希文出轨，并导致机车起火。事故造成至少3人死亡。	Where in Scotland did the Abellio ScotRail InterCity 125 derail on August 12?	Stonehaven, Aberdeenshire	Inverness, Highland	On August 12, an Abellio ScotRail InterCity 125 derailed near Inverness, Highland, Scotland, and its locomotive caught fire, killing at least three people.
Japanese	Date	Cinnamoroll	2023年3月3日、公式YouTubeチャンネルが開設。と同時にYouTube内にてシナモンの誕生日を記念してショートアニメ「シナモンアニメだもん」が開始された。	What date did Cinnamoroll’s official YouTube channel launch?	March 3, 2023	April 7, 2023	On April 7, 2023, Sanrio launched an official YouTube channel for Cinnamon to stream a short anime.
	Organization	Shintaro Tsuji	* 2020年8月27日 - 「普通はテレビ受けないの。今回は絶対に言い残したい事があって最後だから」と約30年ぶりのテレビ取材に応じ、テレビ東京で反戦を語る。	Which TV station aired Shintaro Tsuji talking about anti-war views in a rare TV interview?	TV Tokyo	Nippon TV	In a rare television interview, Tsuji spoke about his anti-war views on Nippon TV.

Appendix

Table 8: Examples of original and conflicting facts across languages and conflict types.

\mathcal{Q}_{\mathrm{en}}

: English question;

\mathcal{A}_{\mathrm{en}}

: Original English answer;

\mathcal{A}^{\prime}_{\mathrm{en}}

: Conflict-injected English answer.

Type	Model	Knowledge Cutoff	Context Window	Post-cutoff QA pairs (/12K)
Open-weight	LLaMA-3.1 8B	2023-12	128K	3,551 (30%)
	Qwen-3 8B	2023-12	128K	3,551 (30%)
	Aya-Expanse 8B	2024-06	128K	2,667 (22%)
Closed-source	GPT-4o	2023-09	128K	5,129 (43%)
	GPT-4.1	2024-05	1M	3,708 (31%)
	DeepSeek-V4-Pro	2025-05	1M	1,821 (15%)

Appendix

Table 9: List of evaluated models. We report each model’s knowledge-cutoff date, context-window size, and number of post-cutoff QA pairs. A post-cutoff QA pair is one whose timestamp postdates the model’s reported knowledge cutoff and the one which the model answers incorrectly when only given the question in both the source language and English. When available, we use non-reasoning mode and the default effort setting for closed-source models.

Type	Model	Agreement (%)			Probability (%)
Type	Model	src	en	$\Delta_{\mathrm{agree}}$	src	en	$\Delta_{\mathrm{prob}}$
Knowledge Gap
	LLaMA-3.1	$86.3$	$68.7$	$17.6$ ***	$36.8$	$20.2$	$16.6$ ***
	Qwen-3	$81.2$	$66.5$	$14.7$ ***	$14.6$	$7.1$	$7.5$ **
	Aya-Expanse	$59.8$	$41.3$	$18.5$ ***	$42.1$	$29.4$	$12.7$ ***
	GPT-4o	$93.1$	$84.7$	$8.4$ **	–	–	–

Appendix

Table 10: Retrieval-augmented robustness results. We perform the same statistical tests as in Figure 2 . Closed-source models do not expose answer probabilities (–).

Figure 4: Answer probability (

\Delta_{\mathrm{prob}}^{\mathrm{en}}

) for conflicting English-supported answer. A larger positive

\Delta

(en-src) indicates stronger query-language preference (i.e., English preference). We use Wilcoxon signed-rank test for differences, with Bonferroni correction for multiple comparisons. ***: statistically significant with

p

<

.001.

	Source-language ✓	Source-language ✗
English ✓	Knowledge parity (Appendix B.5 )	Swapped roles ( Table 2 )
English ✗	Main experiments with Waldo (§ 5 )	Not dealt in our paper

Appendix

Table 11: A 2

\times

2 view of tested variants. ✓: contains the original target fact; ✗: does not contain the target fact. For completeness, we test additional variants in Table 2 and Appendix B.5 .

Type	Model	Agreement (%)			Probability (%)
Type	Model	src	en	$\Delta_{\mathrm{agree}}$	src	en	$\Delta_{\mathrm{prob}}$
	LLaMA-3.1	$96.3$	$98.4$	$-2.1$	$34.7$	$26.5$	$8.3$ **
	Qwen-3	$95.4$	$95.9$	$-0.4$	$11.7$	$10.1$	$1.6$
	Aya-Expanse	$71.6$	$84.0$	$-12.4$	$43.7$	$34.5$	$9.2$ **
	GPT-4o	$99.0$	$99.2$	$-0.1$	–	–	–
	GPT-4.1	$99.8$	$100.0$	$-0.2$	–	–	–

Appendix

Table 12: Answer agreement (

\Delta_{\mathrm{agree}}

) and probability (

\Delta_{\mathrm{prob}}

) for knowledge parity experiment. Agreement (

\mathrm{src}

\mathrm{en}

): answer agreement; Probability (

\mathrm{src}

\mathrm{en}

): probability assigned to the source-supported answer, under source-language and English queries, respectively. We perform the same statistical tests as in Figure 2 . Closed-source models do not expose answer probabilities (–).

Knowledge Gap
Type	Model	F1 score (%)			Cite.( $\mathcal{Q}_{\mathrm{src}}$ ) (%)		Cite.( $\mathcal{Q}_{\mathrm{en}}$ ) (%)		Lang. (%)
Type	Model	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{F1}}$	$\mathrm{src}$	$\mathrm{en}$	$\mathrm{src}$	$\mathrm{en}$	$\mathrm{src}$	$\mathrm{en}$
	LLaMA-3.1	32.1	25.4	6.7	78.6	18.2	59.0	34.6	93.9	97.1
	Qwen-3	29.2	22.8	6.4	83.7	14.7	66.7	33.0	87.0	97.1
	Aya-Expanse	33.2	24.4	8.8	48.2	27.2	38.2	42.3	92.0	69.3
	GPT-4o	33.9	34.2	-0.3	94.3	5.7	89.9	10.1	99.9	94.8

Appendix

Table 13: F1 score gap, citation attribution (Cite.(

\mathcal{Q}_{\mathrm{src}}

), (

\mathcal{Q}_{\mathrm{en}}

)), and response language for each model and setting.

Language	Model	$\mathrm{PPL}(\cdot\|\mathcal{Q}_{\neg}\ell,\mathcal{D}_{\ell})$	$\mathrm{PPL}(\cdot\|\mathcal{Q}_{\ell},\mathcal{D}_{\ell})$	$\Delta_{\mathrm{PPL}}$	$p$
$\mathcal{A}_{\mathrm{en}}^{\prime}$
French	LLaMA-3.1	4.22	3.42	0.81
	Qwen-3	8.25	8.09	0.16
	Aya-Expanse	21.76	10.04	11.72
Chinese	LLaMA-3.1	3.80	2.93	0.88
	Qwen-3	8.78	5.43	3.36

Appendix

Table 14: Results for B.1 Pre-training Effect hypothesis. Wilcoxon signed-rank test performed against zero; *: statistically significant with

p

<

.05; **:

p

<

.01; unmarked: not significant.

$\mathcal{Q}_{\mathrm{en}}$
Language	LLaMA-3.1	Qwen-3	Aya-Expanse
French	$0.137$	$0.189$	$0.171$
Chinese	$0.055$	$0.008$	$0.048$
Japanese	$0.042$	$0.274\,**$	$0.073$
Korean	$-0.028$	$-0.115$	$0.138$
Bengali	$0.108$	$-0.080$	$0.132$

Appendix

Table 15: Results for B.2 Embedding Similarity hypothesis. Point-biserial correlation is performed; *: statistically significant with

p

<

.05; **:

p

<

.01; unmarked: not significant.

$\Delta_{\mathrm{length}}$
Language	LLaMA-3.1	Qwen-3	Aya-Expanse
French	$-0.117$	$-0.107$	$-0.118$
Chinese	$0.046$	$-0.227*$	$-0.255*$
Japanese	$-0.098$	$-0.112$	$-0.066$
Korean	$-0.105$	$-0.142$	$-0.124$
Bengali	$-0.082$	$0.109$	$-0.032$

Appendix

Table 16: Results for B.3 Surface Cues hypothesis. Pearson correlation is performed; *: statistically significant with

p

<

.05; unmarked: not significant.

Language	Model	$\Delta_{\mathrm{agree}}$ (Specific)	$\Delta_{\mathrm{agree}}$ (Agnostic)	Difference
French	LLaMA-3.1	58.3	45.3	0.130
	Qwen-3	48.8	40.0	0.088
	Aya-Expanse	16.3	21.8	$-$ 0.055
Chinese	LLaMA-3.1	90.0	55.0	0.150
	Qwen-3	57.5	70.0	$-$ 0.125
	Aya-Expanse	33.8	25.0	0.088

Appendix

Table 17: Results for B.4 Topic Locality hypothesis. We perform Welch’s two-sample

t

-test, where unmarked: not statistically significant.

QPHA ( $k=$ 2)
Approach	LLaMA-3.1						Qwen-3						Aya-Expanse
	Agreement			Probability			Agreement			Probability			Agreement			Probability
	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$
Vanilla	$78.7$	$15.7$	$63.0$	$47.6$	$17.7$	$29.9$	$76.9$	$17.9$	$59.0$	$17.8$	$5.7$	$12.1$	$72.4$	$13.6$	$58.8$	$58.0$	$24.5$	$33.5$
Random (Zero)	$78.6$	$15.9$	$62.7$	$47.5$	$18.2$	$29.3$	$76.4$	$19.6$	$56.8$	$17.8$	$6.8$	$11.0$	$72.4$	$13.5$	$58.9$	$57.4$	$24.3$	$33.1$
Random (Mean)	$78.4$	$15.3$	$63.1$	$47.8$	$17.8$	$30.0$	$76.5$	$18.4$	$58.1$	$18.1$	$6.7$	$11.4$	$72.7$	$13.3$	$59.4$	$57.1$	$24.0$	$33.1$

Appendix

Table 18: Effect of intervention approaches on agreement (

\Delta_{\mathrm{agree}}

) and probability (

\Delta_{\mathrm{prob}}

) gaps. For both metrics, lower is better (↓), which indicate reduced query-language preference. We perform Wilcoxon signed-rank test against Vanilla, with Bonferroni correction. *: significant with

p

<

.05; **:

p

<

.01; ***:

p

<

.001; unmarked: not significant.

QPHA ( $k=$ 8)
Approach	LLaMA-3.1						Qwen-3						Aya-Expanse
	Agreement			Probability			Agreement			Probability			Agreement			Probability
	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$	$\mathrm{src}$	$\mathrm{en}$	$\Delta$
Vanilla	$22.5$	$86.3$	$-63.8$	$30.4$	$18.5$	$11.9$	$23.6$	$77.9$	$-54.3$	$11.9$	$5.6$	$6.3$	$27.8$	$70.3$	$-42.5$	$20.9$	$17.2$	$3.7$
Zero	$24.1$	$82.1$	$-58.0**$	$28.8$	$21.1$	$7.7*$	$27.4$	$77.0$	$-49.6*$	$11.9$	$7.2$	$4.7*$	$30.6$	$65.0$	$-34.4***$	$20.8$	$19.5$	$1.3*$
Mean	$27.7$	$85.4$	$-57.7**$	$30.1$	$26.5$	$3.6**$	$27.0$	$76.1$	$-49.1*$	$12.6$	$7.3$	$5.3$	$32.0$	$69.4$	$-37.4**$	$20.8$	$18.6$	$2.2*$

Appendix

Table 19: Effect of intervention approaches on agreement (

\Delta_{\mathrm{agree}}

) and probability (

\Delta_{\mathrm{prob}}

) gaps for swapped roles. For both metrics,

\Delta

values closer to zero indicate reduced query-language preference. We perform Wilcoxon signed-rank test against Vanilla, with Bonferroni correction. *: significant with

p

<

.05; **:

p

<

.01; ***:

p

<

.001; unmarked: not significant.

Figure 5: Answer agreement and probability gap with increasing

k

. Both metrics decrease as the number of ablated attention heads (

k

) increases, with QPHA showing larger gap than random control. Top: Qwen-3 ; Bottom: Aya-Expanse .

Model	$k$	Ablated pairs ( layer , head )
LLaMA-3.1	2	(12,1), (15,9)
	4	(12,1), (15,9), (31,14), (13,31)
	8	(12,1), (15,9), (31,14), (13,31), (8,6), (16,28), (11,1), (13,19)
Qwen-3	2	(17,18), (22,1)
	4	(17,18), (22,1), (16,19), (20,10)
	8	(17,18), (22,1), (16,19), (20,10), (22,31), (17,30), (10,7), (17,20)

Appendix

Table 20: Ablated attention heads per model and

k

. LLaMA-3.1 and Aya-Expanse have a total of 32 layers, and Qwen-3 has 36 layers.

Panel of Figure 6 — Figure 6: Full attention-head score map for each model. Darker-colored cells indicate higher QPHA scores (i.e., heads more strongly associated with query-language preference).

Language	Model	Agreement (%)			Probability (%)
Language	Model	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{agree}}$	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{prob}}$
French	LLaMA-3.1	90.4	84.3	6.0	27.9	17.6	10.3
	Qwen-3	80.7	74.7	6.0	10.9	5.1	5.8
	Aya-Expanse	67.5	60.2	7.2	42.7	33.5	9.2
	GPT-4o	95.5	93.6	1.8	–	–	–
	GPT-4.1	100.0	94.0	6.0	–	–	–

Appendix

Table 21: Per-language answer agreement (

\Delta_{\mathrm{agree}}

) and probabilities (

\Delta_{\mathrm{prob}}

) under knowledge gap setting. Agreement (

\mathrm{src}

\mathrm{en}

): answer agreement; Probability

\mathrm{src}

\mathrm{en}

): probability assigned to the source-supported answer, under source-language and English queries, respectively. Closed-source models do not expose answer probabilities (–). A larger positive

\Delta

value indicates stronger query-language preference.

Lang.	Model	Agreement (%)			Probability (%)
Lang.	Model	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{agree}}$	$\mathrm{src}$	$\mathrm{en}$	$\Delta_{\mathrm{prob}}$
French	LLaMA-3.1	78.3	19.3	59.0	25.9	10.8	15.1
	Qwen-3	53.0	19.3	33.7	9.0	3.3	5.7
	Aya-Expanse	54.2	28.9	25.3	37.5	20.4	17.1
	GPT-4o	51.1	25.0	26.1	–	–	–
	GPT-4.1	72.5	20.9	51.5	–	–	–

Appendix

Table 22: Per-language answer agreement (

\Delta_{\mathrm{agree}}

) and probabilities (

\Delta_{\mathrm{prob}}

) under knowledge conflict setting. Agreement (

\mathrm{src}

\mathrm{en}

): answer agreement; Probability

\mathrm{src}

\mathrm{en}

): probability assigned to the source-supported answer, under source-language and English queries, respectively. Closed-source models do not expose answer probabilities (–). A larger positive

\Delta

value indicates stronger query-language preference.

Figure 7: Per-language examples propagated into Prompt A.1. Native speakers manually collected these examples from sourced Wikipedia revisions.

Explore similar work

Language Specific Knowledge: Do Models Know Better in X than in English?
May 21, 2025Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-TürMultilingual Language Models Language Pairs
Department of Computer Science University of Illinois, Urbana-Champaign
XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering
Aug 23, 2026Iman Barati, Arash Ghafouri, Behrouz Minaei-BidgoliMultimodal Query Multi-Hop Reasoning
School of Computer Engineering, Iran University of Science and Technology, Tehran 16846-13114, Iran
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Benchmark Multilingual
ByteDance Seed · Nanjing University · M-A-P

May 21, 2025, cs.CLJ/K move · Enter open · S save

Language Specific Knowledge: Do Models Know Better in X than in English?

Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-Tür

Department of Computer Science University of Illinois, Urbana-Champaign

Multilingual Language Models Language Pairs

Open paper AskPDF arXiv

Often, multilingual language models are trained with the objective to map semantically similar content (in different languages) in the same latent space. In this paper, we show a nuance in this training objective, and find that by changing the language of the input query, we can improve the question answering ability of language models. We make two main contributions. First, we introduce the term Language Specific Knowledge (LSK) to denote queries that are best answered in an ``expert language'' for a given LLM, thereby enhancing its question-answering ability. We introduce the problem of language selection -- for some queries, language models can perform better when queried in languages other than English, sometimes even better in low-resource languages -- and the goal is to select the optimal language for the query. Second, we introduce a variety of simple to strong baselines to empirically motivate the language selection problem (including one of our own methods called LSKExtractor). During our evaluation, we employ three datasets that contain knowledge about both cultural and social behavioral norms. Overall, the results show that principled language selection can improve the performance of a language model, and that the expected question-to-language map is not always intuitive: Gemma models know most about China and Middle East in Spanish; Qwen models know most about authority and responsibility in Arabic and Chinese. Broadly, our research contributes to the open-source development of language models that are inclusive and more aligned with the cultural and linguistic contexts in which they are deployed.

Abstract

Explore similar work

Language Specific Knowledge: Do Models Know Better in X than in English?

XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark