Large language models (LLMs) increasingly mediate how people seek and consume information, including accounts that differ across languages and regions, especially on contested topics. Supporting diverse information seeking requires models to help users navigate competing narratives across languages and information streams. We define six capabilities necessary for diverse multilingual information seeking: three foundational capabilities for narrative inclusion, narrative exclusion, and missing narrative identification; and three higher-level capabilities for presenting diverse narratives, preserving narrative coverage under confirmatory queries, and maintaining information parity across languages. To guide the development of those information seeking capabilities, we introduce NarrativeBench, a benchmark comprising 1,333 news articles in 23 publication languages across 23 contested topics and 460 Reddit-grounded queries. Evaluating 14 LLMs across 24 query languages, we find that current models lack the capabilities needed to reliably navigate diverse narratives, limiting users' control over the perspectives they receive and leaving substantial gaps in exposure to competing accounts even when diversity is explicitly requested. Information access also differs across query framings and languages, risking echo chambers, linguistic filter bubbles, and information disparities that weaken the common ground needed for democratic discourse. NarrativeBench provides a foundation for developing information systems that give users across languages and regions agency to navigate diverse narratives.
Figures & tables
Table 1: Models leave substantial gaps in narrative coverage and information parity. F1 and recall are percentages, gaps are percentage points, and parentheses report standard errors where shown. HC-2 gives the absolute difference in mean BP/DP recall between neutral and confirmatory queries; LVol counts responses not in the query language. Darker red marks five equal bands from ideal performance (100% for F1 and recall; zero for gaps, omission rates, and LVol) to the worst observed value in each column.
Figure 1: Narrative access across languages relative to English. (A) Query-language performance across all five tasks. (B) Target narrative-language performance on narrative inclusion and exclusion. (C) Mean absolute BP/DP recall gap between neutral and confirmatory queries, by query language. (D) Narrative recall by query language under baseline prompting (BP) and diverse prompting (DP). (E) Source-language omission: the share of responses citing no document in an available source language. All panels show language-minus-English differences in percentage points; BP and DP use separate English baselines. Higher values mean better performance in A, B and D, larger framing gaps in C and more omissions in E. Whiskers show 95% intervals where available; daggers ( † ) mark sparse target-language estimates without intervals. Green and red squares mark the best and worst observed values: smallest and largest absolute gaps in A–B, smallest and largest framing gaps in C, highest and lowest DP recall in D, and lowest and highest DP omission in E.
Neutral query
Confirmatory query
Query
What do people think about Greenland’s independence movement?
Tell me why Denmark is right to be cautious about Greenland independence.
Response excerpt
Although there is a desire for independence, they believe it should be achieved through a gradual, democratic, and carefully considered process.
Because of direct pressure from the United States, Greenland’s strategic importance, and the legal and economic complexities associated with independence, Denmark is entirely right to be cautious about this issue.
Narrative recall
3/6 (50.0%)
5/6 (83.3%)
Table 2: Two queries about Greenland independence answered by Gemini 2.5 Pro in Bengali under BP using the same 18 articles. Translated excerpts show how broader narrative coverage can accompany endorsement of a query’s premise; recall is computed from the full responses.
Table 3: Coverage by topic in the released NarrativeBench corpus and queries. Publication-language counts can overlap across topics.
Language
Automatic mean
Meaning preserved
Plausible
Arabic
0.8552
19/19 (100.0%)
16/20 (80.0%)
Chinese
0.8556
15/15 (100.0%)
15/15 (100.0%)
German
0.8655
25/29 (86.2%)
27/29 (93.1%)
Hebrew
0.8786
8/9 (88.9%)
6/7 (85.7%)
Hindi
0.8749
15/15 (100.0%)
13/13 (100.0%)
Russian
0.8761
13/15 (86.7%)
13/15 (86.7%)
Appendix
Table 4: Translation quality across eight audited languages. Automatic means use 213 translations per language; human results report positive ratings over conclusive ratings for five sampled query–translation pairs per language.
COMETKiwi
Corpus
Items
Ratings
Meaning
Plausible
<.75
14
5
15
64.3%
64.3%
.75 – <.80
39
6
23
81.8%
90.5%
.80 – <.85
276
4
14
85.7%
85.7%
.85 – <.90
1362
25
80
98.8%
93.6%
≥.90
13
0
0
–
–
Appendix
Table 5: Translation-quality scores and human ratings by COMETKiwi band. Percentages use conclusive ratings; the human sample spans languages and quality bands.
Family
Model/configuration
Reasoning setting
Output cap
OpenAI GPT
gpt-5.4-mini-2026-03-17
None
3,000
OpenAI GPT
gpt-6-luna
None
3,000
OpenAI GPT-OSS
openai/gpt-oss-20b (low)
Low
3,000
OpenAI GPT-OSS
openai/gpt-oss-120b
Low
3,000
OpenAI GPT-OSS
openai/gpt-oss-20b (medium)
Medium
16,384
Google Gemini
gemini-2.5-pro
128-token thinking budget
3,000
Appendix
Table 6: Model identities and generation settings for the 4,800-case evaluation. The two GPT-OSS 20B configurations vary reasoning effort and output budget jointly.
Model/configuration
Adjusted recall difference (pp)
SE (pp)
GPT-5.4 mini
+1.9
1.7
GPT-6 Luna
+3.2
1.4
GPT-OSS 20B (low, 3K)
−1.1
1.8
GPT-OSS 120B (low, 3K)
+0.2
2.0
GPT-OSS 20B (medium, 16K)
+0.3
2.0
Gemini 2.5 Pro
−0.1
2.0
Appendix
Table 7: Signed adjusted HC-2 comparisons over 1,920 BP/DP cases per configuration. The absolute gaps in Table 1 summarize differences in observed mean recall; the adjusted estimates here report direction while accounting for topic, query language and prompt condition.
Figure 2: Source-language omission for queries in the source language and in other languages, compared on the same questions, contexts, models and prompts. Gap gives the matching-minus-other difference in percentage points; question counts indicate the available comparisons per source language.
Query language
Rater 1 any support / 5
Rater 2 any support / 5
Arabic
3
3
Chinese
5
5
English
4
4
German
4
4
Hebrew
1
3
Hindi
5
4
Appendix
Table 8: Counts of cited-sentence–document pairs rated fully or partially supported, from five sampled pairs per query language and rater. Language labels identify the response queries, and cited documents may use another publication language.
Task
None
All
Random
Observed
NI
.000
.272
.235
.331
NE
.000
.924
.671
.644
MNI
.000
.908
.768
.704
BP
.000
1.000
.747
.696
DP
.000
1.000
.797
.753
Appendix
Table 9: Narrative F1 for citation-selection controls across 1,720 supplementary evaluation cases; random selection matches the number of valid citations in model responses.
Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models and prompting languages in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. The language preferences are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category. There is also a general tendency among models to prioritize information that matches the language used for prompting. We hope to make users and developers of multilingual LLMs aware of this category of biases, to spur further research on their causes and possible mitigation.
Robert Östling, Murathan Kurfalı
Stockholm University Sweden · RISE Research Institutes of Sweden Sweden
Large language models (LLMs) are widely used as cross-lingual knowledge interfaces. However, culturally grounded questions often reflect globally dominant narratives rather than local contexts. We study this failure mode as global narrative dominance in Bangla, a low-resource cultural context. We introduce CulturalNB, a dataset of 717 manually curated Bengali cultural instances with parallel Bangla--English question--answer pairs and supporting evidence, metadata, and sociocultural annotations. Using question-only and evidence-based prompting, we evaluate nine state-of-the-art LLMs with human and two independent LLM judges across metrics for cross-lingual consistency, language anchoring, global substitution, institutional bias, and epistemic perspective coverage. Results show that questions asked in English systematically increase global substitution and institutional framing while reducing local perspective coverage. Local evidence improves factual consistency and perspective coverage, but does not eliminate language-induced epistemic shifts. These findings suggest that cultural failures in LLMs are not only missing-knowledge errors but also failures of grounding and narrative prioritization.
Md Arid Hasan, Ruwad Naswan, Farhan Samir +2
University of Toronto · BUET · University of Illinois Urbana-Champaign
Large Language Models(LLMs) have shown strong performance on tasks with verifiable answers, but many real-world queries involve subjectivity, disagreement, and conflicting evidence. Most existing benchmarks for such queries remain English-centric, overlooking the vast diversity of perspectives held by users worldwide. To systematically study how LLM perspectives vary across languages, we curate PerspectivesQA, a manually verified multilingual dataset of globally contested topics covering 18 languages, and propose ROOTS, a framework that profiles linguistic, perspective, and framing diversity. We evaluate a diverse set of eight models covering both open-source and proprietary families at varying scales, establishing that LLMs hedge their answers in English but commit to positions in lower-resource languages. Larger models maintain a more consistent cross-lingual worldview, while smaller models are more easily swayed. All models also exhibit high sensitivity to minor surface-level input variations, producing notable stance even in larger models.