The World as Viewed by 136 Countries: An Open, Curated, Annotated, and Multilingual Dataset of 118 Million State Media Articles

Bryan Li; Samar Haider; Grace Kim; Fiona Luo; Jinqi Luo; Yuan Yuan; Duncan Watts; René Vidal

cs.CLOct 18, 2026

The World as Viewed by 136 Countries: An Open, Curated, Annotated, and Multilingual Dataset of 118 Million State Media Articles

Authors: Bryan Li, Samar Haider, Grace Kim, Fiona Luo, Jinqi Luo, Yuan Yuan, Duncan Watts, René Vidal,

Organizations: Google · Northwestern University · University of Pennsylvania · University of California, Berkeley ·

Abstract

State media outlets often project their governments' narratives to target both domestic and foreign audiences. Because this content is plentiful and free to access, it appears widely in the web corpora that large language models (LLMs) are trained on and that retrieval systems draw from. However, the prevalence of state media in these corpora and its potential influence on LLM outputs remain largely unexplored. In this paper, we introduce WorldStateMedia, an open dataset of 118 million state media articles from 136 countries in 114 languages. We extract these articles from existing pretraining corpora and annotate each outlet with level of state control from State Media Monitor. Using WorldStateMedia to audit these corpora, we find that, in English web crawls, the share of state media pages from editorially controlled (vs. independent) outlets rose from 39% in 2013 to 93% in 2025. In several low-resource languages, state media make up a substantial share of all web data, such as 17.8% for Uyghur and 7.8% for Belarusian. To characterize the differing narratives of news outlets, we compare Russian and Ukrainian state media in English and in their native languages, revealing asymmetric topical focus and opposing framings of contested issues such as Crimea. Furthermore, in a retrieval-augmented generation (RAG) experiment on territorial disputes, retrieving a country's state media shifts model answers toward that country's claims, most strongly for English-language (foreign-targeted) media. We release WorldStateMedia to support multilingual, cross-national insights on state media at https://huggingface.co/collections/manestay/worldstatemedia

Figures & tables

Dataset	Languages	Size	Countries	Years	Full Text	Deduped	Influence Focused	License
WorldStateMedia	114	118M art.	136	2013–25	Yes	Yes	Yes	ODC-By
Large Multilingual News Corpora
GDELT Leetaru and Schrodt (2013)	100+	$\sim$ 2.5TB/yr	230+	1979–	Yes	No	No	Unrestricted
CC-News Common Crawl (2026)	100+	$\sim$ 1.35B art.	220+	2016–	Yes	No	No	CC ToU
Media Cloud Roberts et al. (2021)	20+	1.7B+ art.	Global	2010–	Mixed	Yes	No	API
Event Registry Leban et al. (2014)	60+	150k src.	Global	2014–	Yes	Yes	No	Commercial

Table 1: Comparison of multilingual news and propaganda datasets with WorldStateMedia . Existing multilingual news corpora provide broad global coverage but lack focus on influence, and have more restrictive licenses. Existing propaganda datasets are more limited in size and language coverage.

Country	Media company	Main assets	Typology	Region	URL_dict
Argentina	Radio y Televisión Argentina (RTA)	Television : Televisión Pública (Canal 7), Canal 12; Radio : Radio Nacional	SC	South America	{Televisión Pública (Canal 7) : tvpublica.com.ar , Radio Nacional : radionacional.com.ar}
China	Xinhua News Agency	News agency : Xinhua; Publishing : Reference News (Cānkǎo Xiāoxī); Television : CNC World	SC	East Asia	{Xinhua : xinhuanet.com/english , Reference News (Cānkǎo Xiāoxī) : cankaoxiaoxi.com}
South Korea	Yonhap News Agency	News agency : Yonhap; Television : Yonhap News TV	IP	East Asia	{Yonhap : en.yna.co.kr Yonhap News TV : www.yonhapnewstv.co.kr}
France	France Médias Monde	Radio : Radio France Internationale (RFI), Monte Carlo Doualiya (MCD) Television : France24	IS	Western Europe	{Radio France Internationale (RFI) : rfi.fr/fr , Monte Carlo Doualiya (MCD) : mc-doualiya.com , France24 : france24.com/fr}
Kenya	Kenya News Agency (KNA)	News agency : KNA	SC	Eastern Africa	{KNA : kenyanews.go.ke}
Qatar	Al Jazeera Media Network (AJMN)	Television : Al Jazeera Arabic, Al Jazeera English	SC	Middle East	{Al Jazeera Arabic : aljazeera.net , Al Jazeera English : aljazeera.com}

Table 2: Excerpted entries from the state media websites table, along with annotations. Although our analyses use binary Ind / Cnt labels, here we show the full 7 label typology from SMM.

	English				Multilingual					English				Multilingual
Country	Rank	Pages	Ind	Cnt	Rank	Pages	Ind	Cnt	Country	Rank	Pages	Ind	Cnt	Rank	Pages	Ind	Cnt
USA	1	15.7m	41.7	58.3	36	208k	0.4	99.6	France	14	1.36m	100.0	0.0	5	2.08m	100.0	0.0
Russia	2	8.26m	0.0	100.0	1	6.22m	0.0	100.0	Egypt	15	1.09m	0.0	100.0	30	296k	0.0	100.0
Australia	3	6.67m	100.0	0.0	72	35k	100.0	0.0	Cambodia	16	1.09m	0.0	9.6	52	82k	0.0	100.0
UK	4	6.35m	100.0	0.0	53	81.3k	100.0	0.0	Germany	17	1.02m	100.0	0.0	3	2.49m	100.0	0.0
Canada	5	4.98m	100.0	0.0	26	363k	100.0	0.0	Switzerland	23	466k	100.0	0.0	11	900k	100.0	0.0

Table 3: Dataset statistics for the 26 countries that rank in the top 16 of either the English or multilingual split of WorldStateMedia (136 countries total). For each country and split, we show the ordinal rank, the number of pages, and the percentage of pages from either independent ( Ind ) or controlled ( Cnt ) state media.

Figure 1: For WorldStateMedia en , we graph three proportions over time: within state media pages, from controlled ( Cnt ) vs. independent ( Ind ) sources; and state media pages over all pages.

Lang.	Hit %	Cnt	Language	Hit	Cnt
Uyghur	17.8	58.5	Māori	6.8	0.0
Turkmen	8.3	62.8	Amharic	6.5	43.7
Belar.	7.8	59.9	Sc. Gaelic	6.4	0.0
Norweg.	7.7	0.0	Pashto	5.9	31.2
Serbian (Lat.)	7.0	95.5	Arabic	5.3	83.3
Hausa	7.0	2.7	Kazakh	5.3	68.0

Table 4: The 14 languages in WorldStateMedia multi with the highest proportion of pages coming from state media outlets (Hit %), and of those the proportion of those from Cnt outlets.

Panel of Figure 2 — Figure 2: Share of extracted facts from Russian and Ukrainian Cnt media across Crimea-related topics. $n$ indicates the number of facts in each topic.

Figure 3: A 2-by-2 grid of word clouds for the corpora from Section 5 . Rows indicate English-language or native-language, while columns indicate from Russia or Ukraine sources. Word size is proportional to weighted log-odds; larger words indicate more distinctive vocabulary. For readability, native words are translated to English.

Figure 4: Concurrence score (CS) metrics with retrieval varying across sources : Wikipedia from Li et al. (2025) , English (foreign) state media, and multilingual (domestic) state media; and across perspectives : topic-level ( rel_langs ) vs. query-level ( qlang ). Foreign state media at the query level yields the most divergent responses across languages, and topic-level retrieval is less divergent than query-level.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Prompt for obtaining URLs for the assets of a state media company.

Split	URL	Date	Language	Country	Subregion	Asset	Typology
2017-30	azertag.az	2017-07-22	azj_Latn	Azerbaijan	Middle East	Azertac	SC
Text	Bakı, 8 iyul, Az e rtac T e bii bal arılar t e r e find e n çiç e kl e rd e n v e meyv e tumurcuqlarından alınaraq yığılan nektarlardan formalaşmış e v e zsiz qida m e hsuludur. M e nş e yin e gör e o çiç e k v e şir e balları kimi növl e r e …
2017-34	mediapermata.com.bn	2017-08-16	zsm_Latn	Brunei	Southeast Asia	Media Permata	SC
Text	Hanoi, Vietnam mengekalkan pengharaman menangkap ikan lepas pantai di empat wilayah tengah setahun selepas kilang keluli milik Taiwan mengeluarkan toksik ke laut dan mengakibatkan bencana persekitaran paling teruk di negara itu…
2013-20	thepeninsulaqatar.com	2013-05-18	en_Latn	Qatar	Middle East	The Peninsula	Ca
Text	- Special Pages London: Manchester United, Manchester City, Arsenal and holders Chelsea all avoided each other in yesterday’s FA Cup fourth-round draw at Wembley, but only Chelsea and City are assured of a future in the competition…

Appendix

Table 5: Sample entries from WorldStateMedia , two from each split. We show the article text along with the metadata of split, URL, data, language, country, region, asset, and typology.

Lang	Hit %	Total Pages	Ind	Cnt	Lang	Hit %	Total Pages	Ind	Cnt
rus	1.0	699m	4.2	94.0	afr	0.1	1.99m	0.0	100.0
cmn	0.2	636m	24.7	74.8	tel	0.0	1.96m	5.1	94.9
deu	0.4	496m	93.6	6.3	hye	0.6	1.76m	0.2	99.8
spa	0.5	441m	17.2	82.4	khk	4.0	1.62m	0.0	100.0
jpn	0.0	400m	89.8	10.2	khm	6.8	1.59m	14.9	85.1
fra	0.8	360m	86.5	13.2	eus	1.5	1.57m	99.9	0.1

Appendix

Table 6: For each of the 114 languages from WorldStateMedia multi , the hit % from state media outlets over all pages in Fineweb-2, the total pages (all content types), and the proportions from independent ( Ind ) and controlled ( Cnt ) state media. Percentages may not sum to 100 where some outlets lack a typology label.

Split	Hit %	Total Pages	Ind	Cnt	Split	Hit %	Total Pages	Ind	Cnt
2013-20	0.4	215m	61.2	38.6	2019-13	0.2	219m	33.5	64.3
2013-48	0.4	220m	72.9	26.7	2019-18	0.3	223m	36.3	61.4
2014-10	0.4	217m	65.6	34.0	2019-22	0.3	227m	35.1	63.2
2014-15	0.4	200m	63.2	36.3	2019-26	0.3	217m	37.2	61.6
2014-23	0.4	234m	68.4	31.4	2019-30	0.3	219m	36.3	62.1
2014-35	0.4	216m	64.8	34.9	2019-35	0.3	239m	37.3	61.5

Appendix

Table 7: For each Common Crawl dump of WorldStateMedia en , the hit % from state media outlets over all pages in English Fineweb, the total pages (all content types), and the proportions from independent ( Ind ) and controlled ( Cnt ) state media.

	English				Multilingual					English				Multilingual
Country	Rank	Pages	Ind	Cnt	Rank	Pages	Ind	Cnt	Country	Rank	Pages	Ind	Cnt	Rank	Pages	Ind	Cnt
United States	1	15.7m	41.7	58.3	36	208k	0.4	99.6	Myanmar	69	32.5k	0.0	100.0	120	578	0.0	100.0
Russia	2	8.26m	0.0	100.0	1	6.22m	0.0	100.0	Angola	70	30.6k	0.0	100.0	51	84.8k	0.0	100.0
Australia	3	6.67m	100.0	0.0	72	35k	100.0	0.0	Palestine	71	29.1k	0.0	100.0	74	32.6k	0.0	100.0
United Kingdom	4	6.35m	100.0	0.0	53	81.3k	100.0	0.0	Nepal	72	29k	0.0	99.2	87	18.5k	0.0	80.8
Canada	5	4.98m	100.0	0.0	26	363k	100.0	0.0	Malawi	73	28.8k	0.0	100.0	123	262	0.0	100.0

Appendix

Table 8: Dataset statistics for WorldStateMedia en and WorldStateMedia multi over all 136 countries. For each country and split, we show the ordinal rank, the number of pages (state media only), and the percentage of articles from either independent ( Ind ) or controlled ( Cnt ) state media. Percentages may not sum to 100 where some outlets lack a typology label.

Figure 6: Illustration of the 6 retrieval settings studied in § 6 , which vary sources (columns) and perspectives (rows). For each box, its fill color indicates the language, while its outline color indicates the source. The Wikipedia settings are single-source, and the state media settings are multi-source.

Full Metric Name	Formula			Attribute
Concurrence Score	CS $(c_{a},c_{b})$	$=$	$100\cdot\begin{cases}1&\text{if }c_{a}=c_{b},\\ 0&\text{otherwise}\end{cases}$
Knowledge-base CS	KB_CS $(\mathcal{R}_{t})$	$=$	CS $(c^{KB},c_{\text{en}})$	Factuality
Controller CS	Con_CS $(\mathcal{R}_{t})$	$=$	CS $(c^{KB},c_{\text{con}})$
Non-controller CS	Non_CS $(\mathcal{R}_{t})$	$=$	$\displaystyle\dfrac{1}{\|C_{\text{non}}\|}\sum_{c\in C_{\text{non}}}\texttt{CS}(c^{KB},c)$
Delta CS	$\Delta$ _CS $(\mathcal{R}_{t})$	$=$	$\dfrac{\texttt{Con\_CS}(\mathcal{R}_{t})-\texttt{Non\_CS}(\mathcal{R}_{t})}{\texttt{Non\_CS}(\mathcal{R}_{t})}$	Language Fairness (Geopolitical Bias)
Consistency CS	Cst_CS $(\mathcal{R}_{t})$	$=$	$\dfrac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{\begin{subarray}{c}j=1,j\neq i\end{subarray}}^{n}\texttt{CS}(c_{i},c_{j})$	Consistency

Appendix

Table 9: Formulas for concurrence score ( CS ) metrics. Each metric is calculated over LLM’s multilingual response set for a territory

t

, designated as

\mathcal{R}_{t}=c_{1},...,c_{n}

. The response in the controller language is

c_{\text{con}}

, and the set of responses in non-controller languages is

C_{\text{non}}=c_{i},...,c_{j}

. The gold response is

c^{\text{KB}}

Panel of Figure 7 — Figure 7: Topic skew between Russia and Ukraine SC media. Each bar is one fact cluster, positioned by its log-odds of Ukraine-vs.-Russia representation (right/blue over-indexes in Ukraine SC, left/red in Russia SC). $n$ is the number of extracted facts in the cluster (pooled across both sources), a measure of topic volume that is independent of the bar length.

Russia SC	Ukraine SC
2014 Crimean referendum
[claim] The Crimea peninsula rejoined Russia in March.	[claim] Russia annexed Ukraine’s Crimea peninsula in early 2014.
[claim] The Crimean parliament voted overwhelmingly to hold a referendum.	[claim] The March 16, 2014 referendum in Crimea and Sevastopol was conducted under pressure from the Russian military and with the support of Russian propagandist media.
Crimean Tatar Mejlis
[attribution] Crimea’s Prosecutor Natalia Poklonskaya warned Refat Chubarov early in July that the Crimean authorities might recognize the Crimean Parliament (Mejlis) to be an extremist organization.	[attribution] Refat Chubarov is the Head of the Mejlis of the Crimean Tatar people and a Member of Parliament.
[claim] Refat Chubarov, the chief of the Crimean Parliament (Mejlis), has been barred from entering Crimea for five years.	[attribution] Deputy Chairman of the Mejlis of the Crimean Tatar people Akhtem Chiygoz stated that the Russian Federation is conducting state terrorism in Crimea against the indigenous population.

Appendix

Table 10: Representative decontextualized facts extracted from Russia and Ukraine SC English corpus on three Crimea related topics. Bracketed tags represent the automatically assigned fact categories from our pipeline. The two sources often cover the same events with divergent terminology and framing.

Russia SC	Ukraine SC
2014 Crimean referendum
[claim] Crimea became a Russian region after a referendum held in March 2014.	[claim] Crimea, along with the “LNR” and “DNR,” are considered temporarily occupied territories of Ukraine.
[statistic] In the March 2014 referendum, 96.77% of voters in the Republic of Crimea and 95.6% of residents in Sevastopol voted to join Russia.	[claim] The annexation of Crimea and the subsequent attempt to create “Novorossiya” are described as brazen actions by Russia.
Crimean Tatar Mejlis
[attribution] Poklonskaya stated that the Mejlis exists to incite the population, commit acts of sabotage, promote war, incite hatred, and disgrace the Crimean Tatar people and their symbols.	[attribution] Refat Chubarov, the head of the Mejlis of the Crimean Tatar people, believes the UN resolution’s importance lies in enabling Ukraine to demand that specialized structures and international organizations send monitoring missions to the peninsula.
[claim] Refat Chubarov and Mustafa Dzhemilev were banned from entering Russia for five years starting in 2014 due to extremist statements that incited national discord.	[claim] Russia must immediately revoke the decision to declare the Mejlis of the Crimean Tatar People an extremist organization and ban its activities, and also annul the decision to prohibit Mejlis leaders from entering Crimea.

Appendix

Table 11: Representative decontextualized facts extracted from Russia and Ukraine native-language SC media on three Crimea topics. Bracketed tags represent the auto-assigned fact categories from our pipeline. The two sources often cover the same events with divergent terminology and framing.

Russia SC
Outlet	Foreign	Domestic	Total
	(EN)	(native)
rt.com	105,579	–	105,579
tass.com	50,191	1	50,192
sputniknews.com	43,491	1	43,492
rbth.com	15,897	1	15,898

Appendix

Table 12: Corpus composition by outlet and split. Counts are URL-deduplicated articles. Foreign/domestic split is based on the article’s publication language (English

\rightarrow

foreign-targeted, Russian or Ukrainian

\rightarrow

domestic). Outlets with fewer than 20 articles are grouped into “Other.”

Panel of Figure 8 — Figure 8: Prompt for decontextualized fact extraction.

Figure 9: For articles on topic Crimea, frequency of each query being pursued by the information pursuit system (query descriptions are in Table 13 ).

Category	ID	Query
Territorial Status Framing	Q1	Does the article describe Crimea as occupied territory?
	Q2	Does the article describe Crimea as part of Russia?
	Q3	Does the article frame Crimea’s status change as a reunification?
	Q4	Does the article reference the 2014 Crimean referendum?
Agency Attribution	Q5	Does the article frame Russia as the aggressor in the conflict?
	Q6	Does the article report Ukrainian military successes against Russian targets?

Appendix

Table 13: Full query bank used in the Information Pursuit algorithm of Crimea-related articles. Each row corresponds to one binary yes/no question posed to the LLM answerer. The Category column groups queries by the narrative framing dimension they target.

Political Bias in Language Models

Political bias in language models concerns how training, prompts, languages, and interactions shape ideological outputs and unequal treatment.

See all 61 papers

May 18, 2026

Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles

Large-scale news corpora support a wide range of research in Computational Social Science and NLP, yet access remains constrained: commercial archives impose prohibitive costs and licensing restrictions, while open alternatives like Common Crawl's CC-News require terabyte-scale storage and computationally intensive processing. We present Infini-News, a retrieval toolkit and index for the entire CC-News archive from August 2016 to the latest available snapshot. Our contributions are threefold. First, we extract, clean the text, and parse the structured metadata of over 1.35B articles. Second, we enrich the corpus with language detection using three frontier language classifiers (GlotLID, lingua, and CommonLingua), and with multi-source geographic attribution that resolves a country of origin for 83.4% of articles across 222 countries. Third, we construct Infini-gram indexes: suffix-array structures that let researchers search the full archive for arbitrary text patterns in sub-second time. Together, these resources lower the barrier to longitudinal, cross-national media research.

Ruggero Marino Lazzaroni, Jana Lasser, Kirill Solovev

University of Graz, Austria

Information Retrieval

Ask PDF

Sep 6, 2026

LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages

Large language models (LLMs) are increasingly used to generate media, but whether their content perpetuates gender stereotypes is unknown: standard benchmarks rely on selection-based formats rather than long-form generation, and surveyed baselines for local gender associations are scarce outside the West. We collect gender associations for 22 occupational and domestic roles from 695 respondents across the United States, India, Kenya, and Nigeria, and evaluate eight LLMs under two regimes: direct questioning and media generation. Models track the surveyed associations under direct questioning but skew substantially more male under media generation in major local-language cells, consistent with the male bias documented in human-produced media. Outside the US, the shift is much smaller and non-significant under English prompting, so English-only or country-agnostic evaluation would miss this bias in the languages where these models are most deployed. Instruction prompting reduces the shift directionally, but trades off against alignment with the surveyed associations. Evaluating LLM gender bias for global deployment therefore requires generation-format testing, local-language prompting, and locally-collected human baselines.

Sharif Kazemi, Tanya Popli, Neil K. R. Sehgal +7

World Bank Group · Cornell University · University of Pennsylvania +2

Gender Bias in Language Models Language Model Bias Evaluation Language Model Generation Evaluation Cross-Cultural Language Model Evaluation

Ask PDF

Apr 30, 2026

DeGenTWeb: A First Look at LLM-dominant Websites

Many recent news reports have claimed that content generated by large language models (LLMs) is taking over the web. However, these claims are typically not based on a representative sample of the web and the methodology underlying them is often opaque. Moreover, when aiming to minimize the chances of falsely attributing human-authored content to LLMs, we find that detectors of LLM-generated text perform much worse than advertised. Consequently, we lack an understanding of the true prevalence and characteristics of LLM content on the web. We describe DeGenTWeb which systematically identifies LLM-dominant websites: sites whose content has been generated using LLMs with little human input. We show how to adapt detectors of LLM-generated text for use on web pages, and how to aggregate detection results from multiple pages on a site for accurate site-level categorization. Using DeGenTWeb, we find that LLM-dominant sites are highly prevalent both in data from Common Crawl and in Bing's search results, and that this share is growing over time. We also show that continuing to accurately identify such sites appears challenging given the capabilities of the latest LLMs.

Sichang Steven He, Calvin Ardi, Ramesh Govindan +1

University of Southern California · Unaliated

AI-Generated Text Detection

Ask PDF

Abstract

Political Bias in Language Models

Explore similar work

Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles

LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages

DeGenTWeb: A First Look at LLM-dominant Websites