FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

Yusser Al Ghussin; Eva Gavaller; Cristina España-Bonet; Josef van Genabith; Simon Ostermann

cs.CLSep 21, 2026

FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

Authors: Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet, Josef van Genabith, Simon Ostermann

Organizations: Saarland University · German Research Center for Artificial Intelligence (DFKI) · Barcelona Supercomputing Center (BSC-CNS), Barcelona, Catalonia, Spain

Abstract

Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely indexed with comparable metadata. Benchmarks increasingly target culturally situated phenomena at the level of languages, regions, and locale-specific practices, while web-scale corpora are usually organized only by language. A shared culture-language-region layer makes these resources comparable, enabling audits of whether a target cultural phenomenon is represented in pretraining data, evaluated by benchmarks or both. To this end, we introduce FineWeb-CLaR, a large-scale annotated dataset derived from FineWeb and FineWeb-2 that places web documents on a shared culture-language-region axis for corpus auditing and benchmark alignment. FineWeb-CLaR annotates the full 30.9B-document collection from FineWeb and FineWeb-2 with URL-derived region labels and cultural-topic provenance. Our region resolver assigns a non-empty region to 25.61% of documents (7.92B). For cultural-topic analysis, we induce locale-specific topics and project them onto the 14 leaves of the Cultural Taxonomy of Liu et al. (2025), producing Locale Topic Distributions (LTDs) for corpus-side comparison. We also annotate 277 cultural NLP benchmarks with the same taxonomy, language coverage, and region coverage. Together, these resources enable direct comparison between corpus-side pretraining evidence and benchmark-side evaluation coverage.

Figures & tables

#	Source	Signal	Conf
1	query	explicit locale parameter	high
2	host_path	lookup table	inherited
3	host	lookup table	inherited
4	domain	lookup table	inherited
5	url_hint	curated URL token	url_hint
6	tld	non-branded ccTLD	ccTLD

Table 1: Search-time priority chain for region resolution, with lookup-backed sources inheriting table confidence (high, medium or low) and fallback sources receiving weak confidence labels.

Setting	Source / tier	Documents	%
	query	15,283,520	0.05
lookup-table	host_path	222,919,273	0.72
	host	3,458,247,367	11.19
	domain	946,609,661	3.06
fallback	url_hint	342,035,324	1.11
	tld	2,932,873,160	9.49

Table 2: Region-resolution diagnostics on the full FineWeb + FineWeb-2 release, by resolution source (top) and released confidence tier (middle). high + medium + low (15.02%) equals lookup-table coverage plus query ( high includes query ); the weak url_hint and ccTLD tiers map one-to-one onto the url_hint and tld source rows and are not repeated.

Tier	IP	og:locale	All $=$	All $\neq$
high	21.9	41.4	9.1	14.4
medium	19.8	16.1	3.6	19.0
low	17.1	9.6	3.8	22.6
broad-only	37.8	17.1	9.3	33.2

Table 3: Resolver evaluation against independent automatic references on stratified random samples of 1,000 URLs per tier, comparing three notions of “region”. Cells are percentages of the tier’s URLs with the required signal(s) available. All

=

/ All

\neq

: all three sources coincide / are pairwise distinct.

Panel of Figure 1 — Figure 1: Cultural Taxonomy label space used for corpus-side topic alignment.

Locale	Topic	CT leaf
en-US	Christian Religion	Values-General
en-US	Restaurant Dining	Concepts
en-US	US Politics	Demographics
ar-EG	Travel Destinations	Concepts
ar-EG	Sports Media	Context
zh-CN	Digital Media & Devices	Artifacts

Table 4: Example FASTopic topics and final CT leaf assignments for selected locales.

Panel of Figure 2 — Figure 2: Number of audited benchmark datasets per CT leaf ( $n=277$ ).

Panel of Figure 3 — Figure 3: Coverage over CT leaf $\times$ region cells on the full release (30.9B documents). A cell is corpus-covered if at least $k=50$ documents support the CT-leaf/region pair, and benchmark-covered if at least one audited benchmark targets it. All 266 cells are corpus-covered; light cells (56. 21%) have corpus evidence but no audited benchmark.

Panel of Figure 4 — Figure 4: CT leaf $\times$ locale coverage for the smallest topic-modeled main-language locales on the full release. Locale-level analysis reveals the double blind spots and benchmark-only cells that macro-region aggregation hides.

Panel of Figure 5 — Figure 5: Within-language LTD divergence (JSD) across all languages with $\geq 2$ locales: per-language median (dots) and IQR (bars), with illustrative same-language locale pairs labelled. The shaded band and dashed line show the split-half noise floor ( $p_{95}=1.9\times 10^{-3}$ ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Rule	Evidence pattern	Weight
A1	ccTLD + HTML lang + Content-Language agree	3.0
A2	ccTLD + one header agree	2.5
A3	ccTLD beats high-resolution default	2.0
A4	ccTLD overrides country-specific header	1.0
A5	ccTLD alone (no header signal)	2.0
A6	Path/subdomain hint + header	2.0

Appendix

Table 5: Vote weights applied by each evidence rule during offline lookup-table construction. Higher weights correspond to stronger multi-signal agreement. Rules A9 and A10 are recorded for diagnostic completeness but do not contribute to weighted aggregation.

Column	Dtype	Meaning / values
key_type	str	domain , host , or host_path
key	str	URL signature (normalised, lowercased)
n_docs	int	raw record count contributing to the signature
top_region	str	winning region (ISO 3166-1 alpha-2) or empty
share	float	vote share of the winning region $\in[0,1]$
n_distinct_regions	int	distinct candidate regions seen

Appendix

Table 6: Schema of the released URL-signature lookup table.

Variant	Kernel	266 region cells all covered	locale classes moved $>$ 2 pt (max)	all 5 pairs above floor	5-pair order preserved	Spearman $\rho$ , language medians mean [95% interval]
swap-soft	uniform	100%	0% (0.52)	100%	44%	0.77 [0.72, 0.81]
swap-soft	softmax	100%	0% (0.41)	100%	59%	0.83 [0.79, 0.87]
hard	uniform	100%	100% (25.79)	100%	15%	0.46 [0.42, 0.52]
hard	softmax	100%	100% (24.97)	100%	15%	0.48 [0.44, 0.54]

Appendix

Table 7: Label-perturbation study (

B=1000

replicates per condition): every merged topic keeps its adjudicated leaf with the observed human-agreement probability 0.224 and is otherwise resampled (uniform or softmax-confusable kernel), entering the pipeline as a soft swap or a hard reassignment. Columns: fraction of replicates in which all 266 region-level cells stay corpus-covered at

k=50

; fraction in which any locale-level four-class percentage moves by more than 2 points (largest observed move, in points); fraction in which all five labelled pairs stay above the pooled

p_{95}

noise floor; fraction preserving the full five-pair JSD ordering; Spearman correlation of per-language median JSDs (182 languages) with the unperturbed baseline.

	Inside	Outside
Languages	94	203
Locales	5,419	535
Merged topics	728,950	72,394
Sampled documents	27,922,687	1,772,582
Adjudicator change rate	7.5%	3.5%
95% CI (cluster bootstrap)	[7.3, 7.6]	[3.1, 3.9]

Appendix

Table 8: Topic-to-CT behaviour inside vs. outside BGE-M3’s documented language coverage (CC-100/XLM-R inventory of its backbone; standard varieties for macrolanguages). Near-uniform:

\max_{c}p(c\mid t)\leq 1.02/14

. The lower adjudicator change rate outside coverage is not evidence of better projections there — the median outside-coverage projection carries no discriminative signal, which also weakens the candidate rankings shown to the adjudicator.

		Split-half JSD ( $\times 10^{-4}$ )
Locale size	Locales	Median	IQR	$p_{95}$
1–2k	1,047	10.6	[6.2, 16.3]	28.2
2–5k	1,106	6.6	[3.7, 10.3]	18.2
5–10k	650	4.3	[2.3, 6.7]	11.6
10k (cap)	1,912	3.2	[1.6, 5.0]	8.7

Appendix

Table 9: Split-half JSD noise floor conditioned on locale sample size (50 splits per locale, seed 42). The floor decreases with locale size; the five labelled pairs of Figure 5 all lie in the 10k bin, so the pooled floor used in the main text if anything understates their separation from noise.

	English-locale mass			Non-English main-language mass (full)		Main-lang. locales
Region	sample	full	growth	docs	share of region	w/ double-blind cells
Caribbean	1.4M	66.5M	$\times$ 48.0	3.6M	5%	3
Oceania	13.8M	719.0M	$\times$ 52.1	309k	$<$ 1%	0
C. America	372k	18.8M	$\times$ 50.5	33.9M	60%	0
Middle Africa	449k	20.5M	$\times$ 45.8	1.6M	5%	0
C. Asia	85k	4.7M	$\times$ 55.6	15.7M	76%	1

Appendix

Table 10: Why the submitted region-level double-blind-spot pattern changed on the full release, for the regions the submitted figure showed as blind. English-locale document mass grows by

\sim

\times

from the sample-350BT analysis corpus to the full release, erasing the region-level gaps, while non-English main-language mass is unchanged (FineWeb-2 was already at full scale). In the Caribbean, Oceania, and Middle Africa the regions’ own main languages contribute at most 5% of full-release mass; in Central America and Central Asia they dominate. Locale-level double blind spots (

k=50

) persist among main-language locales (last column).

Quantity	100BT snap.	350BT snap.	Full release
Total documents	5,176,421,725	5,547,240,607	30,914,158,759
Resolved (non-XX)	2,960,753,780	3,030,341,027	7,917,968,305
Resolution rate	57.20%	54.63%	25.61%
Lookup-table coverage	1,636,856,362	1,681,071,710	4,627,776,301
Lookup-table rate	31.62%	30.30%	14.97%
query (reported separately)	7,206,682	7,314,090	15,283,520

Appendix

Table 11: Reconciliation and provenance of corpus snapshots. The submitted manuscript inadvertently mixed statistics from the first two snapshots; every number in the revised paper comes from the full-release column. Each column combines the stated FineWeb subset with the full FineWeb-2.

Panel of Figure 6 — Figure 6: URL base-domain diversity per topic-modeled locale (up to 10,000 URLs sampled per locale from the full release). Left: CDF of distinct registered base domains per locale (median 116). Right: CDF of the top-1 domain’s share of locale documents (median 0.35); 34% of locales draw more than half of their documents from a single domain.

Artifact	Format	License
Region-annotated corpus	Parquet shards	ODC-By 1.0
URL-signature lookup table	CSV	CC-BY 4.0
Locale Topic Distributions	Parquet	CC-BY 4.0
Per-document culture annotations	Parquet	ODC-By 1.0
Benchmark survey	BibTeX + CSV	CC-BY 4.0
Pipeline code	Git repository	Apache-2.0

Appendix

Table 12: Released artifacts, formats, and licenses. The region-annotated corpus covers the full FineWeb + FineWeb-2 release (30.9B documents) and inherits their ODC-By 1.0 license. Locale Topic Distributions cover 5,954 locales, and the benchmark survey contains 277 entries.

Panel of Figure 7 — Figure 7: Merged topics per locale across the 5,954 topic-modeled locales (801,344 in total). Per-locale merging of the $K=200$ FASTopic budget at cosine $0.85$ retains an effective average of $\approx$ 135 topics per locale.

Element	Datasets
Ideational
Concepts (21)	Hu et al. (2024) ; Cao et al. (2023) ; Nayak et al. (2024) ; Romero et al. (2024) ; Winata et al. (2025) ; Myung et al. (2024a) ; Nguyen et al. (2023b) ; Guo et al. (2025) ; Majewska et al. (2022) ; Li et al. (2024d) ; Li et al. (2024e) ; Yin et al. (2021) ; Koto et al. (2024b) ; Liu et al. (2021) ; Hu et al. (2023) ; Wang et al. (2021) ; Ayash et al. (2025) ; Becattini et al. (2023) ; Thapliyal et al. (2022) ; Li et al. (2023b) ; Rei et al. (2023)
Knowledge (32)	Romero et al. (2024) ; Koto et al. (2024a) ; Myung et al. (2024a) ; Joy (2025) ; Nguyen et al. (2023b) ; Li et al. (2024c) ; Wibowo et al. (2024) ; Chiu et al. (2024b) ; Li et al. (2024d) ; Shi et al. (2024) ; Keleg and Magdy (2023) ; Hardalov et al. (2020) ; Semenov and Sennrich (2025) ; Yin et al. (2022) ; Singh et al. (2025) ; Koto et al. (2024b) ; Koto et al. (2023) ; Son et al. (2024) ; FitzGerald et al. (2022) ; Sakai et al. (2024) ; Lin et al. (2021) ; Kassner et al. (2021) ; Xia and Monti (2021) ; Pramodya et al. (2025) ; Jiang et al. (2020) ; Ponti et al. (2020) ; Yuan et al. (2024) ; Liu et al. (2024) ; Singh et al. (2024) ; Röttger et al. (2024) ; Lewis et al. (2020) ; Clark et al. (2020)
Values - general (15)	Scaria et al. (2024) ; Li et al. (2024b) ; Kirk et al. (2024) ; Aakanksha et al. (2024) ; Pistilli et al. (2024) ; Karinshak et al. (2024) ; Zahraei and Asgari (2025) ; Xu et al. (2024) ; Cahyawijaya et al. (2025) ; Li et al. (2024a) ; Durmus et al. (2024) ; Zhao et al. (2024) ; Sorensen et al. (2024) ; Al Kautsar et al. (2025) ; Hanges and Dickson (2004)
Values - bias (28)	Zulaika and Saralegi (2025) ; Joshi et al. (2025) ; Ravikiran and Annamalai (2021) ; Palta and Rudinger (2023) ; Nozza et al. (2021) ; Neplenbroek et al. (2024) ; Wang et al. (2025) ; Bhutani et al. (2024) ; Mitchell et al. (2025) ; Nadeem et al. (2021) ; Öztürk et al. (2023) ; Troles and Schmid (2021) ; Stanovsky et al. (2019) ; Kappl (2025) ; Xu et al. (2024) ; Lauscher et al. (2020) ; Parrish et al. (2022a) ; Nangia et al. (2020a) ; Strazda and Spanakis (2025) ; Névéol et al. (2022) ; Sahoo et al. (2024) ; Jin et al. (2023) ; Mukherjee et al. (2023a) ; Zhao et al. (2018) ; Lauscher and Glavaš (2019) ; Tomar et al. (2025) ; Rudinger et al. (2018) ; Röttger et al. (2024)
Values - hate (13)	Joshi et al. (2025) ; Ravikiran and Annamalai (2021) ; Mandl et al. (2021) ; Bassignana et al. (2018) ; Trager et al. (2025) ; Luu et al. (2021) ; Lee et al. (2023) ; Vargas et al. (2022) ; Vargas et al. (2026) ; Mathew et al. (2020) ; Bui et al. (2025) ; Dementieva et al. (2025) ; Wulczyn et al. (2017)

Appendix

Table 13: Datasets organized by taxonomy element, following Liu et al. (2025) . Numbers in parentheses indicate dataset count per category. Multi-category datasets appear under each applicable element.

Explore similar work

The Culture Funnel: You Can't Align What isn't in the Data
Jun 11, 2026Ananya Sahu, Mehrnaz Mofakhami, Daniel D'Souza +3Cultural Awareness Large Language Model Alignment
CohereCorrespondingLabs
AlignCultura: Towards Culturally Aligned Large Language Models?
Apr 21, 2026Gautam Siddharth Kashyap, Mark Dras, Usman NaseemCultural Awareness Large Language Model Alignment
School of Computing, Macquarie University, Australia
BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian
Aug 13, 2026Jophin John, Michael Hoffmann, Jan Fillies +2Multilingual Benchmark Dialects
Leibniz Supercomputing Centre (LRZ), Garching, Germany · Stanford University, Stanford, California, USA · Freie Universität Berlin, Germany +2

Jun 11, 2026, cs.CLJ/K move · Enter open · S save

The Culture Funnel: You Can't Align What isn't in the Data

Ananya Sahu, Mehrnaz Mofakhami, Daniel D'Souza, Thomas Euyang+2

CohereCorrespondingLabs

Cultural Awareness Large Language Model Alignment Natural Language Datasets Large Language Model Pipelines Training Data Tagging

Open paper AskPDF arXiv

Current cultural alignment approaches focus on inference-time interventions, assuming models already contain sufficient cultural knowledge. We argue modern LLM pipelines suffer from a cultural data funnel. Using a multidimensional tagging framework across pretraining, fine-tuning, alignment, and reasoning datasets, we show explicit cultural signals decline sharply during post-training, while geographically concentrated, task-specialized data dominates. Multilinguality enhances geographic diversity of cultural knowledge but does not ensure balanced representation. Our tags improve downstream cultural benchmark performance, demonstrating that advances require shifting focus in training data pipelines. To facilitate future research, we release our culturally tagged dataset with 5.6M samples at https://huggingface.co/datasets/CohereLabs/CultureMarkers.

Abstract

Explore similar work

The Culture Funnel: You Can't Align What isn't in the Data

AlignCultura: Towards Culturally Aligned Large Language Models?

BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian