ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset

DatologyAI; :; Aldo Gael Carranza; Kaleigh Mentzer; Ricardo Pio Monti; Alex Fang; Alvin Deng; Amro Abbas

cs.LGFeb 16, 2026

ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset

Authors: DatologyAI, :, Aldo Gael Carranza, Kaleigh Mentzer, Ricardo Pio Monti, Alex Fang, Alvin Deng, Amro Abbas, +27 more

Organizations: DatologyAI

Abstract

Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.

Figures & tables

Figure 1: A new compute-performance Pareto frontier for English and multilingual capabilities . We report error rate (log-scale; 1-accuracy, lower is better) as a function of training FLOPs (log-scale) across English (MMLU+ARC) and three multilingual benchmarks (Multilingual MMLU, Multilingual ARC, and Belebele). All evaluations use a multiple-choice format; multilingual scores are averaged over 13 languages. The shaded gray region summarizes the performance–compute envelope of representative open-weight baselines (e.g., Qwen3-4B/8B, Granite-4.0-3B). DatologyAI models occupy the bottom-left region relative to these baselines, indicating substantially lower multilingual error at reduced compute. We restrict our English-language evaluations to MMLU and ARC-Challenge for parity with the multilingual evaluations, and reserve comprehensive English and quantitative benchmarking for forthcoming companion releases.

Language	Family	Script	FineWeb2 Documents (M)	Llama 3.2 1B Tokens (B)
Russian	Slavic	Cyrillic	699.1	1004.6
Chinese	Sino-Tibetan	Hanzi	636.1	743.4
German	Germanic	Latin	496.0	407.0
Spanish	Romance	Latin	441.3	352.3
Japanese	Japonic	Kanji + Kana	400.1	404.4
French	Romance	Latin	360.1	306.4

Table 1: Non-English languages included in this study.

Figure 2: Impact of Curation Strategy on Multilingual Performance (bilingual models). Performance comparison for 3B parameter models trained on 60BT tokens (50:50 English:non-English ratio). Results are averaged across multilingual MMLU, ARC, and Belebele. Across 13 languages, results show that improved English curation (light blue bars) consistently improves performance over the uncurated baseline (dark purple bars; improvement in 12 of 13 languages), while combining curated English with curated multilingual data (dark blue bars) yields the highest average scores across all languages.

Figure 3: Correlation between language similarity to English and cross-lingual transfer benefit. We evaluate linguistic distance using two proxies: (a) average log embedding distance across LaBSE, e5-small, and sentence-transformers, and (b) log perplexity of the target language under an English-only model. Both metrics show a significant negative correlation (Pearson

r=-0.62

and

r=-0.70

respectively) with the performance uplift gained from English data curation. These results demonstrate that linguistically similar languages, such as Spanish and French, receive the most pronounced benefits from high-quality English data, while more distant languages like Bengali and Arabic show significantly lower transfer gains.

Figure 4: Non-English Curation Benefits English Performance. Performance comparison for 3B parameter models trained on 60BT tokens (50:50 English:non-English ratio). Results are average of English MMLU and ARC. We contrast performance when when the accompanying multilingual data is uncurated (dark purple) versus curated (dark blue). We observe positive transfer in 12 out of 13 languages, with an overall relative improvement of 1.21%.

Figure 5: Evaluation of benefits associated with Random vs Scored Translation for Low-Resource Languages. Performance curves for Hindi, Bengali and Arabic showing that while augmenting training data with translated English text (red and cyan lines) improves over the uncurated baseline (dark gray), it still falls short of the performance achieved by bespoke DatologyAI curation (dark blue).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Language	MMLU	ARC	Belebele
Spanish	Global MMLU	Okapi	Belebele
Portuguese	Global MMLU	LumiOpen	Belebele
French	Global MMLU	Okapi	Belebele
German	Global MMLU	Okapi	Belebele
Italian	Global MMLU	Okapi	Belebele
Vietnamese	Global MMLU	Okapi	Belebele

Appendix

Table 2: Table describing the choice of evaluation datasets.

Model	Total parameters ( $N$ )	Total tokens ( $D$ )	FLOPs ( $\approx 6ND$ )
DatologyAI 3B	3B	1T	$1.8\times 10^{22}$
DatologyAI 8B	8B	1T	$4.8\times 10^{22}$
Llama-3.2-1B	1B	9T	$5.4\times 10^{22}$
Llama-3.2-3B	3B	9T	$1.6\times 10^{23}$
Llama-3.2-8B	8B	15T	$7.2\times 10^{23}$
SmolLM3-3B	3B	11T	$1.9\times 10^{23}$

Appendix

Table 3: Estimated training compute (FLOPs) for the open-source baselines in Figure 1 , computed as

\approx 6ND

. For MoE models,

N

denotes the number of active parameters per token.

Figure 6: Per-language performance vs. training compute for latin-script languages. Rows correspond to Spanish, Portuguese, French, German, Vietnamese, and Indonesian. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x).

Figure 7: Per-language performance vs. training compute for Indic and Arabic-script languages. Rows correspond to Hindi, Bengali, and Arabic. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x).

Figure 8: Per-language performance vs. training compute for CJK languages and Russian. Rows correspond to Chinese, Japanese, Korean, and Russian. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x). We note that there is no ARC Challenge evaluation available for Japanese.

Figure 9: Per-Language Performance vs. Multilingual Training Tokens. We visualize the number of language-specific training tokens (x-axis, billions) and the average downstream performance across a range of models. We only include models where the number of tokens per langauge could be reasonably estimated based on public information. DatologyAI models were trained with only 6B tokens per language (7.75% mutlilingual overall). The plots demonstrate significant data efficiency improvements from DatologyAI curation compared to open-source baselines such as Llama-3.2, SmolLM2, and language-specific models like Sarvam-2B. We add an asterix beside the name of all non-DatologyAI models, to highlight that we estimated the number of tokens per language to the best of our ability based on publicly available information.

Model	MMLU	ARC	Belebele
Datology 3B	0.46	0.63	0.62
Datology 8B	0.55	0.73	0.76
Llama-3.2 1B	0.30	0.31	0.38
Llama-3.2 3B	0.47	0.56	0.56
Llama-3.1 8B	0.55	0.67	0.74
Qwen3 4B	0.67	0.86	0.85

Appendix

Table 4: Evaluations for language Spanish

Model	MMLU	ARC	Belebele
Datology 3B	0.44	0.58	0.64
Datology 8B	0.55	0.71	0.77
Llama-3.2 1B	0.31	0.30	0.35
Llama-3.2 3B	0.46	0.53	0.55
Llama-3.1 8B	0.54	0.66	0.79
Qwen3 4B	0.65	0.84	0.84

Appendix

Table 5: Evaluations for language Portuguese

Model	MMLU	ARC	Belebele
Datology 3B	0.43	0.58	0.63
Datology 8B	0.55	0.74	0.77
Llama-3.2 1B	0.29	0.31	0.32
Llama-3.2 3B	0.46	0.54	0.58
Llama-3.1 8B	0.54	0.68	0.78
Qwen3 4B	0.66	0.84	0.87

Appendix

Table 6: Evaluations for language French

Model	MMLU	ARC	Belebele
Datology 3B	0.44	0.59	0.62
Datology 8B	0.54	0.71	0.76
Llama-3.2 1B	0.26	0.26	0.28
Llama-3.2 3B	0.44	0.50	0.56
Llama-3.1 8B	0.52	0.63	0.69
Qwen3 4B	0.65	0.85	0.86

Appendix

Table 7: Evaluations for language German

Model	MMLU	ARC	Belebele
Datology 3B	0.43	0.59	0.56
Datology 8B	0.52	0.67	0.73
Llama-3.2 1B	0.27	0.26	0.34
Llama-3.2 3B	0.38	0.46	0.48
Llama-3.1 8B	0.48	0.57	0.67
Qwen3 4B	0.59	0.79	0.81

Appendix

Table 8: Evaluations for language Vietnamese

Model	MMLU	ARC	Belebele
Datology 3B	0.42	0.59	0.56
Datology 8B	0.52	0.68	0.72
Llama-3.2 1B	0.31	0.29	0.36
Llama-3.2 3B	0.42	0.50	0.51
Llama-3.1 8B	0.49	0.60	0.69
Qwen3 4B	0.64	0.83	0.83

Appendix

Table 9: Evaluations for language Indonesian

Model	MMLU	ARC	Belebele
Datology 3B	0.32	0.41	0.45
Datology 8B	0.39	0.54	0.54
Llama-3.2 1B	0.26	0.27	0.29
Llama-3.2 3B	0.34	0.42	0.41
Llama-3.1 8B	0.38	0.49	0.53
Qwen3 4B	0.48	0.71	0.71

Appendix

Table 10: Evaluations for language Hindi

Model	MMLU	ARC	Belebele
Datology 3B	0.32	0.42	0.43
Datology 8B	0.36	0.48	0.51
Llama-3.2 1B	0.25	0.24	0.25
Llama-3.2 3B	0.31	0.33	0.39
Llama-3.1 8B	0.36	0.42	0.53
Qwen3 4B	0.45	0.67	0.71

Appendix

Table 11: Evaluations for language Bengali

Model	MMLU	ARC	Belebele
Datology 3B	0.42	0.59	0.58
Datology 8B	0.49	0.67	0.72
Llama-3.2 1B	0.27	0.27	0.29
Llama-3.2 3B	0.38	0.46	0.50
Llama-3.1 8B	0.43	0.55	0.65
Qwen3 4B	0.56	0.78	0.82

Appendix

Table 12: Evaluations for language Arabic

Model	MMLU	ARC	Belebele
Datology 3B	0.45	0.63	0.64
Datology 8B	0.52	0.72	0.74
Llama-3.2 1B	0.33	0.33	0.35
Llama-3.2 3B	0.46	0.58	0.66
Llama-3.1 8B	0.53	0.67	0.78
Qwen3 4B	0.66	0.84	0.87

Appendix

Table 13: Evaluations for language Chinese

Model	MMLU	Belebele
Datology 3B	0.44	0.55
Datology 8B	0.52	0.66
Llama-3.2 1B	0.26	0.25
Llama-3.2 3B	0.40	0.44
Llama-3.1 8B	0.48	0.65
Qwen3 4B	0.59	0.79

Appendix

Table 14: Evaluations for language Japanese

Model	MMLU	ARC	Belebele
Datology 3B	0.43	0.62	0.61
Datology 8B	0.49	0.69	0.73
Llama-3.2 1B	0.26	0.25	0.26
Llama-3.2 3B	0.39	0.47	0.49
Llama-3.1 8B	0.48	0.59	0.68
Qwen3 4B	0.59	0.81	0.82

Appendix

Table 15: Evaluations for language Korean

Model	MMLU	ARC	Belebele
Datology 3B	0.41	0.58	0.61
Datology 8B	0.52	0.70	0.75
Llama-3.2 1B	0.31	0.31	0.34
Llama-3.2 3B	0.43	0.55	0.56
Llama-3.1 8B	0.53	0.67	0.77
Qwen3 4B	0.64	0.81	0.85

Appendix

Table 16: Evaluations for language Russian

Explore similar work

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
Apr 14, 2026Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin +2Cross-Lingual Reasoning in Language Models Low-Resource Language Processing
UC Santa Barbara · Amazon
On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
Apr 22, 2026Yassine Turki, Vinko Sabolčec, Bettina Messmer +1Language Model Pretraining Multilingual Language Models
Machine Learning Optimization Lab Ecole Polytechnique Fédérale de Lausanne (EPFL) Lausanne, Switzerland

Apr 14, 2026J/K move · Enter open · S save

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin, Dezhi Hong+1

UC Santa Barbara · Amazon

Cross-Lingual Reasoning in Language Models Low-Resource Language Processing Language Model Post-Training Cross-Lingual Transfer

Open paper AskPDF arXiv

Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that English-only post-training is typically suboptimal: incorporating even a single non-English language improves both English performance and cross-lingual generalization. Increasing language diversity during post-training generally yields further gains, particularly for low-resource languages, while performance on high-resource languages tends to plateau rather than degrade. Moreover, greater language diversity enables strong zero-shot transfer to unseen languages, reducing the need for direct inclusion, though gains remain limited for typologically distant, low-resource languages.

Abstract

Explore similar work

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

On the Limits of Model Merging for Multilinguality in Pre-Training

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection