Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.
Figures & tables
Figure 1: A new compute-performance Pareto frontier for English and multilingual capabilities . We report error rate (log-scale; 1-accuracy, lower is better) as a function of training FLOPs (log-scale) across English (MMLU+ARC) and three multilingual benchmarks (Multilingual MMLU, Multilingual ARC, and Belebele). All evaluations use a multiple-choice format; multilingual scores are averaged over 13 languages. The shaded gray region summarizes the performance–compute envelope of representative open-weight baselines (e.g., Qwen3-4B/8B, Granite-4.0-3B). DatologyAI models occupy the bottom-left region relative to these baselines, indicating substantially lower multilingual error at reduced compute. We restrict our English-language evaluations to MMLU and ARC-Challenge for parity with the multilingual evaluations, and reserve comprehensive English and quantitative benchmarking for forthcoming companion releases.
Language
Family
Script
FineWeb2 Documents (M)
Llama 3.2 1B Tokens (B)
Russian
Slavic
Cyrillic
699.1
1004.6
Chinese
Sino-Tibetan
Hanzi
636.1
743.4
German
Germanic
Latin
496.0
407.0
Spanish
Romance
Latin
441.3
352.3
Japanese
Japonic
Kanji + Kana
400.1
404.4
French
Romance
Latin
360.1
306.4
Table 1: Non-English languages included in this study.
Figure 2: Impact of Curation Strategy on Multilingual Performance (bilingual models). Performance comparison for 3B parameter models trained on 60BT tokens (50:50 English:non-English ratio). Results are averaged across multilingual MMLU, ARC, and Belebele. Across 13 languages, results show that improved English curation (light blue bars) consistently improves performance over the uncurated baseline (dark purple bars; improvement in 12 of 13 languages), while combining curated English with curated multilingual data (dark blue bars) yields the highest average scores across all languages.
Figure 3: Correlation between language similarity to English and cross-lingual transfer benefit. We evaluate linguistic distance using two proxies: (a) average log embedding distance across LaBSE, e5-small, and sentence-transformers, and (b) log perplexity of the target language under an English-only model. Both metrics show a significant negative correlation (Pearson r=−0.62 and r=−0.70 respectively) with the performance uplift gained from English data curation. These results demonstrate that linguistically similar languages, such as Spanish and French, receive the most pronounced benefits from high-quality English data, while more distant languages like Bengali and Arabic show significantly lower transfer gains.
Figure 4: Non-English Curation Benefits English Performance. Performance comparison for 3B parameter models trained on 60BT tokens (50:50 English:non-English ratio). Results are average of English MMLU and ARC. We contrast performance when when the accompanying multilingual data is uncurated (dark purple) versus curated (dark blue). We observe positive transfer in 12 out of 13 languages, with an overall relative improvement of 1.21%.
Figure 5: Evaluation of benefits associated with Random vs Scored Translation for Low-Resource Languages. Performance curves for Hindi, Bengali and Arabic showing that while augmenting training data with translated English text (red and cyan lines) improves over the uncurated baseline (dark gray), it still falls short of the performance achieved by bespoke DatologyAI curation (dark blue).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Language
MMLU
ARC
Belebele
Spanish
Global MMLU
Okapi
Belebele
Portuguese
Global MMLU
LumiOpen
Belebele
French
Global MMLU
Okapi
Belebele
German
Global MMLU
Okapi
Belebele
Italian
Global MMLU
Okapi
Belebele
Vietnamese
Global MMLU
Okapi
Belebele
Appendix
Table 2: Table describing the choice of evaluation datasets.
Model
Total parameters ( N )
Total tokens ( D )
FLOPs ( ≈6ND )
DatologyAI 3B
3B
1T
1.8×1022
DatologyAI 8B
8B
1T
4.8×1022
Llama-3.2-1B
1B
9T
5.4×1022
Llama-3.2-3B
3B
9T
1.6×1023
Llama-3.2-8B
8B
15T
7.2×1023
SmolLM3-3B
3B
11T
1.9×1023
Appendix
Table 3: Estimated training compute (FLOPs) for the open-source baselines in Figure 1 , computed as ≈6ND . For MoE models, N denotes the number of active parameters per token.
Figure 6: Per-language performance vs. training compute for latin-script languages. Rows correspond to Spanish, Portuguese, French, German, Vietnamese, and Indonesian. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x).
Figure 7: Per-language performance vs. training compute for Indic and Arabic-script languages. Rows correspond to Hindi, Bengali, and Arabic. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x).
Figure 8: Per-language performance vs. training compute for CJK languages and Russian. Rows correspond to Chinese, Japanese, Korean, and Russian. Columns 1–3 report performance on MMLU, ARC, and Belebele as a function of training FLOPs (x-axis). The rightmost column compares each model’s language-specific average score (y-axis) to its all-language average across multilingual evaluations (x-axis); the dashed line indicates parity (y = x). We note that there is no ARC Challenge evaluation available for Japanese.
Figure 9: Per-Language Performance vs. Multilingual Training Tokens. We visualize the number of language-specific training tokens (x-axis, billions) and the average downstream performance across a range of models. We only include models where the number of tokens per langauge could be reasonably estimated based on public information. DatologyAI models were trained with only 6B tokens per language (7.75% mutlilingual overall). The plots demonstrate significant data efficiency improvements from DatologyAI curation compared to open-source baselines such as Llama-3.2, SmolLM2, and language-specific models like Sarvam-2B. We add an asterix beside the name of all non-DatologyAI models, to highlight that we estimated the number of tokens per language to the best of our ability based on publicly available information.
Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that English-only post-training is typically suboptimal: incorporating even a single non-English language improves both English performance and cross-lingual generalization. Increasing language diversity during post-training generally yields further gains, particularly for low-resource languages, while performance on high-resource languages tends to plateau rather than degrade. Moreover, greater language diversity enables strong zero-shot transfer to unseen languages, reducing the need for direct inclusion, though gains remain limited for typologically distant, low-resource languages.
Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training.
As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in embedding space may show cross-lingual consistency, which would allow high-resource languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B tokens, delivering gains for high resource languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the multilingual signal.