Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Pierre-Carl Langlais; Pavel Chizhov; Catherine Arnett; Carlos Rosas-Hinostroza; Mattia Nee; Eliot Krzystof Jones; Irène Girard; David Mach

cs.CLJun 2, 2025

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Authors: Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas-Hinostroza, Mattia Nee, Eliot Krzystof Jones, Irène Girard, David Mach, +2 more

Organizations: PleIAs, Paris, France · CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt, Germany · EleutherAI

Abstract

Large Language Models (LLMs) are pre-trained on large amounts of data from different sources and domains. Such datasets often contain trillions of tokens, including large portions of copyrighted or proprietary content, which raises questions about the legal use of such models. This underscores the need for truly open pre-training data that complies with data security regulations. In this paper, we introduce Common Corpus, the largest open dataset for LLM pre-training. The data assembled in Common Corpus are either uncopyrighted or under open licenses, totaling about two trillion tokens. The dataset contains a wide variety of languages, ranging from the high-resource European languages to some low-resource languages rarely represented in pre-training datasets. In addition, it includes a large amount of code data. The diversity of data sources in terms of covered domains and time periods opens up the paths for both research and entrepreneurial needs across diverse areas of knowledge. In this paper, we present the detailed provenance of data assembling and the details of dataset filtering and curation. We train two small language models on Common Corpus and find that they perform comparably to other models of their size, indicating that our dataset is suitable for multilingual pretraining. Common Corpus represents a key contribution to the ecosystem for open science research on Large Language Models.

Figures & tables

Figure 1: Proportional treemap of Common Corpus collections and their most popular languages.

Figure 2: A schematic world map of languages in Common Corpus with a log-scaled distribution of document counts. For each language, we chose a city that is located in the region where this language is most specific to. To avoid outliers, we show only languages with 10,000+ documents.

Panel of Figure 3 — Figure 3: Temporal and semantic overview of the Common Corpus collections.

	KL3M	Dolma	C4	ROOTS	DCAD	FineWeb 2	Common	Common
	KL3M	Dolma	C4	ROOTS	2000	FineWeb 2	Pile	Corpus
Multidomain	✗	✓	✓	✓	✓	✓	✓	✓
Beyond Web Crawl	✓	✓	✗	✗	✗	✗	✓	✓
Multilingual	✗	✗	✓	✓	✓	✓	✗	✓
Open data	✓	✗	✗	✗	✗	✗	✓	✓

Table 1: Comparison of the contemporary datasets for LLM training.

Figure 4: Stacked histograms of probability density for qualitative evaluations of Common Corpus on a sample of 300,000 documents. Metric descriptions can be found in Appendix G .

Model	Ours	Gemma 3	XGLM	BLOOM	Ours	Gemma 3	XGLM	OLMo
Model	350M	270M	564M	560M	1.2B	1B	1.7B	1B
MultiBLiMP	0.774	0.762	0.711	0.683	0.797	0.799	0.710	0.699
XStoryCloze	0.509	0.533	0.537	0.532	0.526	0.594	0.569	0.517
XCOPA	0.533	0.544	0.550	0.541	0.541	0.593	0.574	0.518

Table 2: Benchmarking results. “Ours” refers to PleIAs models pre-trained on Common Corpus.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Documents	Words	Tokens
Open Government	0 75,652,998	0, 257,561,830,682	0, 407,067,554,189
Open Culture	0 93,156,602	0, 549,608,763,966	0, 885,982,490,090
Open Science	0 19,220,942	0, 147,305,783,453	0, 281,193,563,789
Open Code	202,765,051	0,0 77,669,169,092	0, 283,227,402,898
Open Web	0 96,165,348	0,0 33,208,509,065	0,0 73,217,485,489
Open Semantic	0 30,072,707	0,0 23,284,201,782	0,0 67,958,671,827

Appendix

Table 3: Dataset composition of Common Corpus. For each collection, we report the total number of documents, words (whitespace-separated), and tokens.

License type	Tokens
Public Domain	1,138,508,375,958
CC-By	0, 287,749,264,457
MIT	0, 142,694,227,607
CC-By-SA	0,0 74,768,060,836
Apache-2.0	0,0 68,750,977,037
BSD-3-Clause	0,0 18,483,944,333

Appendix

Table 4: Token counts for the ten most common licenses in Common Corpus.

Language	Documents	Words	Tokens
English	154,175,907	634,794,970,595	968,757,721,747
French	0 35,245,624	162,061,620,874	275,358,437,630
German	0 11,385,377	0 56,674,819,173	112,127,458,251
Spanish	0 0 6,530,094	0 26,215,767,271	0 46,514,142,421
Latin	0 0 2,367,110	0 16,189,444,325	0 36,031,591,540
Italian	0 0 3,804,052	0 13,207,129,356	0 24,681,637,575

Appendix

Table 5: Top-50 languages in Common Corpus by token count. Each language is presented with its number of documents, words, and tokens in the corpus. The rows are ordered by the token count.

Figure 5: Comparing the fertility of PleIAs tokenizer (marked as “Ours 1B”) and other language models for six languages. The data source for all languages is the devtest set of Flores200 ( Costa-jussà et al., 2022 ) .

Dataset	Main Languages	Documents	Tokens
SEC	English	1,085,113	0 9,653,919,837
WTO	English, Spanish, French, and small partitions of others	0, 772,508	0 2,835,007,015
AMF	French, English	0, 595,397	0 9,823,755,281
TED EU Tenders	German, French, Polish, Spanish, Dutch, Czech, Romanian, English, Swedish, Italian, Bulgarian, Finnish, Latvian, Danish, Lithuanian, Croatian, Estonian, Hungarian, Portuguese, Slovenian, Slovak, Greek, Irish	0, 137,837	00, 650,396,761
GATT Library	English, French, Spanish, Catalan, Portuguese, German	0,0 67,596	00, 224,526,628

Appendix

Table 6: Finance Commons sources distribution with languages.

Dataset	Languages	Tokens
Caselaw Access Project	English	0 13,821,842,995
Court Listener	English	0 22,625,121,735
EUR-lex	Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish	0 65,044,763,781
Eurovoc	English, German, French, Croatian, Italian, Lithuanian, Portuguese, Finnish, Danish, Bulgarian, Dutch, Polish, Greek, Swedish, Hungarian, Czech, Spanish, Maltese, Latvian, Slovak, Slovenian, Romanian, Estonian, Arabic, Tigrinya, Farsi, Russian, Urdu, Serbian, Albanian, Kurdish, Pushto, Irish, Norwegian, Icelandic, Dari, Armenian, Japanese.	0 31,648,136,898
French open data	French	0 24,597,392,089
USPTO	English	200,509,900,178

Appendix

Table 7: Legal Commons sources distribution with languages.

Corpus	Language	Domain	Tokens
English PD	English	Books and Newspapers	174.2B
US PD Books	English	Books	0 82.2B
French PD Books	French	Books	0 24.0B
French PD Newspapers	French	Newspapers	110.8B
French PD Diverse	French	Books and Newspapers	0 69.6B
LoC Books	English	Books	0 10.6B

Appendix

Table 8: Subsets of Open Culture with language coverage, type of document, and token count.

Dataset	Tokens
OpenAlex	191,616,437,384
Open Science Pile	0 11,096,766,324
Open Science French	0 46,961,690,792
Open Science Spanish	0 16,523,491,767
Open Science German	00 7,806,446,050
ArXiv	00 7,188,731,472

Appendix

Table 9: Token count by dataset Open Science.

Language	Tokens
Java	35,697,451,454
JavaScript	28,894,772,110
Python	26,681,331,771
C++	25,481,950,314
C	23,277,000,113
PHP	23,077,121,733

Appendix

Table 10: Token counts by programming language or framework.

Model	Ours	Gemma 3	XGLM	BLOOM	Ours	Gemma 3	XGLM	OLMo
Model	350M	270M	564M	560M	1.2B	1B	1.7B	1B
abk	0.550	0.750	0.475	0.525	0.675	0.675	0.325	0.800
aln	0.733	0.755	0.709	0.700	0.728	0.750	0.675	0.690
amh	0.946	0.929	0.911	0.973	1.000	0.955	0.848	0.964
apu	0.964	0.964	0.893	0.786	0.964	0.929	0.964	0.893
aqz	0.214	0.357	0.429	0.500	0.429	0.429	0.714	0.214

Appendix

Table 11: Multilingual benchmarking results on MultiBLIMP (ISO 639 language codes a*–i* in alphabetical order). “Ours” refers to PleIAs models pre-trained on Common Corpus. Within each model group, the best score is in bold , and the second-best is underlined .

Model	Ours	Gemma 3	XGLM	BLOOM	Ours	Gemma 3	XGLM	OLMo
Model	350M	270M	564M	560M	1.2B	1B	1.7B	1B
kat	0.931	0.951	0.917	0.760	0.951	0.951	0.809	0.907
kaz	0.705	0.792	0.647	0.682	0.780	0.844	0.682	0.688
kir	0.930	0.843	0.914	0.919	0.935	0.924	0.843	0.941
kmr	0.710	0.662	0.588	0.579	0.761	0.748	0.577	0.588
koi	0.628	0.488	0.605	0.558	0.558	0.651	0.628	0.605

Appendix

Table 12: Multilingual benchmarking results on MultiBLIMP (ISO 639 language codes k*–y* in alphabetical order). “Ours” refers to PleIAs models pre-trained on Common Corpus. Within each model group, the best score is in bold , and the second-best is underlined .

Model	Ours	Gemma 3	XGLM	BLOOM	Ours	Gemma 3	XGLM	OLMo
Model	350M	270M	564M	560M	1.2B	1B	1.7B	1B
ar	0.475	0.492	0.500	0.521	0.477	0.572	0.525	0.473
ca	0.514	0.513	0.567	0.561	0.535	0.600	0.602	0.509
en	0.569	0.614	0.606	0.612	0.617	0.698	0.645	0.704
es	0.520	0.558	0.549	0.555	0.543	0.628	0.593	0.556
eu	0.516	0.524	0.531	0.538	0.514	0.531	0.561	0.503

Appendix

Table 13: Multilingual benchmarking results on XStoryCloze. “Ours” refers to PleIAs models pre-trained on Common Corpus. Languages are represented as two-letter codes in ISO 639.

Model	Ours	Gemma 3	XGLM	BLOOM	Ours	Gemma 3	XGLM	OLMo
Model	350M	270M	564M	560M	1.2B	1B	1.7B	1B
es	0.566	0.590	0.604	0.620	0.614	0.678	0.664	0.524
et	0.518	0.498	0.554	0.488	0.500	0.536	0.568	0.480
eu	0.504	0.514	0.512	0.502	0.518	0.502	0.534	0.516
ht	0.522	0.504	0.548	0.500	0.524	0.518	0.556	0.534
id	0.534	0.578	0.574	0.596	0.558	0.690	0.646	0.544

Appendix

Table 14: Multilingual benchmarking results on XCopa. “Ours” refers to PleIAs models pre-trained on Common Corpus. Languages are represented as two-letter codes in ISO 639.

Explore similar work

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Synthetic Training Data Synthetic Data
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6
MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models
Jun 6, 2026Kaixin Lan, Mu You, Tao Fang +3Pretraining Machine-Generated Text Detection
Natural Language Processing & Portuguese-Chinese Machine Translation (NLP2CT) Laboratory, University of Macau, Macau SAR, China · Institute of International Language Services Studies, Macau Millennium College, Macau SAR, China · Innovation Research & Development, BoardWare Information System Limited
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Multilingual Language Models Large Language Model Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14

Sep 29, 2026, cs.CLJ/K move · Enter open · S save

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov+6

PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab+6

Synthetic Training Data Synthetic Data Natural Language Datasets Training Data Text Corpora Domain-Specific Language

Open paper AskPDF arXiv

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with reasoning traces to address cold-start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our SYNTH dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source language model development.

Abstract

Explore similar work

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models

MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages