GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training

Jesse van Oort; Frank Brinkkemper; Erik de Graaf; Bram Vanroy; Saskia Lensink

cs.CLApr 1, 2026

GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training

Authors: Jesse van Oort, Frank Brinkkemper, Erik de Graaf, Bram Vanroy, Saskia Lensink

Organizations: TNO, The Netherlands · GPT-NL, The Netherlands · Instituut voor de Nederlandse Taal, The Netherlands · KU Leuven, Belgium

Abstract

We present the GPT-NL Public Corpus, the biggest permissively licensed corpus of Dutch language resources. The GPT-NL Public Corpus contains 21 Dutch-only collections totalling 36B preprocessed Dutch tokens not present in any other LLM pretraining corpus. Additionally, the corpus includes roughly 207B English, 232B Code, and 48B German/Danish tokens taken from existing sets which we further curated for compliance. This corpus includes curated data from large existing corpora like Common Corpus and Common Crawl, as well as newly created Dutch-specific collections. Most newly created Dutch collections consist of content collected in collaboration with organisations or synthetically augmented content. All data is collected and evaluated with the aim of facilitating the creation of (commercial) language models that are lawful, useful and non-harmful. All data included in the GPT-NL Public Corpus is sourced from datasets with permissive licensing and is curated and redistributed under a CC-BY license. The full dataset is publicly available on the Hugging Face Hub.

Figures & tables

Triple	Willem-Alexander – noble title – Prince of Orange
Dutch Sentence	Willem-Alexander heeft de titel Prins van Oranje.

Table 1: Example mapping from a Wikidata triple to a raw sentence.

Language	Documents	Tokens
Afrikaans	312,262	358,873,448
German	9,530,746	11,362,859,534
English	92,635,372	87,537,859,958
French	9,234,900	12,366,480,025
Frisian	230,910	197,430,774
Italian	10,734,597	11,913,669,333

Table 2: Document and token counts C5

Figure 1: Pipeline transforming raw content into curated and evaluated GPT-NL collections. Here visualized with example manual inspection of the scores in the language dimension (language_score | de).

GPT-NL Corpus	Content / Source Description	NL (B)	EN (B)	Oth (B)
Collected Data (GPT-NL Curated)
Openraadsinformatie	Municipal council documentation	14.1	0.02	0.01
Officiële bekendmakingen	Government announcements	2.8	0.01	-
Woogle	Open Dutch government documents	2.6	0.14	-
Koninklijke Bibliotheek	Public domain Dutch texts	2.4	-	-
De Rechtspraak	Judicial cases	2.3	-	-

Table 3: Overview of datasets in the GPT-NL Public Corpus

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
Jun 2, 2025Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett +7Language Model Pretraining Training Data Curation
PleIAs, Paris, France · CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt, Germany · EleutherAI
Language corpora for the Dutch medical domain
Apr 28, 2026B. van EsHealthcare Low-Resource Language Processing
Central Diagnostic Laboratory, University Medical Center Utrecht, Utrecht, The Netherlands. · R&D, B-lab, Castricum, The Netherlands.
KletterMix: Climbing Toward High-Quality German Pretraining Data
Jun 2, 2026Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +4Language Model Pretraining Synthetic Data Pretraining
AI & ML Group, TU Darmstadt · Lab1141 · Fraunhofer IAIS +3

Jun 2, 2025J/K move · Enter open · S save

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas-Hinostroza+6

PleIAs, Paris, France · CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt, Germany · EleutherAI

Language Model Pretraining Training Data Curation

Open paper AskPDF arXiv

Large Language Models (LLMs) are pre-trained on large amounts of data from different sources and domains. Such datasets often contain trillions of tokens, including large portions of copyrighted or proprietary content, which raises questions about the legal use of such models. This underscores the need for truly open pre-training data that complies with data security regulations. In this paper, we introduce Common Corpus, the largest open dataset for LLM pre-training. The data assembled in Common Corpus are either uncopyrighted or under open licenses, totaling about two trillion tokens. The dataset contains a wide variety of languages, ranging from the high-resource European languages to some low-resource languages rarely represented in pre-training datasets. In addition, it includes a large amount of code data. The diversity of data sources in terms of covered domains and time periods opens up the paths for both research and entrepreneurial needs across diverse areas of knowledge. In this paper, we present the detailed provenance of data assembling and the details of dataset filtering and curation. We train two small language models on Common Corpus and find that they perform comparably to other models of their size, indicating that our dataset is suitable for multilingual pretraining. Common Corpus represents a key contribution to the ecosystem for open science research on Large Language Models.

Abstract

Language Model Pretraining

Explore similar work

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

Language corpora for the Dutch medical domain

KletterMix: Climbing Toward High-Quality German Pretraining Data