A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT

Louis Estève; Christophe Servan; Thomas Lavergne; Agata Savary

cs.CLFeb 25, 2026

A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT

Authors: Louis Estève, Christophe Servan, Thomas Lavergne, Agata Savary

Organizations: Université Paris-Saclay, CNRS, LISN

Abstract

Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons for an investigation of the impact of diversity on the ModernBERT pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, so as to pick the best one. We find that diversity-driven sampling allows in some tasks to gain 10 points relative to randomly-sampled pre-training data of commensurate size. We also see that a model pre-trained for 483h on a diversity-driven dataset of 150M tokens can yield a commensurate performance to a model pre-trained for 1,775h on a randomly-driven dataset of 2.4B tokens.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of sampling algorithms run on UD v2.16 (French). Lee et al. (2009) have the best entropy, but Scholivet et al. (2025) is just below, with only a third in $n^{\prime}$ . For Scholivet et al. (2025) , the last value of $E$ is $20$ , with each preceding value increasing by $10$ (the only exception is at $\left|E\right|=1$ where the only value is $1$ ).

Dataset	Words	$H$ ’ ( $\uparrow$ )	$H$ ( $\uparrow$ )
Topline -2400M	2 379M	$7.54$	$7.55$
Random -400M	401M	$7.57$	$7.59$
Diverse -400M	397M	$8.80$	$8.53$
Random -240M	242M	$7.59$	$7.60$
Diverse -230M	230M	$8.99$	$8.45$
Random -150M	150M	$7.61$	$7.61$

Table 1: Sampled datasets: part of EXTENSION (with entropy

H^{\prime}

) is appended to the BASE (the whole has entropy

H

). Diverse -400M has E only at 1, Diverse -230M from 50 to 20 (4 values), Diverse -150M from 170 to 20 (16 values).

Panel of Figure 2 — Figure 2: Pre-training process for encoders. Loss is smoothed using a moving average (window size: $2\times 10^{5}$ , series sizes from $1.5$ M to $6.3$ M points) applied thrice.

Task (reference)	Domain	Classes	Size (k)
PAWS-X ( Yang et al., 2019 )	Paraphrase classification	$2$	$49$ / $2.0$ / $2.0$
XNLI ( Conneau et al., 2018 )	NLI	$3$	$393$ / $5.4$ / $5.0$
MASSIVE Intent ( FitzGerald et al., 2023 )	Intent detection	$60$	$11.5$ / $2.0$ / $2.9$
WikiNER ( Nothman et al., 2013 )	NER	$7$	$129$ / $0.5$ / $14.3$
MultiATIS++ ( Upadhyay et al., 2018 )	POS-tagging	$131$	$37.0$ / $0.0$ / $7.8$
Media (full) ( Bonneau-Maynard et al., 2006 )	NLU	$152$	$13.7$ / $1.3$ / $3.7$

Table 2: Tasks used for fine-tuning. Size refers to the number of entries (by default, sentences), for TRAIN / DEV / TEST. For NER, the number of classes is after BIO expansion.

Encoder	PTT	Classification			Sequence labelling
Encoder	PTT	PX	XNLI	AMI	WNER	MATIS	MDF
Topline -2400M	1775h	84.7 $\pm$ 1.1	75.9 $\pm$ 0.6	81.7 $\pm$ 2.5	89.8 $\pm$ 0.5	92.5 $\pm$ 0.7	82.6 $\pm$ 0.5
Random -400M	483h	85.9 $\pm$ 1.9	75.1 $\pm$ 1.7	83.6 $\pm$ 0.7	90.0 $\pm$ 0.4	92.1 $\pm$ 1.2	83.2 $\pm$ 0.4
Diverse -400M	483h	85.6 $\pm$ 1.5	74.7 $\pm$ 0.6	82.6 $\pm$ 1.0	90.6 $\pm$ 0.9	91.8 $\pm$ 1.8	82.0 $\pm$ 1.1
Random -240M	483h	82.6 $\pm$ 5.2	76.4 $\pm$ 1.0	82.5 $\pm$ 0.8	90.1 $\pm$ 0.7	92.8 $\pm$ 0.8	82.7 $\pm$ 0.3
Diverse -230M	483h	86.3 $\pm$ 1.6	74.9 $\pm$ 1.8	82.2 $\pm$ 1.5	90.8 $\pm$ 0.4	92.8 $\pm$ 0.8	82.5 $\pm$ 0.2

Table 3: Encoder & head fine-tuning. Evaluation on TEST. PTT is pre-training time. PX is PAWS-X, AMI is Amazon Massive Intent, WNER is WikiNER, MATIS is MultiATIS++, MDF is MEDIA (full). Mean

\pm

standard deviation, across five seeds. Bold means

\Delta\geq 1

Encoder	PTT	MATIS	MDF
Topline -2400M	1775h	83.2 $\pm$ 0.5	58.9 $\pm$ 0.5
Random -400M	483h	84.0 $\pm$ 0.5	60.3 $\pm$ 0.3
Diverse -400M	483h	84.5 $\pm$ 0.3	59.1 $\pm$ 0.8
Random -240M	483h	84.4 $\pm$ 0.5	59.7 $\pm$ 0.6
Diverse -230M	483h	86.4 $\pm$ 0.4	61.7 $\pm$ 0.2
Random -150M	483h	80.2 $\pm$ 0.2	51.9 $\pm$ 0.4

Table 4: Head-only fine-tuning. Evaluation on TEST. PTT is pre-training time. MDF is MEDIA (full), MATIS is MultiATIS++. Mean

\pm

standard deviation, across five seeds. Bold means

\Delta\geq 1

, underline means

\Delta\geq 5

Panel of Figure 3 — Figure 3: Fine-tuning (head only) for MEDIA (full). Dashed thick lines delimit dataset traversals. Thick crosses denote maximum value.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 422 papers

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling
Sep 26, 2026Zailin Ma, Quzhe Huang, Yujun Li +2Language Model Pretraining
Peking University · Wizard Intelligence Learning Lab · Tsinghua University
Growing a Tail: Increasing Output Diversity in Large Language Models
Nov 5, 2024Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1LLM Evaluation LLM Prompting
Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.
Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs
Sep 15, 2026Kirill Skobelev, Eric Fithian, X. Y. HanOutput Diversity in Language Models Mode Collapse
Northwestern University · University of Chicago

Sep 26, 2026J/K move · Enter open · S save

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

Zailin Ma, Quzhe Huang, Yujun Li, Congyuan Rao+1

Peking University · Wizard Intelligence Learning Lab · Tsinghua University

Language Model Pretraining

Open paper AskPDF arXiv

Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversity collapse: by favoring texts similar to high-quality reference corpora (e.g., educational or QA-style data), it systematically excludes valuable data from underrepresented domains. In contrast, diversified selection preserves domain balance and encourages robust downstream performance, yet existing methods either focus on coverage-oriented objectives that indirectly enhance diversity, or directly optimize for diversity via costly covariance matrix recomputation that limits scalability. To address these issues, we introduce Leverage Score Sampling (Lev), which iteratively selects samples that maximally expand the determinantal volume of the embedded data via leverage scores, a computationally efficient criterion that eliminates matrix recomputation and enables scalable selection. Empirically, Lev delivers up to

72\times

speedup and improves dataset diversity, measured by the Vendi score, by

9.2\%

over the strong diversification baseline DiSF. On CommonCrawl (CC) web data selection, Lev improves accuracy across seven downstream tasks by up to

1.31\%

over existing baselines. For domains where robust quality criteria are inherently difficult to define (e.g., code), Lev serves as an effective unsupervised curation alternative: on StarCoderData, the selected subset reduces bits-per-byte by

3.08\%

over DiSF. Notably, we uncover a cross-domain collapse of quality filtering: CC data filtered by DCLM-fastText fail to retain sufficient code-related content, yielding inferior code performance relative to Lev-selected data. These findings advocate for integrating diversity-aware practices into quality filtering for more effective data curation in language model pretraining.

Abstract

Language Model Pretraining

Explore similar work

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

Growing a Tail: Increasing Output Diversity in Large Language Models

Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs