Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

Vinko Sabolčec; Bettina Messmer; Yassine Turki; Martin Jaggi

cs.CLOct 8, 2026

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

Authors: Vinko Sabolčec, Bettina Messmer, Yassine Turki, Martin Jaggi

Organizations: EPFL

Abstract

Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.

Figures & tables

		GMMLU c	INCLUDE c	ARC	mARC	HS	mHS	XNLI	OBQA	XWG	Avg.
Classifier	Emb. Model
1B Model, 100B Tokens from Scratch, Training and Evaluation on English + Top 20 Languages
mFW-edu	XLM-R	0.2814	0.2944	0.4812	0.2936	0.5412	0.4034	0.4161	0.3800	0.6046	0.4106
mFW-edu	mmBERT	0.2989	0.2913	0.4897	0.2937	0.5469	0.4026	0.4121	0.3740	0.6180	0.4141
mDCLM	XLM-R	0.2845	0.2965	0.4802	0.2815	0.5651	0.4080	0.4227	0.3640	0.6277	0.4145
mDCLM	mmBERT	0.2848	0.2938	0.4554	0.2744	0.5454	0.4012	0.4142	0.3780	0.6314	0.4087

Table 1: Benchmark performance of 1B parameter multilingual LLMs trained on 100B tokens. We compare models trained on data selected using our multilingually adapted mFW-edu and mDCLM quality classifiers, paired with XLM-RoBERTa and mmBERT embeddings. We retain the top 10% of documents based on the classifier scores.

Figure 1: Benchmark performance of 1B parameter multilingual LLMs trained on 100B tokens. We compare models trained on data selected using our multilingually adapted mFW-edu classifier with mmBERT embeddings trained on a varying amount of languages, and the FW+FW2, FWHQ+FW2HQ, and FWHQ+FW2HQ + baselines. We retain the top 10% of documents based on the classifier scores, except for FW+FW2 which is not model-filtered.

1B Model, 100B Tokens from Scratch, Training and Evaluation on English + Top 20 Languages
	GMMLU c	INCLUDE c	ARC	mARC	HS	mHS	XNLI	OBQA	XWG	Avg.
Model
FW+FW2	0.2743	0.2817	0.3880	0.2586	0.5434	0.3887	0.4086	0.3180	0.6051	0.3852
FWHQ+FW2HQ	0.2893	0.2870	0.4461	0.2747	0.5641	0.4035	0.4214	0.3680	0.6308	0.4094
FWHQ+FW2HQ +	0.2952	0.2959	0.4505	0.2816	0.5762	0.4120	0.4116	0.3640	0.6221	0.4121
mFW-edu (Ours)	0.2989	0.2913	0.4897	0.2937	0.5469	0.4026	0.4121	0.3740	0.6180	0.4141

Table 2: Benchmark performance of 1B, 3B, and 8B parameter multilingual LLMs. We compare models trained on data selected using our multilingually adapted mFW-edu and mDCLM quality classifiers paired with mmBERT embeddings, and the FW+FW2, FWHQ+FW2HQ, and FWHQ+FW2HQ + baselines. We retain the top 10% of documents based on the classifier scores, except for FW+FW2 which is not model-filtered.

1B Model, 30B Tokens From Scratch, Monolingual Training and Evaluation
	Chinese	Japanese	French	Portuguese	Avg.
Model
FWHQ+FW2HQ +	0.3578	0.3453	0.3938	0.3140	0.3527
mFW-edu (Ours)	0.3358	0.3653	0.4153	0.3230	0.3598
mFW-edu ${}_{\text{Eng}}$ (Ours)	0.3541	0.3453	0.3771	0.3339	0.3526

Table 3: Benchmark performance of 1B monolingual Chinese, Japanese, French and Portuguese LLMs on the INCLUDE benchmark. We compare models trained on data selected using our multilingually adapted mFW-edu quality classifier paired with mmBERT embeddings trained on all data (mFW-edu), trained on English (mFW-edu

{}_{\text{Eng}}

), and the FWHQ+FW2HQ + baseline. We retain the top 10% of documents based on the classifier scores.

8B Model, 63B Token Cooldown from 9.5T Token Checkpoint, Training and
	China	Japan	France	Brazil	Avg. (All Subsets)
Model
Evaluation on All Available Languages
Pre-cooldown	0.4915	0.6792	0.6429	0.4000	0.5764
FWHQ+FW2HQ +	0.5424	0.7925	0.6429	0.3600	0.6143
mFW-edu (Ours)	0.5763	0.7170	0.5714	0.4000	0.6164

Table 4: Benchmark performance of 8B multilingual LLMs on the easy subset of the CulturalBench benchmark. We compare models trained on data selected using our multilingually adapted mFW-edu quality classifier paired with mmBERT embeddings trained on all data (mFW-edu), trained on English (mFW-edu

{}_{\text{Eng}}

), and the FWHQ+FW2HQ + baseline. We retain the top 10% of documents based on the classifier scores.

Figure 2: Comparison of quality scores between our mFW-edu classifier with mmBERT embeddings across selected languages-script pairs (ISO 639-3) among the top 100, evaluated on FineWeb (FW), FineWeb 2 (FW2), and synthetic encyclopedia data.

Figure 3: Comparison of our mFW-edu classifier with mmBERT embeddings against LLM-based scores ( gemma-3-27b-it , gpt-oss-120b , Qwen3.5-35B-A3B ). Top row: in-distribution (top 100), bottom row: out-of-distribution language-script pairs (beyond top 100; ISO 639-3).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Score distribution of the original classifiers and the multilingual adaptations (left), score error distribution (middle), and the correlation between between their scores (right) on English FineWeb data.

Figure 5: Score distribution of our multilingually adapted mFW-edu classifier with mmBERT embeddings on the English synthetic encyclopedia data.

1B Model, 100B Tokens from Scratch, Training and Evaluation on English + Top 20 Languages
	GMMLU c	INCLUDE c	ARC	mARC	HS	mHS	XNLI	OBQA	XWG	Avg.
Model
FW+FW2	0.2743	0.2817	0.3880	0.2586	0.5434	0.3887	0.4086	0.3180	0.6051	0.3852
FWHQ+FW2HQ	0.2893	0.2870	0.4461	0.2747	0.5641	0.4035	0.4214	0.3680	0.6308	0.4094
FWHQ+FW2HQ +	0.2952	0.2959	0.4505	0.2816	0.5762	0.4120	0.4116	0.3640	0.6221	0.4121
mFW-edu ${}_{\text{Eng}}$	0.2925	0.2954	0.4766	0.2880	0.5275	0.3999	0.4202	0.3680	0.6022	0.4078

Appendix

Table 5: Benchmark performance of 1B parameter multilingual LLMs trained on 100B tokens. We compare models trained on data selected using our multilingually adapted mFW-edu classifier paired with mmBERT embeddings trained on a varying amount of languages, and the FW+FW2, FWHQ+FW2HQ, and FWHQ+FW2HQ + baselines. We retain the top 10% of documents based on the classifier scores, except for FW+FW2 which is not model filtered.

Explore similar work

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
Apr 22, 2026Yassine Turki, Vinko Sabolčec, Bettina Messmer +1Language Model Pretraining Multilingual Language Models
Machine Learning Optimization Lab Ecole Polytechnique Fédérale de Lausanne (EPFL) Lausanne, Switzerland
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
Aug 4, 2026Adnan Al Ali, Kathy Hämmerl, Jindřich Libovický +1Multilingual Language Model Evaluation Cross-Lingual Representation Alignment
Charles University, Faculty of Mathematics and Physics, Czech Republic · Technical University of Munich, Germany · Munich Center for Machine Learning
English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
Apr 14, 2026Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin +2Cross-Lingual Reasoning in Language Models Low-Resource Language Processing
UC Santa Barbara · Amazon

Apr 22, 2026, cs.CLJ/K move · Enter open · S save

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

Yassine Turki, Vinko Sabolčec, Bettina Messmer, Martin Jaggi

Machine Learning Optimization Lab Ecole Polytechnique Fédérale de Lausanne (EPFL) Lausanne, Switzerland

Language Model Pretraining Multilingual Language Models Low-Resource Language Processing Cross-Lingual Transfer

Open paper AskPDF arXiv

As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in embedding space may show cross-lingual consistency, which would allow high-resource languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B tokens, delivering gains for high resource languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the multilingual signal.

Abstract

Explore similar work

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment \unicodex2013\unicode{x2013}\unicodex2013 Is English Enough?

English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?