MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Daniel Tamayo; Iñaki Lacunza; Paula Rivera-Hidalgo; Severino Da Dalt; Javier Aula-Blasco; Aitor Gonzalez-Agirre; Marta Villegas

cs.CLFeb 24, 2026

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Authors: Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo, Severino Da Dalt, Javier Aula-Blasco, Aitor Gonzalez-Agirre, Marta Villegas

Organizations: Barcelona Supercomputing Center

Abstract

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.

Figures & tables

Figure 1: Token distribution per language for the Pre-Training phase. The table is shown in logarithmic format for visualization purposes.

task	xlm-roberta-base (279M)	mRoBERTa (283M)	mmBERT (308M)	mGTE (306M)	MrBERT (308M)
UD-POS (F1)	85.55	8 5.36	84.33	82.50	83.74
PANX (F1)	73.69	75.65	7 3.89	73.05	72.06
XNLI (Acc.)	78.25	79.09	8 0.54	77.90	81.26
PAWS-X (Acc.)	89.50	90.36	92.34	89.55	9 1.32
TyDiQA (F1)	5 6.41	53.96	63.95	51.07	56.34
MLQA (F1)	68.91	68.67	71.48	68.05	7 0.67

Table 1: Multilingual performance on XTREME benchmark tasks. Models fine-tuned on English data with learning rates selected by validation performance.

tasks	xlm-roberta -base (279M)	mRoBERTa (283M)	mmBERT (308M)	mGTE (306M)	MrBERT (308M)	MrBERT-es (150M)
UD-POS-es (F1)	99.01	99.03	99.09	98.92	99.06	9 9.08
CoNLL-NERC-es (F1)	86.91	87.77	87.01	86.96	87.42	8 7.77
STS-es (Pearson)	80.88	79.69	82.88	8 4.52	84.18	85.23
PAWS-X-es (Acc.)	90.35	91.30	9 1.35	89.70	91.25	91.90
MlDoc (Acc.)	47.67	91.28	95.10	96.13	95.28	9 5.55
Massive (Acc.)	21.89	86.45	86.79	8 7.19	87.46	87.05

Table 2: Performance on Spanish language tasks from the EvalES benchmark.

tasks	xlm-roberta -base (279M)	mRoBERTa (283M)	roberta-ca (125M)	mmBERT (308M)	mGTE (306M)	MrBERT (308M)	MrBERT-ca (150M)
AnCora-ca-ner (F1)	87.61	8 8.33	89.70	88.14	87.20	87.32	88.04
AnCora-ca-pos (F1)	98.91	98.98	99.00	9 9.01	98.77	99.01	99.03
STS-ca (Pearson)	74.67	79.52	82.99	8 3.16	78.65	83.00	85.42
TeCla (Acc.)	72.57	72.41	72.81	74.11	7 4.68	73.79	74.97
TECA (Acc.)	79.59	82.38	82.14	83.18	79.40	8 4.03	86.92
ViquiQuAD (F1)	86.93	87.86	87.31	89.86	86.78	89.25	8 9.59

Table 3: Performance on Catalan language tasks from the CLUB benchmark.

Task Name	Task Type	mmBERT (308M)	MrBERT (308M)	MrBERT -es (150M)	BioClinical- MdnBERT (150M)	Clinical MdnBERT (137M)	MrBERT -biomed (308M)
bsc-bio -distemist-ner (ES)	NER	7 8.00	77.84	78.07	75.45	70.22	77.93
cantemist (ES)	NER	78.03	68.73	7 3.40	66.68	30.91	70.78
pharmaconer (ES)	NER	8 9.66	88.58	88.97	87.66	81.69	89.92
AbSanitas (ES)	Retrieval	34.68	34.16	53.49	30.41	18.08	5 1.01
R2Med (EN)	Retrieval	10.87	1 0.15	8.65	9.97	5.91	9.76
SciDocs (EN)	Retrieval	1 0.00	9.75	9.90	9.33	3.64	10.05

Table 4: Biomedical domain evaluation on Spanish and English retrieval and classification tasks. The R2Med score is reported as the average over the bioinformatics, biology, and clinical subsets. Retrieval performance is measured using nDCG@10, while NER is evaluated using F1.

Task Name	Task Type	mmBERT (308M)	MrBERT (308M)	MrBERT-es (150M)	legal-bert- base-uncased (110M)	MrBERT -legal (308M)
LexBOE (ES)	Text Classification	96.84	9 7.02	97.28	95.36	96.80
small-spanish -legal-dataset (ES)	Retrieval	4 2.58	40.78	46.92	19.79	38.75
EURLEX (EN)	Text Classification	97.43	97.40	97.41	9 7.42	97.33
AILAStatutes (EN)	Retrieval	1 4.31	13.90	12.28	13.49	16.33
legal_summarization (EN)	Retrieval	53.33	5 3.84	46.41	52.40	55.05
LegalBench (EN)	Retrieval	6 0.15	58.88	58.26	63.42	58.04

Table 5: Legal domain evaluation on Spanish and English retrieval and classification tasks. The LegalBench score is reported as the average over the consumer contracts and corporate lobbying subsets. Retrieval tasks are evaluated using nDCG@10, while text classification tasks are evaluated using accuracy.

Figure 2: MrBERT performance across XTREME, CLUB, and EvalES benchmarks comparing AttMAT (attention head pruning), MAT (MLP hidden size reduction), and standard models (100%). Only average scores shown.

Figure 3: Inference throughput of matryoshka variants (sequence length: 8,192 tokens).

Figure 4: Performance comparison of matryoshka models at different compression levels (25%, 50%, 75%, 100% of the attention heads) against MrBERT models without matryoshka training across four benchmark tasks. Bars represent the average performance difference to the previous higher benchmark value.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset name	Languages	Domain	Usage	Citation	URL
Academic Slovene KAS 2.0	SL	Education	Pre-Train	( Erjavec et al., 2021 )	URL
[0.5pt/2pt] ACAD-Train	All	MT-Science	Pre-Train	( Lacunza et al., 2025 )	URL
[0.5pt/2pt] AEPD (juridical resolutions)	ES	Legal	Pre-Train Language Adapatation Domain Adapatation	–	Crawled 9 9 9 Crawled from https://www.aepd.es/informes-y-resoluciones/resoluciones until 09/2025.
[0.5pt/2pt] ALIA-TOURISM	ES	General	Pre-Train Language Adapatation Domain Adapatation	( Espinosa Zaragoza et al., 2025a )	URL
[0.5pt/2pt] ALIA-DOGV	ES	General	Pre-Train Language Adapatation Domain Adapatation	( Espinosa Zaragoza et al., 2025b )	URL
[0.5pt/2pt] ALIA-Legal-Administrative	ES	Legal	Pre-Train Language Adapatation Domain Adapatation	–	URL

Appendix

Table 6: Data sources used throughout this work.

Language	Tokens	Language	Tokens	Language	Tokens
EN	4,120,759,329,876	PL	44,252,717,879	LT	7,017,976,396
ES	523,633,854,143	CS	37,284,432,759	EU	6,999,041,964
DE	174,456,549,084	BG	33,060,589,015	NO	6,798,808,558
FR	172,729,856,318	CA	26,664,618,307	GL	5,173,500,585
CODE	156,565,808,119	RO	24,395,563,081	LV	4,970,822,927
RU	141,760,491,684	SK	21,968,084,510	TRANSLATIONS	3,800,022,519

Appendix

Table 7: Token distribution by language during Pre-Training phase.

Domain Class	Description
Adult	Sexual content, pornography, or age-restricted material
Arts_and_Entertainment	Music, movies, theater, celebrities, pop culture
Autos_and_Vehicles	Cars, motorbikes, vehicle news and reviews
Beauty_and_Fitness	Skincare, cosmetics, wellness, workout routines
Books_and_Literature	Novels, literary criticism, poetry, book reviews
Business_and_Industrial	Enterprise, corporate, manufacturing, B2B topics

Appendix

Table 8: Domain class descriptions for NVIDIA’s multilingual domain classifier, based on manual inspection of sample instances.

	MrBERT	MrBERT-es	MrBERT-ca	MrBERT-biomed	MrBERT-legal
Scheduler	WSD	WSD	Warmup + Cosine	Warmup + Cosine	Warmup + Cosine
Learning Rate	1e-3	4e-4	1e-3	2e-3	3e-3
Total Tokens	6,100B	615B	47.4B	24.1B	9B
Epochs	1	1	1	2	10
Warmup Tokens	3B	3B	4.7B	2.4B	9B
Decay Tokens	100B	100B	42.7B	45.9B	81B

Appendix

Table 9: List of hyperparameters chosen for each model. Throughput measurements (samples/s) were obtained from speed tests launched on a single NVIDIA H100 GPU with 64 GB of memory. Reported values have an estimated error bar of

\pm 0.1

samples/s. Each inference sample consists of 8,192 tokens.

Model	UD-POS	XNLI	PAWS-X	PANX	TyDiQA	MLQA	XQuAD	Average
	(F1)	(Acc.)	(Acc.)	(F1)	(F1)	(F1)	(F1)
MrBERT	83.74	81.26	91.32	72.06	56.34	70.67	77.91	76.19
MrBERT AttMAT (25%)	81.79	79.57	89.85	68.51	46.79	67.10	72.99	72.37
MrBERT MAT (25%)	80.53	79.08	88.98	68.27	49.05	66.98	73.73	72.37
MrBERT AttMAT (50%)	87.77	80.92	91.53	70.46	53.49	69.04	76.24	75.63
MrBERT MAT (50%)	81.71	80.31	90.38	70.47	53.63	68.70	76.48	74.53

Appendix

Table 10: Evaluation results on the Xtreme benchmark over different experiments using Matryoshka in MrBERT.

Model	AnCora-ca -ner (F1)	AnCora-ca -pos (F1)	STS-ca (Pearson)	TeCla (Acc.)	TECA (Acc.)	ViquiQuAD (F1)	XQuAD (F1)	Average
MrBERT	8 7.32	99.01	83.00	73.79	84.03	89.25	73.96	84.34
MrBERT AttMAT (25%)	86.09	98.87	77.69	74.02	79.55	85.82	68.60	81.52
MrBERT MAT (25%)	85.07	98.75	79.31	72.47	78.22	86.41	69.55	81.40
MrBERT AttMAT (50%)	86.71	98.87	78.81	74.13	8 3.66	87.49	71.11	82.97
MrBERT MAT (50%)	87.40	98.86	82.16	73.23	81.62	88.56	70.03	83.12
MrBERT AttMAT (75%)	87.20	98.86	81.73	7 4.03	82.48	87.87	72.57	83.53

Appendix

Table 11: Evaluation results on the CLUB benchmark over different experiments using Matryoshka in MrBERT and MrBERT-ca.

Model	UD-POS -es (F1)	CoNLL -NERC-es (F1)	STS-es (Pearson)	PAWS-X -es (Acc.)	MlDoc (Acc.)	Massive (Acc.)	SQAC (F1)	Average
MrBERT	99.06	87.42	84.18	91.25	95.28	87.46	81.96	89.52
MrBERT AttMAT (25%)	99.02	86.8	78.60	89.70	96.05	86.31	76.24	87.53
MrBERT MAT (25%)	98.95	86.29	84.64	90.30	94.65	86.42	77.79	88.43
MrBERT AttMAT (50%)	99.04	86.44	82.44	90.75	96.07	87.09	78.49	88.62
MrBERT MAT (50%)	99.01	86.26	82.28	90.65	95.6	86.82	78.56	88.46
MrBERT AttMAT (75%)	99.02	87.45	83.03	90.40	95.97	86.68	79.74	88.90

Appendix

Table 12: Evaluation results on the EvalES benchmark over different experiments using Matryoshka in MrBERT and MrBERT-es.

	bsc-bio- distemist -ner (ES)	cantemist (ES)	pharma -coner (ES)	AbSanitas (ES)	R2Med (EN)	SciDocs (EN)	SciFact (EN)	TREC -COVID (EN)	Average (EN)	Average (EN + ES)
MrBERT-biomed	77.93	70.78	8 9.92	51.01	9.76	10.05	3 0.25	48.76	24.71	48.56
MrBERT-biomed AttMAT (25%)	77.68	68.24	88.67	49.66	9.66	8.87	27.81	44.08	22.60	28.02
MrBERT-biomed AttMAT (50%)	77.79	68.67	90.05	53.69	9 .96	9.71	28.72	4 5.98	2 3.59	2 9.61
MrBERT-biomed AttMAT (75%)	78.28	6 9.94	89.04	52.73	10.18	9.71	30.03	39.37	22.32	28.40
MrBERT-biomed AttMAT (100%)	7 8.14	69.73	89.09	5 3.19	9.58	9 .96	30.54	41.64	22.93	28.98

Appendix

Table 13: Evaluation results on different biomedical benchmarks over attention matryoshka in MrBERT-biomed.

Model Variant	LexBOE (ES)	small -spanish -legal -dataset (ES)	EURLEX (EN)	AILA Statutes (EN)	legal_ summarization (EN)	Legal Bench (EN)	Nano Touche 2020 (EN)	Average (EN)	Average (EN + ES)
MrBERT-legal	9 6.80	38.75	97.33	16.33	55.05	5 8.04	44.74	54.30	58.15
MrBERT-legal AttMAT (25%)	96.96	39.14	97.24	13.49	53.31	56.84	45.01	53.18	57.43
MrBERT-legal AttMAT (50%)	96.73	39.84	9 7.26	10.82	53.69	56.16	46.39	52.86	57.27
MrBERT-legal AttMAT (75%)	96.66	40.31	97.25	1 5.82	5 4.87	56.88	45.23	5 4.01	5 8.15
MrBERT-legal AttMAT (100%)	96.74	4 0.07	97.24	12.92	54.48	58.30	4 5.34	53.66	57.87

Appendix

Table 14: Evaluation results on different legal benchmarks over attention matryoshka in MrBERT-legal.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
Dec 28, 2025Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1Language Model Pretraining Long-Context Language Modeling
Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Language Model Pretraining Portuguese Language Models
UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI

Dec 28, 2025J/K move · Enter open · S save

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör, Susan Üsküdarlı

Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany

Language Model Pretraining Long-Context Language Modeling Language Modeling

Open paper AskPDF arXiv

The introduction of BERT established encoder-only transformer models as a foundational paradigm in natural language processing. Encoder-only models remain the standard tool for classification, tagging and retrieval, where contextual representations and low inference cost matter more than text generation, yet Turkish lacks a monolingual encoder trained from scratch with the advances consolidated in ModernBERT (rotary positional embeddings, FlashAttention, refined normalization). We introduce TabiBERT, a monolingual Turkish encoder based on the ModernBERT architecture, pretrained from scratch for one trillion tokens sampled from an 86.58B-token multi-domain corpus of web text (72%), scientific publications (19%), source code (6%) and mathematical content (0.3%). The model supports a context length of 8,192 tokens, sixteen times that of existing Turkish BERT models, and inherits the ModernBERT architecture's efficiency at long context. For rigorous and reproducible evaluation we introduce TabiBench, a benchmark of 27 datasets across eight task categories with standardized splits and evaluation protocols, summarized as a GLUE-style macro-average on a 0-100 scale. TabiBERT leads the Turkish models in five of eight categories and BERTurk, the previous best, in six of eight; the gains concentrate on question answering (+9.55 F1) and code retrieval (+2.41 NDCG@10), while the four short-text categories are near saturation. Its average of 77.28 exceeds BERTurk's 75.66; the multilingual mmBERT reaches 78.98 with twice the parameters and three times the training tokens, at 41% more tokens per Turkish input. We release model weights, training configurations and evaluation code as a transparent and reproducible foundation for future Turkish encoder research.

Abstract

Multilingual Language Models

Explore similar work

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

Polish ModernBERT: The Long and Short of Polish Language Understanding

moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT