Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

Hafsteinn Einarsson

cs.CLOct 26, 2026

Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

Authors: Hafsteinn Einarsson

Organizations: University of Iceland

Abstract

IceBERT, the reference encoder for Icelandic, found that a cleaned web crawl is enough to train a strong model and that starting from a multilingual model can help on some tasks. We revisit both findings with the ModernBERT architecture. We train Icelandic encoders from 12M to 1B parameters on 9B tokens of open data, and rerun IceBERT's benchmarks and five newer tasks for 30 encoders under one protocol with confidence intervals. Our 1B model trained from scratch, to our knowledge the first monolingual Icelandic encoder of that size, is significantly better than IceBERT on all eleven tasks; warm-started from the English Ettin encoder of the same shape, it has the best score of all encoders on every task we rank. Because our model shapes match the English Ettin encoders and the multilingual mmBERT, we can also test the warm start under controlled conditions, at six model sizes and four token budgets. With our vocabulary transplanted, a warm start was never significantly worse than scratch training on the same Icelandic text, and from base size up it was not significantly worse on four tasks than scratch training on 4 times the text (8 times at base and 1B). As models grow, most of the transfer comes from the source model's transformer layers, so even an English source helps. Keeping the source vocabulary instead cost 5 F1 on question answering and nearly three times the compute.

Figures & tables

Size	Layers	Hidden	Params	Tokens	Node-h
xs	7	256	12.4M	0.99B	0.1
s	10	384	25.3M	2.02B	0.2
m	19	512	59.4M	4.75B	1.1
base	22	768	136.1M	10.89B	4.5
large	28	1024	377.8M	30.23B	25
1B	28	1792	999.8M	79.98B	135

Table 1: Model sizes, with training tokens at 80 tokens per parameter (embeddings included) and node-hours on MareNostrum 5 (four H100 GPUs per node) estimated from measured throughput (1B: 135 node-hours measured, on 16 nodes). The models trained on 160 tokens per parameter see twice the tokens; the 1B model was not trained at 160. Parameter counts are below the names of the Ettin models of the same shape (17M to 1B) because our vocabulary has 32,768 tokens against Ettin’s 50,368. Appendix B gives the hyperparameters.

Icelandic and Nordic baselines
Model	Size	PoS	NER	Const.	GED	GED-5	WinoG.	QA	ScaLA	IceEC	LAS-H	LAS-M
IceBERT	163M	97.8	94.1	90.5	68.5	32.9	53.8	56.9	73.6	47.1	86.6	79.2
IceBERT-large	406M	97.8	94.3	91.2	69.8	35.6	54.1	57.7	73.2	48.5	86.9	79.0
IceBERT-XLMR-IC3	279M	97.6	93.9	90.2	64.2 ‡	30.3	50.8	55.0	67.9	46.7	86.4	78.5
ELECTRA-base (IGC)	110M	97.5	93.5	90.1	71.1	38.4	52.3	56.0	75.1	52.0	85.6	78.8
ConvBERT-base (IGC)	107M	97.6	94.1	90.5	71.3	36.5	52.4	56.4	76.3	53.2	86.0	79.3

Table 2: Fine-tuning results (

\times

100), means over five fine-tuning seeds (PoS: the ten official MIM-GOLD folds; WinoGrande: five folds

\times

three seeds). IceBERT’s tasks: PoS accuracy, NER entity F1, constituency F1 (GreynirCorpus), token-level error detection macro F1 (binary and five categories), WinoGrande accuracy. Added: QA token F1 (NQiI + RUQuAD), ScaLA and sentence-level IceEC MCC, LAS on UD IcePaHC (the 58% of test sentences not in our pretraining data) and UD Modern. Warm starts: from Ettin, or from mmBERT with our vocabulary transplanted (mmBERT-transplant) or with its own vocabulary kept (mmBERT-keep). Bold: best mean per column. † Not ranked: the source model was trained on English WinoGrande training data (Section 7.1 ). ‡ One of five fine-tuning seeds collapsed (Section 7.1 ). Appendix A gives 95% confidence intervals for these and 11 further encoders.

Figure 1: Validation MLM loss and downstream scores against Icelandic training tokens per parameter, for each arm and size. Bars are 95% confidence intervals (two pretraining seeds at xs, s and m; one at base, large and 1B). The open marker at 160 is the scratch model trained on twice the tokens. At large, the scratch curve is a second run of the recipe of the large model in Table 2 . mmBERT-keep is left out of the loss row, as its loss is per Gemma token.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	PoS	NER	Const.	GED	GED e	GED-5	GED-5 e	WinoG.
IceBERT	97.8 $\pm$ 0.3	94.1 $\pm$ 0.7	90.5 $\pm$ 0.7	68.5 $\pm$ 1.0	39.1 $\pm$ 2.1	32.9 $\pm$ 1.3	34.2 $\pm$ 2.3	53.8 $\pm$ 2.7
IceBERT-large	97.8 $\pm$ 0.3	94.3 $\pm$ 0.7	91.2 $\pm$ 0.7	69.8 $\pm$ 1.1	41.6 $\pm$ 2.0	35.6 $\pm$ 2.7	35.7 $\pm$ 2.3	54.1 $\pm$ 2.9
IceBERT-XLMR-IC3	97.6 $\pm$ 0.3	93.9 $\pm$ 0.7	90.2 $\pm$ 0.8	64.2 $\pm$ 7.8 ‡	30.4 $\pm$ 15.5	30.3 $\pm$ 1.3	31.6 $\pm$ 2.2	50.8 $\pm$ 2.5
ELECTRA-base (IGC)	97.5 $\pm$ 0.3	93.5 $\pm$ 0.7	90.1 $\pm$ 0.7	71.1 $\pm$ 1.0	44.3 $\pm$ 2.0	38.4 $\pm$ 3.5	38.0 $\pm$ 2.2	52.3 $\pm$ 3.0
ConvBERT-base (IGC)	97.6 $\pm$ 0.3	94.1 $\pm$ 0.7	90.5 $\pm$ 0.6	71.3 $\pm$ 1.0	44.5 $\pm$ 2.0	36.5 $\pm$ 2.3	38.6 $\pm$ 2.3	52.4 $\pm$ 2.7
ScandiBERT	97.6 $\pm$ 0.3	93.5 $\pm$ 0.7	90.2 $\pm$ 0.7	68.2 $\pm$ 1.0	38.5 $\pm$ 2.1	33.4 $\pm$ 2.6	33.7 $\pm$ 2.3	53.9 $\pm$ 2.6

Appendix

Table 3: Means and 95% confidence intervals (

\pm

the larger half-width) for IceBERT’s tasks, all 30 encoders; hierarchical bootstrap over fine-tuning seeds and test items (1,000 draws). Groups as in Table 2 ; the scratch models are named by size and tokens per parameter. GED e and GED-5 e : F1 of the error classes alone (micro-averaged over the error tokens); GED and GED-5 average F1 over all classes, the no-error class included, as IceBERT did. † , ‡ : as in Table 2 .

Model	QA	ScaLA	IceEC	LAS-H	LAS-H (all)	LAS-M
IceBERT	56.9 $\pm$ 1.5	73.6 $\pm$ 3.8	47.1 $\pm$ 2.3	86.6 $\pm$ 0.4	87.2 $\pm$ 0.3	79.2 $\pm$ 1.0
IceBERT-large	57.7 $\pm$ 1.6	73.2 $\pm$ 4.0	48.5 $\pm$ 2.3	86.9 $\pm$ 0.5	87.4 $\pm$ 0.3	79.0 $\pm$ 0.9
IceBERT-XLMR-IC3	55.0 $\pm$ 1.5	67.9 $\pm$ 4.0	46.7 $\pm$ 2.5	86.4 $\pm$ 0.5	86.9 $\pm$ 0.3	78.5 $\pm$ 1.0
ELECTRA-base (IGC)	56.0 $\pm$ 1.5	75.1 $\pm$ 3.7	52.0 $\pm$ 2.4	85.6 $\pm$ 0.5	86.2 $\pm$ 0.3	78.8 $\pm$ 1.0
ConvBERT-base (IGC)	56.4 $\pm$ 1.4	76.3 $\pm$ 3.9	53.2 $\pm$ 2.4	86.0 $\pm$ 0.4	86.5 $\pm$ 0.3	79.3 $\pm$ 1.0
ScandiBERT	54.3 $\pm$ 2.7	68.1 $\pm$ 4.5	46.7 $\pm$ 2.6	86.4 $\pm$ 0.4	86.8 $\pm$ 0.3	78.6 $\pm$ 0.9

Appendix

Table 4: Means and 95% confidence intervals (

\pm

the larger half-width) for added tasks, all 30 encoders; hierarchical bootstrap over fine-tuning seeds and test items (1,000 draws). Groups as in Table 2 ; the scratch models are named by size and tokens per parameter. LAS-H: the clean 58% of the IcePaHC test sentences; LAS-H (all): the full test set. † , ‡ : as in Table 2 .

			vs. scratch, same budget				vs. scratch
Task	Size	Arm	10	20	40	80	80 vs. 160
QA	xs	Ettin	+8.5 [+7.3, +9.8]	+5.7 [+3.9, +7.4]	+2.2 [+0.4, +4.4]	+0.4 [-1.1, +1.8]	+2.2 [+0.8, +3.5]
	xs	Ettin-body	+4.3 [+1.7, +6.4]	+3.3 [+1.5, +5.0]	+1.8 [-0.2, +3.9]	-0.3 [-1.6, +1.0]	+1.4 [+0.2, +2.8]
	s	Ettin	+4.1 [+3.0, +5.1]	+2.1 [+0.9, +3.2]	+2.2 [+1.0, +3.6]	+2.0 [+1.0, +3.1]	+1.4 [+0.4, +2.3]
	s	Ettin-body	+3.0 [+2.0, +4.1]	+1.6 [+0.4, +2.6]	+0.6 [-1.6, +2.5]	+1.2 [+0.02, +2.5]	+0.6 [-0.5, +1.9]
	m	Ettin	+5.9 [+4.3, +7.7]	+4.8 [+2.7, +7.1]	+4.0 [+2.5, +5.4]	+3.0 [+1.5, +4.5]	+1.3 [+0.1, +2.6]

Appendix

Table 5: Warm start minus scratch (

\times

100) at the same size and token budget, and warm start at 80 tokens per parameter minus scratch at 160, with paired 95% confidence intervals (hierarchical bootstrap over pretraining seeds, fine-tuning seeds and test items); bold where the interval excludes zero (bounds that round to 0.0 are given to two decimals). Two pretraining seeds per arm at xs, s and m (one for Ettin-body), one at base, large and 1B. PoS is measured at 80 tokens per parameter only; 1B has no scratch model at 160. The 170 same-budget differences counted in Section 7.3 are the same-budget cells of Ettin, Ettin-body and mmBERT-transplant on the five tasks.

Task	Size	Arm	10	20	40	80
QA	xs	Ettin	+1.6 [+0.5, +2.6]	+2.0 [+0.9, +3.2]	+2.3 [+1.2, +3.4]	+2.2 [+0.8, +3.5]
	xs	Ettin-body	-2.7 [-5.3, -0.7]	-0.3 [-1.7, +1.1]	+1.8 [+0.3, +3.2]	+1.4 [+0.2, +2.8]
	s	Ettin	+0.4 [-0.7, +1.4]	0.0 [-1.3, +1.2]	+0.5 [-0.6, +1.5]	+1.4 [+0.4, +2.3]
	s	Ettin-body	-0.7 [-1.9, +0.5]	-0.5 [-1.8, +0.7]	-1.2 [-3.2, +0.5]	+0.6 [-0.5, +1.9]
	m	Ettin	+0.9 [-0.1, +2.0]	+1.3 [-0.1, +2.6]	+1.0 [+0.1, +2.1]	+1.3 [+0.1, +2.6]
	m	Ettin-body	+0.5 [-0.5, +1.6]	+0.7 [-0.4, +1.8]	+0.9 [+0.02, +1.8]	+1.3 [+0.3, +2.2]

Appendix

Table 6: Warm start at each budget (tokens per parameter) minus scratch at 160 tokens per parameter (

\times

100), with paired 95% confidence intervals; bold where the interval excludes zero (bounds that round to 0.0 are given to two decimals). These are the comparisons behind the data savings in Section 7.3 .

Size	Heads	GeGLU	Peak LR	Batch	Warmup	WD
xs	4	384	2e-3	0.26M	30M	3e-4
s	6	576	2e-3	0.26M	61M	3e-4
m	8	768	1.5e-3	0.52M	143M	3e-4
base	12	1152	8e-4	1.05M	327M	1e-5
large	16	2624	4e-4	1.05M	907M	1e-5
1B	28	3840	3e-4	2.10M	2.40B	5e-5

Appendix

Table 7: Pretraining hyperparameters per size (layers and width are in Table 1 ). GeGLU: inner width of the feed-forward layer. Batch and warmup in tokens; warmup is 3% of the 80-token budget. WD: decoupled weight decay.

Source	Docs in	Shared	Tokens
IGC-2024 (open)	2.15M	22%	2.01B
FinePDFs ∗	0.36M	8%	1.86B
FineWeb-2 ∗	3.05M	84%	1.61B
HPLT 3.0 ∗	4.30M	65%	1.38B
MaCoCu ∗	2.42M	34%	1.17B
MADLAD-400 ∗	1.56M	63%	0.56B

Appendix

Table 8: Pretraining sources. Shared : share of a source’s documents with a near-duplicate in another source. Tokens : training tokens after filtering, deduplication in the listed priority order (IGC first) and decontamination, in our 32k tokenizer. ∗ Used in the 8,192-token phase (IGC and IC3-v2 do not keep paragraph order). CulturaX was added for the 160 tokens per parameter models only, deduplicated last against the nine sources, which stay unchanged.

Task		reused rate	re-tuned rate
QA	Scratch	1e-4: 51.6	1e-4: 51.6
	Ettin	1e-4: 56.5	4e-4: 55.8
	Ettin $-$ Scratch	+4.9 [+3.5, +6.3]	+4.2 [+2.8, +5.5]
ScaLA	Scratch	6e-5: 52.7	3e-5: 56.3
	Ettin	1e-4: 60.9	3e-5: 62.9
	Ettin $-$ Scratch	+8.2 [+3.3, +13.1]	+6.5 [+2.1, +11.0]

Appendix

Table 9: Re-tuning the fine-tuning learning rate at 10 tokens per parameter (base size): the rate and test score (

\times

100, mean of three seeds) with the rate reused from the 80-token model and with the rate chosen on development data from the 80-token model’s grid, and the paired difference with its 95% interval (bold: excludes zero).

Model	1k	2k	4k	8k	1k $-$ 8k
xs, Scratch	2.533	2.464	2.462	2.470	+0.063 [+0.041, +0.089]
s, Scratch	2.090	2.025	2.009	2.016	+0.075 [+0.054, +0.099]
m, Scratch	1.673	1.620	1.602	1.604	+0.069 [+0.051, +0.089]
base, Scratch	1.399	1.360	1.349	1.351	+0.048 [+0.034, +0.066]
large, Scratch	1.117	1.076	1.064	1.066	+0.051 [+0.038, +0.067]
1B, Scratch	0.884	0.855	0.846	0.847	+0.037 [+0.028, +0.048]

Appendix

Table 10: Masked-token loss (nats) on the last 1,024 positions of 54 held-out documents of at least 8,192 tokens, when the input is the last 1k, 2k, 4k or all 8k tokens, for the models at 80 tokens per parameter; the last column is the gain from the full context with its 95% interval over documents. The gain is positive in all 25 of the 25 models we ran with our vocabulary.

Model	Task	full rate	half rate	half $-$ full	full $-$ Scratch	half $-$ Scratch
Ettin	MLM loss	1.384	1.557	+0.173
	QA	56.5	56.5	0.0 [-0.8, +0.9]	+4.9 [+3.4, +6.3]	+4.9 [+3.6, +6.3]
	NER	92.9	92.5	-0.4 [-0.9, +0.1]	+2.1 [+1.4, +2.8]	+1.7 [+1.1, +2.4]
	ScaLA	60.9	54.3	-6.6 [-11.1, -2.1]	+8.2 [+3.5, +13.2]	+1.7 [-3.4, +6.9]
	IceEC	46.3	43.6	-2.6 [-4.7, -0.7]	+4.1 [+2.2, +6.3]	+1.5 [-0.9, +3.6]
mmBERT-keep	MLM loss	0.430	0.440	+0.009

Appendix

Table 11: Pretraining at half the scratch peak learning rate, at base size and 10 tokens per parameter: the warm-started model trained at the full rate (the study’s branch) and one trained straight at half the rate, both fine-tuned at the branch’s rate. MLM loss is the mean validation loss at the end (mmBERT-keep’s is per Gemma token, so it compares only with itself); task scores are test

\times

100, mean of three seeds, with paired differences and 95% intervals (bold: excludes zero). Scratch is the 10-token branch of the scratch curve.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 447 papers

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Feb 24, 2026Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo +4Multilingual Language Models Language Model Pretraining
Barcelona Supercomputing Center
Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
Dec 28, 2025Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1Language Model Pretraining Long-Context Language Modeling
Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany

Feb 24, 2026J/K move · Enter open · S save

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo, Severino Da Dalt+3

Barcelona Supercomputing Center

Multilingual Language Models Language Model Pretraining Domain Adaptation for LLMs Efficient Language Model Inference

Open paper AskPDF arXiv

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.

Abstract

Language Model Pretraining

Explore similar work

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Polish ModernBERT: The Long and Short of Polish Language Understanding

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish