Pretraining and Benchmarking Modern Encoders for Latvian

Arturs Znotins

cs.CLMar 16, 2026

Pretraining and Benchmarking Modern Encoders for Latvian

Authors: Arturs Znotins

Organizations: Institute of Mathematics and Computer Science, University of Latvia, 29 Raina bulv., Riga, LV-1459, Latvia

Abstract

Encoder-only transformers remain essential for practical NLP tasks. While recent advances in multilingual models have improved cross-lingual capabilities, low-resource languages such as Latvian remain underrepresented in pretraining corpora, and few monolingual Latvian encoders currently exist. We address this gap by pretraining a suite of Latvian-specific encoders based on RoBERTa, DeBERTaV3, and ModernBERT architectures, including long-context variants, and evaluating them across a diverse set of Latvian diagnostic and linguistic benchmarks. Our models are competitive with existing monolingual and multilingual encoders while benefiting from recent architectural and efficiency advances. Our best model, lv-deberta-base (111M parameters), achieves the strongest overall performance, outperforming larger multilingual baselines and prior Latvian-specific encoders. We release all pretrained models and evaluation resources to support further research and practical applications in Latvian NLP.

Figures & tables

Web-scale and curated sources
Name	Word count (M)
FineWeb2	5371
HPLT-v2 (cleaned)	3460
Crawled news	1100
Tweets	453
Books	534

Table 1: Overview of curated Latvian text corpora. Word counts are in millions.

Model	Params	Vocab
mdeberta-v3-base	278M	250K
xlm-roberta-large	560M	250K
xlm-roberta-base	278M	250K
mmBERT-base	307M	256K
mmBERT-small	140M	256K
bert-base-multi	178M	120K

Table 2: Model size (millions of parameters) and vocabulary size (thousands).

Existing multilingual pretrained models
Model	LTEC		ScaLA		FSNER		WikiQA		COPA
	MCC	MF1	MCC	MF1	mF1 †	mF1	F1	EM	MCC	ACC
mdeberta-v3-base	49.1±1.1	65.8±1.2	54.7±2.2	75.9±1.7	86.8±0.8	81.4±1.0	64.6±1.6	49.2±2.0	17.6 ±5.5	58.8 ±2.8
xlm-roberta-large	52.0 ±1.8	68.0 ±1.9	56.0 ±0.9	77.3 ±0.4	87.2 ±0.5	81.5 ±0.8	71.6 ±0.5	57.1 ±0.7	1.0±4.5	50.5±1.7
xlm-roberta-base	47.6±2.3	64.4±2.0	44.4±1.6	70.1±1.0	83.6±1.1	78.4±0.7	51.6±3.2	39.6±1.9	2.0±2.6	51.0±1.3
mmBERT-base	45.1±1.7	62.8±1.7	44.6±0.4	71.7±0.7	80.4±1.5	74.8±2.2	64.3±1.8	49.6±2.2	16.4±2.9	58.2±1.4

Table 3: Results for sentiment classification (LTEC), linguistic acceptability (ScaLA), named entity recognition (FSNER), question answering (WikiQA), and commonsense reasoning (COPA). MCC denotes Matthews correlation coefficient; MF1 macro-F1; mF1 micro-F1; and EM exact match. † indicates scores excluding the MISC class. ⋄ marks few-shot prompting for LLMs, while all other LLM results are zero-shot. The best overall result in each column is highlighted in bold, and the best result within each model group is underlined.

Existing multilingual pretrained models
Model	UPOS	XPOS	UFeats	AllTags	Lemmas	UAS	LAS	CLAS	MLAS	BLEX
mdeberta-v3-base	98.7±0.0	93.8±0.1	97.0±0.2	93.5±0.2	97.7±0.0	94.4±0.3	92.0±0.3	90.5±0.2	86.0±0.2	87.9±0.2
xlm-roberta-large	98.8 ±0.1	94.2 ±0.2	97.2 ±0.1	94.0 ±0.2	98.0 ±0.1	94.8 ±0.3	92.5 ±0.2	91.0 ±0.3	86.9 ±0.3	88.8 ±0.2
xlm-roberta-base	98.5±0.0	93.5±0.1	96.7±0.1	93.3±0.0	97.5±0.1	93.8±0.2	91.3±0.2	89.6±0.2	85.0±0.3	87.0±0.1
mmBERT-base	98.6±0.0	93.7±0.0	96.7±0.1	93.4±0.0	97.5±0.2	93.8±0.2	91.4±0.2	89.8±0.2	85.1±0.1	87.1±0.4
mmBERT-small	98.3±0.1	93.1±0.2	96.4±0.3	92.8±0.3	97.1±0.1	93.2±0.3	90.6±0.1	88.8±0.1	83.8±0.4	85.7±0.1

Table 4: Universal Dependencies results on the Latvian UD treebank (UD v2.16) under joint multi-task fine-tuning. Evaluation is performed using the standard CoNLL-U scoring script.

Existing multilingual pretrained models
Model	Match Acc.	Sense Acc.
mdeberta-v3-base	79.6±0.6	73.5±0.7
xlm-roberta-large	80.9 ±1.1	76.9 ±0.5
xlm-roberta-base	75.3±0.6	65.4±0.6
mmBERT-base	76.9±0.8	69.3±1.0
mmBERT-small	74.1±1.3	62.1±0.5

Table 5: WSD results. Match Acc. denotes binary accuracy for context–sense pairs, and Sense Acc. denotes top-1 sense selection accuracy.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Polish ModernBERT: The Long and Short of Polish Language Understanding
Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model Pretraining Long-Context Language Model Evaluation
National Information Processing Institute, Warsaw, Poland
MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Feb 24, 2026Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo +4Multilingual Language Models Language Model Pretraining
Barcelona Supercomputing Center
LAMB: A Modern Contextual Language Model for Latin
Oct 22, 2026Aidan McCarthyLanguage Modeling Language Model Pretraining

Sep 1, 2026J/K move · Enter open · S save

Polish ModernBERT: The Long and Short of Polish Language Understanding

Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec

National Information Processing Institute, Warsaw, Poland

Language Model Pretraining Long-Context Language Model Evaluation Long-Context Language Modeling Language Modeling

Open paper AskPDF arXiv

Encoder-only Transformers remain effective for discriminative and representation-learning tasks, yet Polish encoders still largely rely on BERT/RoBERTa-style architectures. We introduce Polish ModernBERT, a family of four Polish encoders available at Base and Large scales, each with 512-token and 8K context variants. We adapt the ModernBERT pretraining recipe through staged selection experiments and release a long-context benchmark covering legal topic classification, ideological decision-direction prediction, factual-consistency assessment over literary plot summaries, and human-rights violation assessment. Across 30 tasks, Polish ModernBERT achieves the best overall performance among the evaluated Polish encoders, reaching 83.99 and 85.11 for the Base-8K and Large-8K models, respectively. On long-context tasks, the 8K variants improve over matched Polish RoBERTa-8K baselines from 67.47 to 77.15 and from 75.88 to 78.49 at the Base and Large scales, respectively. The Base-8K model achieves this gain with 22% fewer parameters (149M vs.\ 190M). Efficiency measurements in representative inference setups show lower peak memory usage and latency than matched Polish RoBERTa baselines in both 512-token and 8K settings. Polish ModernBERT-8K-Base additionally achieves the best result on a Polish retrieval benchmark among the evaluated encoders below 300M parameters.

Abstract

Language Model Pretraining

Explore similar work

Polish ModernBERT: The Long and Short of Polish Language Understanding

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

LAMB: A Modern Contextual Language Model for Latin