EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Authors: Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts
Organizations: Institute of Computer Science, University of Tartu, Tartu, Estonia · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia · Institute of the Estonian Language, Tallinn, Estonia · School of Humanities, Tallinn University, Tallinn, Estonia
Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can improve Estonian capabilities in multilingual LLMs while preserving English and general reasoning performance. Using Llama 3.1 8B and Apertus 8B as base models, we apply CPT with Estonian-enriched multilingual replay, followed by mostly English supervised fine-tuning, preference optimization, and chat vector merging. Evaluation on Estonian benchmarks, targeted pairwise human evaluation, and an Estonian Chatbot Arena-style setup shows consistent improvements in Estonian language competence, reasoning, translation, and instruction-following. Although Apertus exhibits stronger Estonian capabilities before adaptation, the more English-centric Llama model achieves substantially larger gains after adaptation. While some English capabilities regress relative to the original instruction-tuned models, chat vector merging substantially restores English instruction-following and reasoning performance. These findings suggest that CPT with balanced multilingual replay and lightweight post-training alignment can substantially improve single-language capabilities in multilingual LLMs.
Figures & tables
Figure 1: Overview of the EstLLM adaptation pipeline.
Domain
Source
Tokens
Estonian
Estonian National Corpus
8.6B
English
Cosmopedia v2
6.9B
Code
Python-Edu
3.3B
Math
FineMath-4+
9.5B
Instructions
Instruct-PT
7.4B
Total
35.7B
Table 1: Continued pretraining data mixture. Token counts produced using the Llama 3.1 tokenizer.
ET Discriminative Benchmarks ( acc % )
Average
Model
Belebele
Exam
Grammar
Declension
Trivia
WinoGr.
XCOPA
GPIQA
Score
Rank
Baselines:
EuroLLM-9B
69.9
61.8
66.3
44.0
37.1
69.2
71.2
69.0
61.1
4.4
Ministral-3-8B-Base-2512
26.3
52.8
64.1
58.5
31.6
62.3
56.0
60.0
51.4
6.5
Llammas-base
38.7
46.2
53.8
26.9
33.6
69.7
68.6
76.0
51.7
5.9
salamandra-7B
44.8
50.5
69.9
26.8
29.6
67.3
65.8
71.0
53.2
6.4
Table 2: Base model performance comparison across Estonian language benchmarks
EN Discriminative Bench. ( acc % )
Average
MT (BLEU)
Avg
Model
Belebele
WinoGr.
MMLU-R.
Score
Rank
EN-ET
ET-EN
Rank
Baselines:
EuroLLM-9B
77.3
73.2
55.7
68.7
7.0
29.0
41.2
1.0
Ministral-3-8B-Base-2512
89.7
77.1
72.9
79.9
2.0
12.6
29.6
7.5
Llammas-base
45.0
72.0
35.0
50.7
8.7
22.0
32.7
5.5
salamandra-7b
53.1
70.6
44.9
56.2
8.3
14.7
18.2
7.5
Table 3: Base model performance comparison on English language benchmarks and FLORES200 generative machine translation (MT) benchmark (BLEU).
Knowledge and Reasoning
Language competence
IFEval
Wino
Global
Truthful
Declen-
Word
Model
ET
grande
Trivia
Exam
PIQA
QA
Grammar
sion
Meanings
EuroLLM-9B-Instruct
54.0
58.5
37.4
55.9
55.0
28.9
76.4
36.7
92.6
Ministral-3-8B-Instruct-2512
48.9
58.1
31.3
50.1
48.0
35.3
56.2
48.3
84.0
Apertus-8B-Instruct-2509
54.8
51.1
34.5
55.2
51.0
36.6
51.2
36.6
90.3
salamandra-7b-instruct
51.9
28.8
28.7
35.6
55.0
30.1
59.4
26.7
80.8
Table 4: Estonian benchmark results of instruction-following models.
IFEval
Knowledge and Reasoning
Model
EN
Winogrande
PIQA
TruthfulQA
MMLU-Redux
GSM8K
EuroLLM-9B-Instruct
70.0
50.6
58.0
29.6
57.4
59.4
Ministral-3-8B-Instruct-2512
68.5
65.0
77.0
51.9
74.2
39.3
Apertus-8B-Instruct-2509
78.1
51.3
60.0
39.1
61.1
56.8
Llama-3.1-8B-Instruct
81.1
56.3
76.0
52.4
69.6
77.1
Llammas
43.7
49.8
0.0
19.7
34.2
14.6
Table 5: English benchmark results for instruction-tuned models.
Figure 2: Pairwise human evaluation results (counts of left/tie/right preference). Significantly better model determined using bootstrap resampling indicated with * ( p=0.05 ).
Rank
Model
Size
Score
1
DeepSeek-V3 (0324)
685B
1457
1
Kimi-K2-Instruct
1T
1442
1
Llama 4 Maverick
402B
1432
1
Gemma-3-27B-it
27B
1413
1
Llama-EstLLM-8B-Instruct-CV
8B
1384
2
Llama-EstLLM-8B-Instruct
8B
1376
Table 6: Snapshot of the open-model leaderboard from an Estonian Chatbot Arena style evaluation platform.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Type
Source
Pairs
Estonian
natural
Language Help
46K
Estonian
natural
EstNER
17K
Estonian
natural
EstCOPA
800
Estonian
natural
EIC
200
Estonian
synthetic
EuroLLM
175
English
natural
Flan V2
58K
Appendix
Table 7: Instruction tuning mixture components and the number of instruction-response pairs in each component.
Hyperparameter
Llama-3.1-8B
Apertus-8B
Learning rate
5e-5
Optimizer
AdamW
Adam ϵ
1e-8
Adam β1
0.9
Adam β2
0.95
Sequence length
4096
Appendix
Table 8: Continued pre-training (CPT) hyperparameters. Apertus-8B CPT uses the same setting as Llama-3.1-8B if not stated otherwise.
Epochs
Disc. ( Avg acc % )
MT ( BLEU )
ET
EN
EN-ET
ET-EN
No training
54.0
76.9
13.5
33.7
Epoch 1
69.5
75.4
28.1
36.8
Epoch 2
70.2
75.5
27.0
36.4
Epoch 3
70.5
74.4
28.2
36.4
Appendix
Table 9: Repeating Estonian data: average accuracy of discriminative benchmarks and FLORES-200 BLEU scores (full results in Table 10 ).
Epoch
Benchmark
0
1
2
3
Discriminative ( acc )
ET
Exam
44.7
57.0
57.5
57.2
Belebele
67.0
77.2
75.7
77.6
GlobalPIQA
53.0
78.0
79.0
81.0
Grammar
65.8
87.5
89.7
85.0
Appendix
Table 10: Benchmark results depending on how many epochs the Estonian training data was repeated ( Llama-3.1-8B ). Epoch=0 is the base model before CPT. Full results in Appendix C.1 Table 10 .
Training DS
∣D∣
Avg Acc. %
Avg Rank
no training
0
54.0
3.1
ENC
8.6B
62.2
1.7
FW2
15.9B
61.3
2.9
ET FW mix
19.1B
59.6
2.1
Appendix
Table 11: Estonian discriminative benchmark scores after continued pretraining with Estonian corpora (monolingual). ∣D∣ is the dataset size in Llama 3.1 tokens. Full results in Appendix 3.1 Table 12 .
Dataset
∣D∣
Exam
Belebele
Grammar
Declension
Trivia
XCOPA
WinoGrande
Avg Acc
Avg Rank
no training
0
44.7
67.0
65.8
58.7
30.0
53.2
59.6
54.1
3.1
ENC
8.6B
54.6
67.0
72.1
48.2
45.6
75.4
72.8
62.2
1.7
FW2
15.9B
53.4
65.6
71.6
48.0
41.9
74.4
74.2
61.3
2.9
ET FW mix
19.1B
55.1
67.4
59.0
42.7
43.4
75.0
74.5
59.6
2.1
Appendix
Table 12: Estonian discriminative benchmark accuracies (%) after continued pretraining with Estonian corpora (monolingual).
Figure 3: Pairwise human evaluation results. Significant results indicated with * ( p=0.05 ).
Figure 4: Pairwise human evaluation results. Significant results indicated with * ( p=0.05 ).
Figure 5: Annotation interface used for human evaluation. The interface provides task instructions and definitions of the evaluation categories (shown in yellow), side-by-side model responses (shown in teal) to two consecutive prompts (shown in blue), and checkboxes for indicating preferences in each category. Original interface text is shown in Estonian. The English translation of the instructions is as follows: “ Read the conversations held with two anonymous models thoroughly. Then evaluate the models’ responses in the categories provided below. A conversation consists of two prompts and two responses. Choose the better model in each category. If the models’ responses are equally good or equally bad, choose a tie. Base your decision on the definitions of the categories listed here. ”
Benchmark
Num fewshot
Belebele (EN/ET)
5
Exam
5
FLORES
5
GlobalPIQA
0
Grammar
5
Declension
5
Appendix
Table 13: Number of few-shot examples in base model evaluation.
No
Rank
Model
Model size
Score
95% CI
Votes
1
1
DeepSeek-V3 (0324)
685B
1457
− 28/ + 41
964
2
1
Kimi-K2-Instruct
1T
1442
− 40/ + 36
729
3
1
Llama 4 Maverick
402B
1432
− 31/ + 38
749
4
1
Gemma-3-27B-it
27B
1413
− 38/ + 35
750
5
1
Llama-EstLLM-8B-Instruct-CV
8B
1384
− 47/ + 53
110
6
2
Llama-EstLLM-8B-Instruct
8B
1376
− 40/ + 39
541
Appendix
Table 14: Top-20 ranked open models leaderboard from the Chatbot Arena style evaluation page in Estonian.
Figure 6: Results of base model SLERP merging, where t=0 is Llama-3.1-8B and t=1 is Llama-3.1-EstLLM-8B . The colored area represents the 95% confidence interval calculated from the standard error reported by LM evaluation harness Gao et al. (2024) .