AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Authors: Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães
Organizations: NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações · Instituto Superior Técnico, Universidade de Lisboa · Fundação para a Ciência e Tecnologia
Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.
Figures & tables
Task
Dataset
Language
Proportion
#Tokens
License / Source
Persona-PT Instruction Following
PT
0.332%
10 741 619
Apache 2.0 / Synth. Gen.
Persona-EN Instruction Following
EN
0.807%
26 065 053
Apache 2.0 / Synth. Gen.
Persona Nemotron Instruction Following
PT
0.147%
4 765 186
Apache 2.0 / Synth. Gen.
Instruction Following
Portuguese Linguistic Instructions
PT
0.001%
18 884
Apache 2.0 / Manual
AMALIA Hardcoded
PT & EN
0.001%
41 756
Apache 2.0 / Manual
Persona-PT General
PT
5.945%
192 102 099
Apache 2.0 / Synth. Gen.
Table 1: Data mixture used in the Supervised Fine-Tuning stage. Datasets were generated synthetically (Synth. Gen.), created manually (Manual) or pre-existent in HuggingFace (HF).
Dataset
Category
Source Language
CoT
#Shots
Metric
Reference
ALBA
PT Grammar
0
LLM-Judge
New
PT Completions
pt-PT Generation
0
Accuracy
New
PT Exams
General Knowledge
✓
0
Accuracy
New
PT Exams Open Questions
General Knowledge
✓
0
LLM-Judge
New
P3B3
pt-PT/BR Generation
0
pt-PT Level
New
FRMT
Translation
5
chrF
Riley et al. (2023)
Table 2: Summary of datasets used for evaluation. New denotes novel datasets created for pt-PT evaluation. New (MT) are datasets newly translated into pt-PT via our pt-PT MT model. Note: Some BBH tasks were removed from pt-PT evaluation as they do not translate correctly to Portuguese (e.g. hyperbaton task).
Model
ARC-C
MMLU
TQA
GSM8K
IFEval
PT-E (CoT)
PT-C
PiQA
SiQA
FRMT
BBH
Fully open models
OLMo 2-7B
47.8
42.6
50.8
51.6
50.6
45.0
32.9
61.0
41.9
61.0
37.0
Salamandra-7B
52.2
47.1
52.6
18.7
24.2
37.7
44.3
70.4
44.8
65.6
36.3
EuroLLM-9B
71.2
52.7
54.9
52.5
49.4
55.2
58.6
71.7
40.8
70.4
47.6
Apertus-8B
68.3
57.1
57.9
48.8
58.6
55.7
40.0
73.4
43.2
67.2
48.3
AMALIA-9B-SFT
77.9
55.4
57.0
5 8.8
56.7
63.1
71.4
71.7
44.7
70.5
5 0.3
Table 3: Results of instruction-tuned variants. Bold represents best model and underline best fully open model.
Model
PT-Exams Open Questions
ALBA
P3B3
Fully open models
OLMo 2-7B
43.0
16.9
18.6
Salamandra-7B
34.1
27.4
42.7
EuroLLM-9B
56.1
38.5
70.5
Apertus-8B
54.7
38.7
28.1
AMALIA-9B-SFT
62.0
37.1
91.3
Table 4: Language generation results of instruction-tuned variants on the newly created pt-PT benchmarks.
Model
ADV-Bench ↓
SST ↓
XSTest ↓
Fully open models
OLMo 2-7B
6.9
20.4
17.4
Salamandra-7B
6.6
18.4
23.4
EuroLLM-9B
1.4
3.4
1.9
Apertus-8B
2.7
4.4
3.9
AMALIA-9B-SFT
1.9
3.4
1.9
Table 5: ASR results of instruction-tuned variants on safety benchmarks assessed by Qwen3Guard-8B.