As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT.
Figures & tables
Figure 1: ALBA allows the assessment of LLM generative capabilities across eight linguistic dimensions.
Dimension
Example
Language Variety
Transforma esta frase que está em português do Brasil e coloca-a em português Europeu. Respeita o registo: se houver termos informais ou gíria na frase, tenta encontrar equivalentes na língua de chegada. "Valeu, cara. A gente leva ele junto."
Culture-bound Semantics
Dá-me exemplos de expressões portuguesas que mencionem chuva.
Discourse Analysis
Podes criar uma frase com dois pleonasmos?
Word Plays
Dá-me um exemplo de um pangrama ou pantograma em português. Inclui todas as letras e acentos do alfabeto português europeu.
Syntax
Coloca esta frase na forma passiva: "Ele vendeu a mota."
Morphology
Cria uma lista de dez palavras com oito letras que acabem em "ão".
Table 1: Example questions of all 8 ALBA dimensions.
Input Question
Correct Output (5)
Moderate Output (4)
Incorrect Output (1)
Qual é o significado da expressão "lágrimas de crocodilo"?
A expressão "lágrimas de crocodilo" representa lágrimas que são fingidas ou exageradas e é usada para descrever um sofrimento que não é genuíno. Exemplo: "Antes de chegar a polícia, aquela mulher esteve aqui meia hora aos berros a dizer que o acidente era culpa nossa. Agora está ali com lágrimas de crocodilo para ver se não perde a carta de condução."
A expressão "lágrimas de crocodilo" representa lágrimas que são fingidas ou exageradas e é usada para descrever um sofrimento que não é genuíno. Exemplo: "Antes da polícia chegar, aquela mulher esteve aqui meia hora gritando, dizendo que o acidente era culpa nossa. Agora tá ali com lágrimas de crocodilo, tentando não perder a carteira de motorista."
A expressão "lágrimas de crocodilo" refere-se a lágrimas provocadas por medo de crocodilos.
Table 2: Input-Output pair example from Culture-bound Semantics with accompanying expert created answers at various correctness levels.
Figure 4
Model
ALBA
Language Variety
Culture-bound Semantics
Discourse Analysis
Word Plays
Syntax
Morphology
Lexicology
Phonetics & Phonology
Fully open models
OLMo 2-7B OLMo et al. (2025)
16.9
9.5
9.5
43.3
4.0
24.5
26.0
14.0
4.8
Salamandra-7B Gonzalez-Agirre et al. (2025)
27.4
27.8
30.5
52.8
4.8
26.5
36.3
29.3
11.5
EuroLLM-9B Martins et al. (2024)
38.5
41.0
40.0
67.0
11.3
43.3
44.0
47.5
13.8
Apertus-8B Apertus (2025)
38.7
37.8
38.3
70.3
13.8
47.5
45.0
45.8
11.5
AMALIA-9B Simplício et al. (2026)
43.6
48.3
47.8
73.0
14.8
42.3
49.8
53.8
19.0
Table 3: Performance of models on ALBA across linguistic dimensions, based on LLM-as-a-judge evaluation with original 1–5 ratings rescaled to a 0–100 range. Bold indicates the best overall model, while underlining denotes the best-performing non-closed-source model.
Figure 4: Illustrative answers and judge evaluations from Ministral on various ALBA dimensions.
Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.
Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19
NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações +2
As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.
This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.
João Silva, Luís Gomes, António Branco
University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal