CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

João Silva; Luís Gomes; António Branco

cs.CLMar 13, 2026

CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

Authors: João Silva, Luís Gomes, António Branco

Organizations: University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal

Abstract

This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.

Figures & tables

Benchmark	Type	Size
Tuguesice	Cultural align.	327
DoNotAnswer	Safeguards	939
MuSR	Chain-of-though	756
Omniscience	NLU Q&A	600
MMLU	NLU Q&A	14,042
GPQA Diamond	NLU Q&A	198

Table 1: The test benchmarks.

Figure 3: Screenshot of the main leaderboard page. The top right has links to the “About” page and to the page where evaluation requests are submitted. The various feature selection and filtering buttons are automatically populated depending on the existing results.

Model	Tug.	DNA	MuSR	Omni.	GPQA	MMLU	M.Pro	CoPA	MRPC	RTE
Gerv. 70B	39.76	86.9	54.50	16.53	45.96	82.04	58.67	96.00	79.13	90.97
Llama 70B	25.69	91.8	57.80	15.50	45.96	81.67	61.61	95.00	72.93	89.89
Mistral 24B	15.60	88.5	32.94	5.69	45.45	71.02	47.75	94.00	48.23	84.12
Gerv. 8B	11.31	83.6	24.07	4.58	34.85	62.15	36.79	87.00	77.45	77.62
Llama 8B	9.79	83.6	23.94	3.58	32.32	61.49	36.10	83.00	75.25	79.42

Table 2: Current leaderboard results. Some names are abbreviated for readability (Tug. is Tuguesice-PT, DNA is DoNotAnswer-PT, Omni. is AA-Omniscience-Public, GPQA is GPQA Diamond, and M.Pro is MMLU Pro).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Mar 27, 2026Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19Portuguese Language Models LLM Training
NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações +2
ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Mar 27, 2026Inês Vieira, Inês Calvo, Iago Paulo +6LLM Evaluation Language Model Generation Evaluation
1NOVA University of Lisbon, Portugal · 2NOVA LINCS
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil

Mar 27, 2026J/K move · Enter open · S save

AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares+18

NOVA School of Science and Technology · NOVA LINCS · Instituto de Telecomunicações+2

Portuguese Language Models LLM Training Low-Resource Language Processing LLM Evaluation Multilingual Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.

Abstract

LLM Evaluation

Explore similar work

AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese