Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs
Authors: Vinicius Anjos de Almeida, Sandro Saorin da Silva, Josimar Chire, Leonardo Vicenzi, Nícolas Henrique Borges, Helena Kociolek, Sarah Miriã de Castro Rocha, Frederico Nassif Gomes, +3 more
Organizations: Spesia, Curitiba - PR, Brazil · Faculdade de Medicina, Universidade de São Paulo, São Paulo - SP, Brazil · Pontifícia Universidade Católica do Paraná (PUCPR), Curitiba - PR, Brazil · Universidade Federal do Paraná (UFPR), Curitiba - PR, Brazil · Florida Atlantic University, Boca Raton - FL, USA
Clinical notes contain valuable unstructured information. Named entity recognition (NER) enables the automatic extraction of medical concepts; however, benchmarks for Portuguese remain scarce. In this study, we aimed to evaluate BERT-based models and large language models (LLMs) for clinical NER in Portuguese and to test strategies for addressing multilabel imbalance. We compared BioBERTpt, BERTimbau, ModernBERT, and mmBERT with LLMs such as GPT-5 and Gemini-2.5, using the public SemClinBr corpus and a private breast cancer dataset. Models were trained under identical conditions and evaluated using precision, recall, and F1-score. Iterative stratification, weighted loss, and oversampling were explored to mitigate class imbalance. The mmBERT-base model achieved the best performance (micro F1 = 0.76), outperforming all other models. Iterative stratification improved class balance and overall performance. Multilingual BERT models, particularly mmBERT, perform strongly for Portuguese clinical NER and can run locally with limited computational resources. Balanced data-splitting strategies further enhance performance.
Figures & tables
Entity
Number of records
Examples
Procedures
984
‘medicada’, ‘soroterapia’, ‘mudança de decúbito’
Disorders
979
‘sonolenta’, ‘dependente’, ‘diabetes’
Concepts & Ideas
887
‘2 dias’, ‘2 anos’
Living Beings
715
‘filho’, ‘neto’
Anatomy
705
‘calcâneo’, ‘pescoço’, ‘perna E’
Chemicals & Drugs
601
‘tazocin’, ‘ATB’, ‘cálcio’
Table 1: Number of records per entity in the SemClinBr dataset and entity examples. The UMLS semantic groups ‘Genes & Molecular Sequences’, ‘Occupations’, and ‘Geographic Areas’ were not included in this study due to the small number of examples in the dataset.
Entity
Number of records
Example (entity in bold)
ER
346
‘IHQ: RE- /RP-/Her2 3+/ gata3 +++’
HER2
343
‘IHQ: RE-/RP-/ Her2 3+ / gata3 +++’
Histological type
336
‘ adenocarcinoma pouco diferenciado associado a…’
PR
335
‘IHQ: RE-/ RP- /Her2 3+/ gata3 +++’
Surgery
296
‘ Mastectomia mama dir + EA em 2011’
Menopause status
212
‘Paciente de 51 anos, pré-menopausa ’
Table 2: Number of records per entity in the breast cancer clinical notes dataset and entity examples. ER, estrogen receptor; HER2, human epidermal growth factor receptor 2; PR, progesterone receptor; BRCA, breast cancer susceptibility gene; FISH, fluorescence in situ hybridization; CISH, chromogenic in situ hybridization; SISH, silver in situ hybridization.
Role
Prompt
system
Extract the following entities from the given clinical note in order of appearance. Use exact text for extractions. Do not paraphrase or overlap entities. The entities to extract are: {entities} . See the examples below for the expected format. # Clinical note example: {example_text} # Ideal response example: {example_response}
user
# Clinical note: {input_text}
Table 3: Prompt strategy used for entity extraction in clinical notes using LLMs. The fields defined with curly brackets were replaced by the respective entities, text and JSON object expected as the response.
Effort level
Time
Token usage
Cost
minimal
18.3
3064
9.57
low
81.4
6090
39.83
medium
115.5
10,508
84.01
Table 4: Resource usage of the gpt-5 model in each reasoning effort level. The measures of time, token usage and cost represent the average across all responses. Time is measured in seconds. Cost is estimated per 1000 responses and is presented in USD considering the pricing at November 14th, 2025. Token usage refers to the total number of tokens (prompt and completion tokens).
Model
SemClinBr
Breast cancer dataset
Macro F1-score
Micro F1-score
Macro F1-score
Micro F1-score
IO
BIO
IO
BIO
IO
BIO
IO
BIO
BioBERTpt
biobertpt-all
0.6443
0.6207
0.6830
0.6573
0.7473
0.3671
0.7602
0.6124
biobertpt-clin
0.6489
0.6470
0.6981
0.6959
0.7527
0.3982
0.7728
0.6508
biobertpt-bio
0.5839
0.5543
0.6548
0.6208
0.7268
0.3741
0.7458
0.6319
Table 5: Benchmark metrics for each model (BERT and LLMs) and each dataset. With respect to BERT models, the reported results reflect the use of weighted loss function. Iterative stratification was used to split the data. Results may vary due to stochastic processes on training, such as in weight initialization and token sampling in the case of LLMs. Specific LLMs versions: gpt-5 refers to gpt-5-2025-08-07 ; gpt-5-nano, gpt-5-nano-2025-08-07 ; gpt-5-mini, gpt-5-mini-2025-08-07 ; gpt-4.1, gpt-4.1-2025-04-14 . Gemini models are stable models and have the same name in the API. * ‘minimal’, ‘low’, ‘medium’ and ‘high’ refer to the reasoning effort. ** Inference with gpt-5 with high reasoning effort was aborted due to very long inference time and higher cost, even with optimized prompt.
Methods
Macro F1-score ( ± std)
Micro F1-score ( ± std)
SemClinBr
Random data split
0.6567 ( ± 0.0200)
0.6904 ( ± 0.0287)
IS (Iterative Stratification)
0.7139 ( ± 0.0050)
0.7646 ( ± 0.0094)
IS with weighted loss
0.6672 ( ± 0.0094)
0.6773 ( ± 0.0140)
IS with weighted loss and clamped weights
0.7118 ( ± 0.0125)
0.7604 ( ± 0.0202)
IS with OS (oversampling)
0.7154( ± 0.0117)
0.7634 ( ± 0.0073)
Table 6: Comparison of methods to deal with imbalanced classes using mmBERT-base model. For this comparison, both datasets were used with the IO data annotation scheme. Training for each data split method was repeated for 10 iterations. The mean and standard deviation (std) of the micro and macro F1-score are reported. Clamped weights, in this context, mean fixing the minimum weight for the classes at 1.0.
Figure 1: Left image: precision-recall curve using random data split. Right image: precision-recall curve using iterative stratification. The iterative stratification results in curves that better approximate the top-right corner of the plot, demonstrating better precision-recall trade-off.