Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

Gonçalo Vinagre; Rui Pedro Guerra; Pedro Gomes; Miguel Moura Ramos; Duarte Miguel Alves; Afonso Simplício; Diogo Tavares; David Semedo

cs.CLDate pending

Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

Authors: Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, +2 more

Abstract

Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.

Explore similar work

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
Apr 30, 2026Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman +8Portuguese Modernbert
SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark
May 18, 2026Khalid Yusuf DahirText Corpora Low-Resource Languages
Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders
Jul 5, 2026Lucas Hideki Takeuchi Okamura, Alexandre Alcoforado, Anna Helena Reali CostaDomain-Invariant Text Embeddings Portuguese

Apr 30, 2026 · cs.CLJ/K move · Enter open · S save

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman, Rosimeire P. Costa+7

Itaú Unibanco, São Paulo, SP, Brasil · ICTi, São Paulo, SP, Brasil

High-quality corpora are essential for advancing Natural Language Processing (NLP) in Portuguese. Building on previous encoder-only models such as BERTimbau and Albertina PT-BR, we introduce NorBERTo, a modern encoder based on the ModernBERT architecture, featuring long-context support and efficient attention mechanisms. NorBERTo is trained on Aurora-PT, a newly curated Brazilian Portuguese corpus comprising 331 billion GPT-2 tokens collected from diverse web sources and existing multilingual datasets. We systematically benchmark NorBERTo against Strong baselines on semantic similarity, textual entailment and classification tasks using standardized datasets such as ASSIN 2 and PLUE. On PLUE, NorBERTo-large achieves the best results among the encoder models we evaluated, notably reaching 0.9191 F1 on MRPC and 0.7689 accuracy on RTE. On ASSIN 2, NorBERTo-large attains the highest entailment F1 (~0.904) among all encoders considered, although Albertina-900M and BERTimbau-large still hold an advantage. To the best of our knowledge, Aurora-PT is currently the largest openly available monolingual Portuguese corpus, surpassing previous resources. NorBERTo provides a modern, mid-sized encoder designed for realistic deployment scenarios: it is straight-forward to fine-tune, efficient to serve, and well suited as a backbone for retrieval-augmented generation and other downstream Portuguese NLP systems.

Portuguese Modernbert Multilingual Language Models Encoders Context Length

Open paper Ask about it PDF

Abstract

Explore similar work

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark

Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders