cs.CLJun 15, 2026

P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

Authors: Rafael FerreiraInês VieiraInês CalvoJames FurtadoIago PauloDiogo TavaresDiogo Glória-SilvaDavid Semedo+1 more

Organizations: NOVA University of Lisbon, Portugal · NOVA LINCS

Abstract

As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.

Explore similar work

CardsList
  1. Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

    May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6Conversational BenchmarksGrading

  2. CARTE: A Benchmark for Mapping Language Model Knowledge Across France

    Jun 1, 2026Sarah Almeida Carneiro, Christos Xypolopoulos, Xiao Fei +2LuxembourgishCultural Knowledge

  3. Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese

    Jun 5, 2026Giordano de Pinho Souza, Glaucia Melo, Josefino Cabral Melo Lima +1PortugueseMultilingual Benchmark