Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

Amaia Murillo; Olatz-Perez-de-Viñaspre; Naiara Perez

cs.CLMar 9, 2026

Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

Authors: Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara Perez

Organizations: HiTZ Center - Ixa, University of the Basque Country UPV/EHU

Abstract

Large language models (LLMs) and machine translation (MT) systems are increasingly used in our daily lives, but their outputs can reproduce gender bias present in the training data. Most resources for evaluating such biases are designed for English and reflect its sociocultural context, which limits their applicability to other languages. This work addresses this gap by introducing two new datasets to evaluate gender bias in translations involving Basque, a low-resource and genderless language. WinoMTeus adapts the WinoMT benchmark to examine how gender-neutral Basque occupations are translated into gendered languages such as Spanish and French. FLORES+Gender, in turn, extends the FLORES+ benchmark to assess whether translation quality varies when translating from gendered languages (Spanish and English) into Basque depending on the gender of the referent. We evaluate several general-purpose LLMs and open and proprietary MT systems. The results reveal a systematic preference for masculine forms and, in some models, a slightly higher quality for masculine referents. Overall, these findings show that gender bias is still deeply rooted in these models, and highlight the need to develop evaluation methods that consider both linguistic features and cultural context.

Figures & tables

Figure 1: Overview of WinoMT eus . Sentences with gender-neutral ( N ) occupation mentions, taken from WinoMT ( Stanovsky et al., 2019 ) and translated to Basque, are translated into a gendered language; we compare the resulting gendered occupation distribution (male M or female F ) with real-world data.

Figure 2: Overview of FLORES +Gender. Using Basque-gendered language pairs from FLORES+, 3 3 3 https://huggingface.co/datasets/openlanguagedata/flores_plus we create a contrastive version of the gendered sentence (male M or female F ). Both are translated into Basque, and translation quality is compared to asses the effect of source-sentence gender.

	ME	PN	UM
Spanish	24.52	19.65	60.01
English	29.03	49.68	n/a

Table 1: Quantification of FLORES +Gender annotated phenomena (% over total sentences).

	EU-ES		EU-FR
Llama 3.1 8B Instruct	0.066		0.158
Latxa 3.1 8B Instruct	0.270	**	0.311
Llama 3.1 70B Instruct	0.215		0.254	**
Latxa 3.1 70B Instruct	0.409		0.352
DeepSeek-V3.2-Exp	0.259	**	0.275	**
Claude 4 Sonnet	0.191		0.166

Table 2: Pearson correlation coefficients (

r

) between Lanbide’s labour statistics and model translations from WinoMT eus to Spanish and French. The highest correlation per language pair is highlighted in bold. ***

p<0.01

, **

p<0.05

, *

p<0.1

Occupation	Real female proportion (%)	GRAPE-M
Occupation	Real female proportion (%)	ES	FR
housekeeper	96.5	23.4	24.6
tailor	92.7	12.4	12.5
receptionist	89.3	7.7	7.9
cashier	86.8	6.9	6.6
hairdresser	86.2	5.9	5.9

Table 3: Top five most feminized occupations in the real-world data showing the highest masculine bias scores (GRAPE-M) across all translation models.

	ES-EU		EN-EU
	chrF++	TER	chrF++	TER
Llama 3.1 8B Instruct	0.22	-21.07	0.00	-60.71
Latxa 3.1 8B Instruct	0.14	-0.50	-0.11	0.60
Llama 3.1 70B Instruct	0.20	3.01	-0.42	-0.44
Latxa 3.1 70B Instruct	0.14	0.11	-0.22	-0.44
DeepSeek-V3.2-Exp	-0.24	-0.29	-0.59	1.22

Table 4: Average difference (

\Delta

) between masculine and feminine contrastive sets from FLORES +Gender. Positive values ( green ) indicate higher performance on the masculine set, and negative values ( orange ) indicate higher performance on the feminine set. Statistically significant results are marked in boldface. *

p<0.05

	ES-EU						EN-EU
	ME		PN		UM		ME		PN
	chrF++	TER	chrF++	TER	chrF++	TER	chrF++	TER	chrF++	TER
Llama 3.1 8B Instruct	0.44	-130.60	-0.28	-33.64	1.00	34.65	1.02	-65.17	2.06	0.61
Latxa 3.1 8B Instruct	-0.74	0.17	-0.33	-1.74	0.07	1.29	0.65	-0.75	0.43	0.75
Llama 3.1 70B Instruct	0.08	-28.39	-0.26	-31.92	0.07	-6.83	1.57 ∗	2.52	0.92	2.84
Latxa 3.1 70B Instruct	-0.28	0.31	0.08	0.43	-0.12	-0.24	1.79	2.22	1.87 ∗	2.73

Table 5: Interaction effect (

\Delta

) between gender and linguistic factors (ME = Multi-entity, PN = Proper names, UM = Unmarked masculine) across both Spanish-Basque (ES-EU) and English-Basque (EN-EU) translation directions. Green indicates higher performance on masculine sentences, orange indicates higher performance on feminine ones. Statistically significant results are marked in boldface. **

p<0.05

, *

p<0.1

Explore similar work

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations
Sep 18, 2026Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed +2Machine Translation MT Evaluation
Instituto de Telecomunicações, Lisbon, Portugal · National Technical University of Athens, Greece · University of Amsterdam, Netherlands
Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures
Mar 18, 2026Chiara Manna, Hosein Mohebbi, Afra Alishahi +2Gender Bias in Language Models Language Model Bias Evaluation
Research Center for Cognitive Science and Artificial Intelligence Tilburg School of Humanities and Digital Sciences Tilburg University, Netherlands
ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation
Mar 18, 2026Argentina Anna Rescigno, Eva Vanmassenhove, Johanna MontiMachine Translation MT Evaluation
University of Pisa · University of Naples “L’Orientale” · Tilburg University

Sep 18, 2026J/K move · Enter open · S save

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio+1

Instituto de Telecomunicações, Lisbon, Portugal · National Technical University of Athens, Greece · University of Amsterdam, Netherlands

Machine Translation MT Evaluation

Open paper AskPDF arXiv

Gender bias remains a persistent concern in machine translation (MT), affecting both generated translations and their automatic evaluation. When a source text leaves a person's gender unspecified, translations may realize that person using masculine or feminine forms, and both MT systems and evaluation metrics may exhibit systematic preferences between these alternatives despite the source providing no basis for such a distinction. We study this behavior in the WMT 2026 Automated Translation Quality Evaluation Systems Shared Task using an occupation-balanced subset of GAMBIT+. We consider seven English-source language pairs, six from the original dataset, targeting Arabic, Czech, Greek, Icelandic, Russian, and Ukrainian, and extend the original resource with German. The subset contains 1,308 masculine/feminine translation pairs per target language, with three examples for each of the 436 ISCO-08 occupational groups. We evaluate shared-task submissions and baselines for score prediction and error annotation, examining the direction, magnitude, and frequency of gender-related differences. We find an overall tendency for masculine translations to receive higher scores, as well as differences per occupation following stereotypical gender representations, although the strength and consistency of this preference vary considerably across evaluators and languages. Our results show that gender bias remains present in MT evaluation, but that capturing its extent requires looking beyond a single aggregate measure to complementary dimensions of evaluator behavior.

Abstract

Explore similar work

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation