Organizations: Instituto de Telecomunicações, Lisbon, Portugal · National Technical University of Athens, Greece · University of Amsterdam, Netherlands
Gender bias remains a persistent concern in machine translation (MT), affecting both generated translations and their automatic evaluation. When a source text leaves a person's gender unspecified, translations may realize that person using masculine or feminine forms, and both MT systems and evaluation metrics may exhibit systematic preferences between these alternatives despite the source providing no basis for such a distinction. We study this behavior in the WMT 2026 Automated Translation Quality Evaluation Systems Shared Task using an occupation-balanced subset of GAMBIT+. We consider seven English-source language pairs, six from the original dataset, targeting Arabic, Czech, Greek, Icelandic, Russian, and Ukrainian, and extend the original resource with German. The subset contains 1,308 masculine/feminine translation pairs per target language, with three examples for each of the 436 ISCO-08 occupational groups. We evaluate shared-task submissions and baselines for score prediction and error annotation, examining the direction, magnitude, and frequency of gender-related differences. We find an overall tendency for masculine translations to receive higher scores, as well as differences per occupation following stereotypical gender representations, although the strength and consistency of this preference vary considerably across evaluators and languages. Our results show that gender bias remains present in MT evaluation, but that capturing its extent requires looking beyond a single aggregate measure to complementary dimensions of evaluator behavior.
Figures & tables
ISCO
1111: Legislators
Source
The Member of Parliament delivered a compelling speech about healthcare reform during yesterday’s session.
Masculine
Der Abgeordnete hielt während der gestrigen Sitzung eine überzeugende Rede über die Gesundheitsreform.
Feminine
Die Abgeordnete hielt während der gestrigen Sitzung eine überzeugende Rede über die Gesundheitsreform.
Table 1: An English source text with the paired German translations introduced in the GAMBIT+ extension. The underlined text marks the gender-dependent difference between the masculine and feminine variants; the remaining text is identical.
Submission
AR
CS
DE
EL
IS
RU
UK
Macro-6
AEGIS
+0.66/1.20
-0.03/1.79
+0.26/0.57
+0.45/0.74
+0.61/1.96
+0.67/1.47
+0.19/1.13
+0.39/1.35
† COMETKiwi22
+0.91/1.38
+0.52/0.94
+0.20/0.57
+0.68/0.82
+1.62/1.95
+1.11/1.37
+0.46/0.95
+0.80/1.19
Cohere CAT+ ensemble
+3.56/5.07
+2.68/4.63
+1.08/4.10
+2.19/3.48
+2.59/5.07
+5.13/6.65
+2.12/4.51
+2.86/5.01
FACET_869543
+2.21/3.99
+2.18/4.16
+0.38/4.06
+2.32/4.72
+1.77/4.11
+3.09/5.04
+1.37/3.81
+1.83/4.19
FACET_869546
+2.26/4.29
+2.17/4.33
+0.43/4.24
+2.39/4.81
+1.81/4.17
+3.16/5.22
+1.40/3.86
+1.87/4.35
† Gemini-3.6-Flash
+3.02/5.25
+1.57/3.59
+0.08/3.01
–
+2.19/4.06
+3.24/4.39
+1.17/3.14
+1.88/3.91
Table 2: Signed / mean absolute paired differences ( S%/A% ), in percent of each evaluator's observed score range. Positive signed values indicate higher masculine scores. Each language has 1,308 pairs. Macro-6 equally averages AR, CS, DE, IS, RU and UK; – denotes missing coverage, never zero. RB/RF retain returned reference-based/reference-free labels, and CA abbreviates Confidence Aware; inference configurations were not independently verified. Native scales appear in Appendix A . † denotes a WMT 2026 organizer baseline; unmarked systems are participant submissions.
Figure 1: Signed and mean absolute gender-related score differences on the common six-language set across 22 qualifying evaluators. The signed difference captures the direction of preference, while the mean absolute difference captures its magnitude regardless of direction. Whiskers show percentile 95% intervals from 5,000 resamples of the 436 ISCO blocks, keeping paired observations and included languages together. Values closer to zero indicate smaller gender-related differences.
ISCO
Occupation
AR
CS
DE
EL
IS
RU
UK
Mean
2222
Midwifery Professionals
-7.27
-7.43
-4.45
-9.63
-11.79
-5.69
-7.87
-7.73
3222
Midwifery Associate Professionals
-3.09
-4.82
-3.42
-5.70
-10.23
-5.66
-1.85
-4.97
5311
Child Care Workers
-4.24
-0.27
-0.36
-3.03
-5.11
-8.31
-2.57
-3.41
5241
Fashion and Other Models
+4.41
+0.81
-3.25
-4.86
-3.56
-5.34
-10.69
-3.21
5151
Cleaning and Housekeeping Supervisors in Offices, Hotels and Other Establishments
-2.78
-4.49
-2.51
-2.72
-3.79
-3.44
-0.86
-2.94
2636
Religious Professionals
+8.14
+3.28
+4.85
+8.22
+3.34
+4.44
+8.57
+5.84
Table 3: Occupation-level extremes across the seven target languages, using the fixed panel of 13 Task 2 systems available for all languages. Each language column reports the mean normalized signed difference S% across these systems, and the final column gives the macro-average across languages. Negative values indicate higher scores for feminine variants and positive values higher scores for masculine variants.
Submission
AR
CS
DE
EL
IS
RU
UK
Cohere CAT+
+8.7
+4.2
+2.2
+4.0
+3.7
+3.1
-0.5
FACET_869532
+4.3
+9.3
+1.3
+4.8
+8.6
+10.1
+20.3
FACET_869533
+4.3
+9.6
+1.2
+4.9
+8.6
+10.2
+20.6
FACET_876357
+4.3
+9.3
+1.3
+4.8
+8.6
+10.1
+20.3
FACET_876365
+4.3
+9.6
+1.2
+4.9
+8.6
+10.2
+20.6
† Gemini-3.6-Flash
+16.4
+7.0
+0.3
–
+7.3
+13.4
+4.5
Table 4: Task 1 differences in predicted error-free rate, in percentage points (masculine minus feminine). Positive values indicate that masculine variants are more often predicted as error-free, while negative values indicate a higher error-free rate for feminine variants. † denotes a baseline; unmarked systems are participant submissions.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Submission
L
Observed interval
Rk
Savail
Aavail
S6
A6
AEGIS
7
[0.000, 100.000]
100.000
+0.4030
1.2652
+0.3948
1.3522
† COMETKiwi22
7
[0.342, 0.907]
0.565
+0.0044
0.0064
+0.0045
0.0067
Cohere CAT+ ensemble
7
[18.333, 100.000]
81.667
+2.2575
3.9098
+2.3355
4.0877
FACET_869543
7
[-42.000, 0.000]
42.000
+0.7989
1.7933
+0.7698
1.7619
FACET_869546
7
[-42.000, 0.000]
42.000
+0.8178
1.8556
+0.7867
1.8280
† Gemini-3.6-Flash
6
[0.000, 100.000]
100.000
+1.8793
3.9061
+1.8793
3.9061
Appendix
Table 5: Observed score scales and macro means in native units. L is the number of returned target languages. Absolute means are means of pairwise absolute differences, not absolute values of signed means. † denotes a baseline; unmarked systems are participant submissions.
Large language models (LLMs) and machine translation (MT) systems are increasingly used in our daily lives, but their outputs can reproduce gender bias present in the training data. Most resources for evaluating such biases are designed for English and reflect its sociocultural context, which limits their applicability to other languages. This work addresses this gap by introducing two new datasets to evaluate gender bias in translations involving Basque, a low-resource and genderless language. WinoMTeus adapts the WinoMT benchmark to examine how gender-neutral Basque occupations are translated into gendered languages such as Spanish and French. FLORES+Gender, in turn, extends the FLORES+ benchmark to assess whether translation quality varies when translating from gendered languages (Spanish and English) into Basque depending on the gender of the referent. We evaluate several general-purpose LLMs and open and proprietary MT systems. The results reveal a systematic preference for masculine forms and, in some models, a slightly higher quality for masculine referents. Overall, these findings show that gender bias is still deeply rooted in these models, and highlight the need to develop evaluation methods that consider both linguistic features and cultural context.
While Large Language Models achieve state-of-the-art results across a wide range of NLP tasks, they remain prone to systematic biases. Among these, gender bias is particularly salient in MT, due to systematic differences across languages in whether and how gender is marked. As a result, translation often requires disambiguating implicit source signals into explicit gender-marked forms. In this context, standard benchmarks may capture broad disparities but fail to reflect the full complexity of gender bias in modern MT. In this paper, we extend recent frameworks on bias evaluation by: (i) introducing a novel measure coined "Prior Bias", capturing a model's default gender assumptions, and (ii) applying the framework to decoder-only MT models. Our results show that, despite their scale and state-of-the-art status, decoder-only models do not generally outperform encoder-decoder architectures on gender-specific metrics; however, post-training (e.g., instruction tuning) not only improves contextual awareness but also reduces the masculine Prior Bias.
Chiara Manna, Hosein Mohebbi, Afra Alishahi +2
Research Center for Cognitive Science and Artificial Intelligence Tilburg School of Humanities and Digital Sciences Tilburg University, Netherlands
Handling gender across languages remains a persistent challenge for Machine Translation (MT) and Large Language Models (LLMs), especially when translating from gender-neutral languages into morphologically gendered ones, such as English to Italian. English largely omits grammatical gender, while Italian requires explicit agreement across multiple grammatical categories. This asymmetry often leads MT systems to default to masculine forms, reinforcing bias and reducing translation accuracy. To address this issue, we present the Contextual Gender Annotation (ConGA) framework, a linguistically grounded set of guidelines for word-level gender annotation. The scheme distinguishes between semantic gender in English through three tags, Masculine (M), Feminine (F), and Ambiguous (A), and grammatical gender realisation in Italian (Masculine (M), Feminine (F)), combined with entity-level identifiers for cross-sentence tracking. We apply ConGA to the gENder-IT dataset, creating a gold-standard resource for evaluating gender bias in translation. Our results reveal systematic masculine overuse and inconsistent feminine realisation, highlighting persistent limitations of current MT systems. By combining fine-grained linguistic annotation with quantitative evaluation, this work offers both a methodology and a benchmark for building more gender-aware and multilingual NLP systems.
Argentina Anna Rescigno, Eva Vanmassenhove, Johanna Monti
University of Pisa · University of Naples “L’Orientale” · Tilburg University