Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

Orfeas Menis Mastromichalakis; Giorgos Filandrianos; Wafaa Mohammed; Giuseppe Attanasio; Chrysoula Zerva

cs.CLSep 18, 2026

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

Authors: Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva

Organizations: Instituto de Telecomunicações, Lisbon, Portugal · National Technical University of Athens, Greece · University of Amsterdam, Netherlands

Abstract

Gender bias remains a persistent concern in machine translation (MT), affecting both generated translations and their automatic evaluation. When a source text leaves a person's gender unspecified, translations may realize that person using masculine or feminine forms, and both MT systems and evaluation metrics may exhibit systematic preferences between these alternatives despite the source providing no basis for such a distinction. We study this behavior in the WMT 2026 Automated Translation Quality Evaluation Systems Shared Task using an occupation-balanced subset of GAMBIT+. We consider seven English-source language pairs, six from the original dataset, targeting Arabic, Czech, Greek, Icelandic, Russian, and Ukrainian, and extend the original resource with German. The subset contains 1,308 masculine/feminine translation pairs per target language, with three examples for each of the 436 ISCO-08 occupational groups. We evaluate shared-task submissions and baselines for score prediction and error annotation, examining the direction, magnitude, and frequency of gender-related differences. We find an overall tendency for masculine translations to receive higher scores, as well as differences per occupation following stereotypical gender representations, although the strength and consistency of this preference vary considerably across evaluators and languages. Our results show that gender bias remains present in MT evaluation, but that capturing its extent requires looking beyond a single aggregate measure to complementary dimensions of evaluator behavior.

Figures & tables

ISCO	1111: Legislators
Source	The Member of Parliament delivered a compelling speech about healthcare reform during yesterday’s session.
Masculine	Der Abgeordnete hielt während der gestrigen Sitzung eine überzeugende Rede über die Gesundheitsreform.
Feminine	Die Abgeordnete hielt während der gestrigen Sitzung eine überzeugende Rede über die Gesundheitsreform.

Table 1: An English source text with the paired German translations introduced in the GAMBIT+ extension. The underlined text marks the gender-dependent difference between the masculine and feminine variants; the remaining text is identical.

Submission	AR	CS	DE	EL	IS	RU	UK	Macro-6
AEGIS	+0.66/1.20	-0.03/1.79	+0.26/0.57	+0.45/0.74	+0.61/1.96	+0.67/1.47	+0.19/1.13	+0.39/1.35
† COMETKiwi22	+0.91/1.38	+0.52/0.94	+0.20/0.57	+0.68/0.82	+1.62/1.95	+1.11/1.37	+0.46/0.95	+0.80/1.19
Cohere CAT+ ensemble	+3.56/5.07	+2.68/4.63	+1.08/4.10	+2.19/3.48	+2.59/5.07	+5.13/6.65	+2.12/4.51	+2.86/5.01
FACET_869543	+2.21/3.99	+2.18/4.16	+0.38/4.06	+2.32/4.72	+1.77/4.11	+3.09/5.04	+1.37/3.81	+1.83/4.19
FACET_869546	+2.26/4.29	+2.17/4.33	+0.43/4.24	+2.39/4.81	+1.81/4.17	+3.16/5.22	+1.40/3.86	+1.87/4.35
† Gemini-3.6-Flash	+3.02/5.25	+1.57/3.59	+0.08/3.01	–	+2.19/4.06	+3.24/4.39	+1.17/3.14	+1.88/3.91

Table 2: Signed / mean absolute paired differences (

S^{\%}/A^{\%}

), in percent of each evaluator's observed score range. Positive signed values indicate higher masculine scores. Each language has 1,308 pairs. Macro-6 equally averages AR, CS, DE, IS, RU and UK; – denotes missing coverage, never zero. RB/RF retain returned reference-based/reference-free labels, and CA abbreviates Confidence Aware; inference configurations were not independently verified. Native scales appear in Appendix A . † denotes a WMT 2026 organizer baseline; unmarked systems are participant submissions.

Panel of Figure 1 — Figure 1: Signed and mean absolute gender-related score differences on the common six-language set across 22 qualifying evaluators. The signed difference captures the direction of preference, while the mean absolute difference captures its magnitude regardless of direction. Whiskers show percentile 95% intervals from 5,000 resamples of the 436 ISCO blocks, keeping paired observations and included languages together. Values closer to zero indicate smaller gender-related differences.

ISCO	Occupation	AR	CS	DE	EL	IS	RU	UK	Mean
2222	Midwifery Professionals	-7.27	-7.43	-4.45	-9.63	-11.79	-5.69	-7.87	-7.73
3222	Midwifery Associate Professionals	-3.09	-4.82	-3.42	-5.70	-10.23	-5.66	-1.85	-4.97
5311	Child Care Workers	-4.24	-0.27	-0.36	-3.03	-5.11	-8.31	-2.57	-3.41
5241	Fashion and Other Models	+4.41	+0.81	-3.25	-4.86	-3.56	-5.34	-10.69	-3.21
5151	Cleaning and Housekeeping Supervisors in Offices, Hotels and Other Establishments	-2.78	-4.49	-2.51	-2.72	-3.79	-3.44	-0.86	-2.94
2636	Religious Professionals	+8.14	+3.28	+4.85	+8.22	+3.34	+4.44	+8.57	+5.84

Table 3: Occupation-level extremes across the seven target languages, using the fixed panel of 13 Task 2 systems available for all languages. Each language column reports the mean normalized signed difference

S^{\%}

across these systems, and the final column gives the macro-average across languages. Negative values indicate higher scores for feminine variants and positive values higher scores for masculine variants.

Submission	AR	CS	DE	EL	IS	RU	UK
Cohere CAT+	+8.7	+4.2	+2.2	+4.0	+3.7	+3.1	-0.5
FACET_869532	+4.3	+9.3	+1.3	+4.8	+8.6	+10.1	+20.3
FACET_869533	+4.3	+9.6	+1.2	+4.9	+8.6	+10.2	+20.6
FACET_876357	+4.3	+9.3	+1.3	+4.8	+8.6	+10.1	+20.3
FACET_876365	+4.3	+9.6	+1.2	+4.9	+8.6	+10.2	+20.6
† Gemini-3.6-Flash	+16.4	+7.0	+0.3	–	+7.3	+13.4	+4.5

Table 4: Task 1 differences in predicted error-free rate, in percentage points (masculine minus feminine). Positive values indicate that masculine variants are more often predicted as error-free, while negative values indicate a higher error-free rate for feminine variants. † denotes a baseline; unmarked systems are participant submissions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Submission	$L$	Observed interval	$R_{k}$	$S_{\rm avail}$	$A_{\rm avail}$	$S_{6}$	$A_{6}$
AEGIS	7	[0.000, 100.000]	100.000	+0.4030	1.2652	+0.3948	1.3522
† COMETKiwi22	7	[0.342, 0.907]	0.565	+0.0044	0.0064	+0.0045	0.0067
Cohere CAT+ ensemble	7	[18.333, 100.000]	81.667	+2.2575	3.9098	+2.3355	4.0877
FACET_869543	7	[-42.000, 0.000]	42.000	+0.7989	1.7933	+0.7698	1.7619
FACET_869546	7	[-42.000, 0.000]	42.000	+0.8178	1.8556	+0.7867	1.8280
† Gemini-3.6-Flash	6	[0.000, 100.000]	100.000	+1.8793	3.9061	+1.8793	3.9061

Appendix

Table 5: Observed score scales and macro means in native units.

L

is the number of returned target languages. Absolute means are means of pairwise absolute differences, not absolute values of signed means. † denotes a baseline; unmarked systems are participant submissions.

Explore similar work

Gender Bias in MT for a Genderless Language: New Benchmarks for Basque
Mar 9, 2026Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara PerezGender Bias in Language Models Machine Translation
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures
Mar 18, 2026Chiara Manna, Hosein Mohebbi, Afra Alishahi +2Gender Bias in Language Models Language Model Bias Evaluation
Research Center for Cognitive Science and Artificial Intelligence Tilburg School of Humanities and Digital Sciences Tilburg University, Netherlands
ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation
Mar 18, 2026Argentina Anna Rescigno, Eva Vanmassenhove, Johanna MontiMachine Translation MT Evaluation
University of Pisa · University of Naples “L’Orientale” · Tilburg University

Mar 9, 2026J/K move · Enter open · S save

Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara Perez

HiTZ Center - Ixa, University of the Basque Country UPV/EHU

Gender Bias in Language Models Machine Translation MT Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) and machine translation (MT) systems are increasingly used in our daily lives, but their outputs can reproduce gender bias present in the training data. Most resources for evaluating such biases are designed for English and reflect its sociocultural context, which limits their applicability to other languages. This work addresses this gap by introducing two new datasets to evaluate gender bias in translations involving Basque, a low-resource and genderless language. WinoMTeus adapts the WinoMT benchmark to examine how gender-neutral Basque occupations are translated into gendered languages such as Spanish and French. FLORES+Gender, in turn, extends the FLORES+ benchmark to assess whether translation quality varies when translating from gendered languages (Spanish and English) into Basque depending on the gender of the referent. We evaluate several general-purpose LLMs and open and proprietary MT systems. The results reveal a systematic preference for masculine forms and, in some models, a slightly higher quality for masculine referents. Overall, these findings show that gender bias is still deeply rooted in these models, and highlight the need to develop evaluation methods that consider both linguistic features and cultural context.

Abstract

Explore similar work

Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation