cs.CLMar 18, 2026

Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

Authors: Chiara Manna, Hosein Mohebbi, Afra Alishahi, Frédéric Blain, Eva Vanmassenhove

Organizations: Research Center for Cognitive Science and Artificial Intelligence Tilburg School of Humanities and Digital Sciences Tilburg University, Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While Large Language Models achieve state-of-the-art results across a wide range of NLP tasks, they remain prone to systematic biases. Among these, gender bias is particularly salient in MT, due to systematic differences across languages in whether and how gender is marked. As a result, translation often requires disambiguating implicit source signals into explicit gender-marked forms. In this context, standard benchmarks may capture broad disparities but fail to reflect the full complexity of gender bias in modern MT. In this paper, we extend recent frameworks on bias evaluation by: (i) introducing a novel measure coined "Prior Bias", capturing a model's default gender assumptions, and (ii) applying the framework to decoder-only MT models. Our results show that, despite their scale and state-of-the-art status, decoder-only models do not generally outperform encoder-decoder architectures on gender-specific metrics; however, post-training (e.g., instruction tuning) not only improves contextual awareness but also reduces the masculine Prior Bias.

Figures & tables

Figure 1: EN →\rightarrow IT WinoMT examples showing outputs 2 2 2 Generated by ChatGPT on September 29th, 2025. defaulting to la governante ( fem. ) and il contadino ( masc. ), despite the gender cue ( he / she ).
Figure 2: Prompt templates for the EN →\rightarrow IT task. Plain for Llama2 and TowerBase, chat-style for TowerInstruct.
Figure 3: Example illustrating the neutralization process followed to construct the Neutral Set.
SetModelOverallMasc.Fem.
Decoder–Only Models
REGLlama248.9%80.8%23.3%
 TowerBase47.4%78.4%22.6%
 TowerInstruct58.6%72.9%52.0%
PRO-SLlama264.8%90.2%39.5%
 TowerBase65.2%88.5%41.9%
Table 1: Overall, masculine, and feminine (standard) accuracies on WinoMT for our decoder-only models and previously reported encoder-decoder systems ( Manna et al., 2025 ) , on the regular (REG), pro-stereotypical (PRO-S), and anti-stereotypical (ANTI-S) sets.
Figure 4: Minimal pair illustration. The pro-stereotypical case (top) assigns she to librarian , while the anti-stereotypical case (bottom) assigns he . Accurate EN →\rightarrow IT translations must adapt grammatical gender: la bibliotecaria ( f. ) vs. il bibliotecario ( m. ).
ModelMPA Pro-FPro-M
Decoder–Only Models
Llama213.1% 73.4%26.6%
TowerBase12.3% 77.4%22.6%
TowerInstruct36.7% 50.4%49.6%
Encoder–Decoder Baselines ( Manna et al. 2025 )
OPUS-MT6.1% 82.3%17.7%
Table 2: Minimal Pair Accuracy (MPA) and its breakdown by gender and stereotype for decoder-only models and encoder-decoder baselines (results reported in Manna et al. (2025) ). Pro-F and Pro-M indicate the proportion of accurate minimal pairs where the profession is stereotypically associated with women or men, respectively.
Standard Accuracy
ModelOverallMasculineFeminine
Llama248.9%80.8%23.3%
TowerBase47.4%78.4%22.6%
TowerInstruct58.6%72.9%52.0%
Diagnostic Evaluation
 MPAPrior Bias
Table 3: Standard Accuracy reports traditional WinoMT metrics (Overall, Masculine, Feminine Accuracy). Diagnostic Evaluation combines (i) Minimal Pair Accuracy (MPA); and (ii) Prior Bias , capturing the proportion of masculine vs. feminine gendered forms on the neutralized set, i.e. , in the absence of explicit cues.
Figure 5: Attention from the translated profession noun to the feminine cue across layers (y-axis) and heads (x-axis). Values represent the average attention weight assigned to the cue across the minimum number of accurate minimal pairs observed across models ( n=195n=195 ). We focus on intermediate layers (8–20), where attention is strongest; earlier and later layers exhibit near-zero values. Same color scale applied; darker shades →\rightarrow higher average attention.
 Target (L15)Secondary (L15)
ModelH10H25H10H25
Llama20.110.120.010.02
TowerBase0.170.160.020.01
TowerInstruct0.190.160.040.03
Table 4: Attention to the feminine cue in Layer 15, Heads 10 and 25 when generating the translation of our target v.s. our secondary (non-coreferent) entity.
ModelUnknown (%)
Llama29.4%
TowerBase10.3%
TowerInstruct12.8%
Table 5: Percentage of Unknown gender assignments for each model on the regular WinoMT set, i.e. , instances where the automatic pipeline failed to detect a gender.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Attention from the translated secondary (non-coreferent) entity to the feminine cue across layers (y-axis) and heads (x-axis). Values represent the average attention weight assigned to the cue across accurate minimal pairs ( n=195n=195 ). Same color scale applied; darker shades →\rightarrow higher average attention.

Explore similar work

CardsList

  1. Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

    Sep 18, 2026Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed +2Machine TranslationMT Evaluation

    Instituto de Telecomunicações, Lisbon, Portugal · National Technical University of Athens, Greece · University of Amsterdam, Netherlands

  2. Gender Bias in MT for a Genderless Language: New Benchmarks for Basque

    Mar 9, 2026Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara PerezGender Bias in Language ModelsMachine Translation

    HiTZ Center - Ixa, University of the Basque Country UPV/EHU

  3. Mitigating Gender Bias in English to Romanian Machine Translation

    Aug 9, 2026Ioana Grigore, Sergiu NisioiMachine Translation

    Human Language Technologies Research Center Faculty of Mathematics and Computer Science University of Bucharest