Handling gender across languages remains a persistent challenge for Machine Translation (MT) and Large Language Models (LLMs), especially when translating from gender-neutral languages into morphologically gendered ones, such as English to Italian. English largely omits grammatical gender, while Italian requires explicit agreement across multiple grammatical categories. This asymmetry often leads MT systems to default to masculine forms, reinforcing bias and reducing translation accuracy. To address this issue, we present the Contextual Gender Annotation (ConGA) framework, a linguistically grounded set of guidelines for word-level gender annotation. The scheme distinguishes between semantic gender in English through three tags, Masculine (M), Feminine (F), and Ambiguous (A), and grammatical gender realisation in Italian (Masculine (M), Feminine (F)), combined with entity-level identifiers for cross-sentence tracking. We apply ConGA to the gENder-IT dataset, creating a gold-standard resource for evaluating gender bias in translation. Our results reveal systematic masculine overuse and inconsistent feminine realisation, highlighting persistent limitations of current MT systems. By combining fine-grained linguistic annotation with quantitative evaluation, this work offers both a methodology and a benchmark for building more gender-aware and multilingual NLP systems.
Figures & tables
EN source
Women have been trained to think that we <F1> are overreacting or that we <F1> are being too sensitive or unreasonable.
IT target
Alle donne è stato insegnato a pensare che (noi) siamo troppo reattive <F1> o che (noi) siamo troppo sensibili o irragionevoli.
Table 1: Example of tagging between English and Italian sentences. The tag <F1> highlights corresponding focus elements across languages.
EN source
No, you <A1> ’re still dry, you <A1> ’re just being nice.
IT target
No, (tu) sei ancora asciutta <F1> , (tu) stai solo facendo il bravo <M1> .
Table 2: Example of tagging between English and Italian sentences. Tags such as <A1> , <F1> , and <M1> indicate corresponding referents or gender features across languages.
Gender Tags
EN
IT TowerLLM
IT mBART
M
356
544
570
F
294
210
181
A
909
0
0
Total Tags
1559
754
751
Table 3: Tag distribution by gender in English (EN) and Italian outputs from TowerLLM and mBART .
✓ Match - Sentence 110 (F → F)
EN source
Women have been trained to think that we <F1> are overreacting or that we’re <F1> being too sensitive or unreasonable.
IT TowerLLM
Alle donne è stato insegnato a pensare che siamo troppo reattive <F1> o che siamo troppo sensibili o irragionevoli.
IT mBART
Le donne sono state addestrate a pensare che siamo troppo reattive <F1> o che siamo troppo sensibili o irragionevoli.
✗ Mismatch Error - Sentence 164 (F → M)
EN source
Many of the women working with me <A1> had to leave once they <F2> got married, because their husbands wouldn’t let them <F2> work.
IT TowerLLM
Molte delle donne che lavoravano con me hanno dovuto lasciare il lavoro non appena si sono sposate <F2> perché i mariti non li <M2> volevano vedere lavorare.
Table 4: Examples of gender agreement across translation systems ( Tower , mBART ) for each evaluation category.
TowerLLM
mBART
✓ Match categories
Match M
173
165
Match F
104
88
Total Matches
277
253
✗ Mismatch categories
Bias A → M
215
221
Table 5: Detailed comparison of matching and mismatch categories for TowerLLM vs. mBART .
Model
Match
Total Matches
Total Tags (EN)
Precision
Recall
F1-Score
Male
TowerLLM
173
544
570
31.8%
48.6%
38.3%
mBART
165
570
356
28.9%
46.3%
35.5%
Female
TowerLLM
104
210
294
49.5%
35.4%
41.3%
mBART
88
181
294
48.6%
29.9%
36.9%
Table 6: Performance Metrics of Precision, Recall, and F1-score for TowerLLM and mBART by Gender. Alternating columns are shaded for readability .