ProText: A benchmark dataset for measuring (mis)gendering in long-form texts

Hadas Kotek; Margit Bowler; Patrick Sonnenberg; Yu'an Yang

cs.CLMar 29, 2026

ProText: A benchmark dataset for measuring (mis)gendering in long-form texts

Authors: Hadas Kotek, Margit Bowler, Patrick Sonnenberg, Yu'an Yang

Organizations: Apple, USA

Abstract

We introduce ProText, a dataset for measuring gendering and misgendering in stylistically diverse long-form English texts. ProText spans three dimensions: Theme nouns (names, occupations, titles, kinship terms), Theme category (stereotypically male, stereotypically female, gender-neutral/non-gendered), and Pronoun category (masculine, feminine, gender-neutral, none). The dataset is designed to probe (mis)gendering in text transformations such as summarization and rewrites using state-of-the-art Large Language Models, extending beyond traditional pronoun resolution benchmarks and beyond the gender binary. We validated ProText through a mini case study, showing that even with just two prompts and two models, we can draw nuanced insights regarding gender bias, stereotyping, misgendering, and gendering. We reveal systematic gender bias, particularly when inputs contain no explicit gender cues or when models default to heteronormative assumptions.

Gender Bias in Language Models

Gender bias in language models examines how systems encode and express gender stereotypes and how to evaluate and reduce their effects.

See all 64 papers

GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German
May 28, 2026Fabian Mewes, Anne Lauscher, Vagrant GautamGender Bias in Language Models Linguistic Bias in Language Models
JobMatchMe GmbH, Germany · Trustworthy AI Lab, University of Hamburg, Germany · Heidelberg Institute for Theoretical Studies, Germany
MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation
Apr 20, 2026Mehul Agarwal, Aditya Aggarwal, Arnav Goel +2Multilingual Language Model Evaluation Text Generation
SBILab, Indraprastha Institute of Information Technology Delhi
Ben-Misandry: A Bengali Dataset and Late-Fusion Transformer Framework for Detecting Anti-Male Gender Hate
Oct 17, 2026Md Mehedi Hasan Nipu, Chinmoy Mitra, Tarannum Ahmed Nowshin +1Hate Speech Detection Bengali NLP
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh · Department of Computer Science & Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh +1

May 28, 2026J/K move · Enter open · S save

GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German

Fabian Mewes, Anne Lauscher, Vagrant Gautam

JobMatchMe GmbH, Germany · Trustworthy AI Lab, University of Hamburg, Germany · Heidelberg Institute for Theoretical Studies, Germany

Gender Bias in Language Models Linguistic Bias in Language Models Language Model Robustness LLM Reasoning

Open paper AskPDF arXiv

Third-person singular pronouns have long been used to study stereotypical biases in language models and to test their abilities to reason about reference. More recently, the interplay between reasoning and bias has been investigated with the task of pronoun fidelity, which assesses models' abilities to correctly reuse a previously-specified pronoun for a discourse entity, independent of other potentially distracting discourse entities mentioned in between. However, such research focuses on English, which is a language with limited grammatical gender and almost no gender agreement. In this paper we contribute a novel, large-scale dataset, GRUFF, to measure pronoun fidelity in German, covering four different gender agreement systems in nouns, and four sets of pronouns. With this dataset, we show that LLMs show strong grammatical agreement for masculine and feminine entities in the absence of explicit context, but not for neopronouns xier and en. Models are generally not robust to distractors, but encoder-only models are more robust in German than in English, reflecting the importance of grammatical gender. Finally, we show that occupational stereotypes in this context are poorly correlated across grammatical cases, and across most models, except ones with closely related architectures. We release all code and data to encourage further work on gender-inclusive language and referential reasoning in German.

Abstract

Gender Bias in Language Models

Explore similar work

GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German

MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

Ben-Misandry: A Bengali Dataset and Late-Fusion Transformer Framework for Detecting Anti-Male Gender Hate