GRAP: Gender Rubric-Aware Probing for Analyzing Gender Bias in LLMs
Organizations: Ewha Womans University, Seoul, Republic of Korea
Abstract
As LLMs increasingly mediate everyday interactions, subtle gender biases in open-ended responses pose growing societal risks. Existing probing methods have two key limitations: they rely on fixed-format tasks, such as pronoun resolution or occupation association, that alignment-tuned models can easily handle; and they compare raw group-level rubric scores, ignoring per-dimension variance and obscuring which bias dimensions reliably distinguish gender groups. We introduce GRAP (Gender Rubric-Aware Probing), a framework that uses a lightweight logistic probe to identify the most discriminative bias dimensions from responses scored across 10 rubric dimensions, operating task-agnostically on open-ended instruction following.
Figures & tables
Gender Bias in Language Models
Gender bias in language models examines how systems encode and express gender stereotypes and how to evaluate and reduce their effects.
See all 75 papers