cs.AINov 9, 2026

GRAP: Gender Rubric-Aware Probing for Analyzing Gender Bias in LLMs

Authors: Seona Seo, Geonhee Ahn, Hyunsoo Cho, Sookyung Kim

Organizations: Ewha Womans University, Seoul, Republic of Korea

Abstract

As LLMs increasingly mediate everyday interactions, subtle gender biases in open-ended responses pose growing societal risks. Existing probing methods have two key limitations: they rely on fixed-format tasks, such as pronoun resolution or occupation association, that alignment-tuned models can easily handle; and they compare raw group-level rubric scores, ignoring per-dimension variance and obscuring which bias dimensions reliably distinguish gender groups. We introduce GRAP (Gender Rubric-Aware Probing), a framework that uses a lightweight logistic probe to identify the most discriminative bias dimensions from responses scored across 10 rubric dimensions, operating task-agnostically on open-ended instruction following.

Figures & tables

Explore similar work

CardsList
  1. Effects of Answer Format Variation on Gender Bias in Large Language Models

    Aug 18, 2026Ksenia Merzlyakova, Sebastian Padó, Franziska WeeberLanguage Model Bias EvaluationLLM Evaluation

  2. Gender bias across LLMs is common and highly heterogeneous

    Sep 29, 2026Edoardo Bolzoni, Valerio CapraroGender Bias in Language ModelsLLM Auditing

  3. Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study

    Mar 25, 2026Nour Bouchouchi, Thibault Laugel, Xavier Renard +3Gender Bias in Language ModelsLLM Alignment