cs.CLNov 2, 2026

LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

Authors: Danial Amin, Joni Salminen, Bernard J. Jansen

Organizations: University of Vaasa, Vaasa, Finland · Otherkind Tech, Helsinki, Finland · Peking University, Beijing, China · Nankai University, Tianjin, China

Abstract

Large language models (LLMs) are increasingly used to evaluate candidates for professional roles, raising concerns about their sensitivity to demographic identities. We examine how candidate and evaluator gender assignments affect decisions by LLM-as-a-Judge (LaJ) models and decision language models (DLMs). In a hypothetical city council voting task, eight models evaluated 240 synthetic professional narratives from NVIDIA's US and French Nemotron-Personas datasets under independently varied gender assignments, producing 30,720 judgments. Explicit candidate gender labels increased Yes-vote probability by 1.05-1.73 percentage points relative to unspecified gender, while only the nonbinary evaluator assignment showed a significant increase (1.26 points). Identity matching showed no significant overall association. Yes-vote rates varied across models (81.3-97.2%), and 2.5-3.3% of decisions reversed relative to the unspecified-identity baseline. Under direct substitutions between explicit gender identities, DLMs exhibited fewer decision reversals than LaJ models (1.18% vs. 3.38%), indicating lower observed decision-level gender sensitivity among the tested DLMs. These findings demonstrate that gender sensitivity depends on candidate and evaluator identities and the evaluation model, while highlighting the importance of examining individual decision consistency alongside aggregate differences.

Figures & tables

Explore similar work

CardsList
  1. Gender bias across LLMs is common and highly heterogeneous

    Sep 29, 2026Edoardo Bolzoni, Valerio CapraroGender Bias in Language ModelsLLM Auditing

  2. The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

    Sep 8, 2026Siddharth Vohra, Manikandan RavikiranLanguage Model Bias EvaluationDemographic Bias in Language Models