cs.CLNov 2, 2026

From Language to Country to Valence: A Multilingual Audit of Stereotype Bias Across LLMs

Authors: Zihao Sun

Organizations: University of Technology Sydney Sydney, Australia

Abstract

Large language models (LLMs) are increasingly considered for education, recruitment, workplace decision support, and other consequential applications, making pre-deployment social-bias detection essential. Yet major benchmarks are English-centred or translated, focus mainly on negative stereotypes, and may miss linguistic and cultural variation. We investigate gender, racial, and religious bias using 12,000 locally written Reddit posts in English, Spanish, French, and German. Only search terms are translated, preserving local context. The dataset is balanced by language and category; English gender is additionally balanced by stereotype valence. Three open-weight and three proprietary API-based models are evaluated with a two-blank design that reduces grammatical-gender cues. Religion has the highest absolute choice imbalance in every model-language condition. Spanish is highest for gender, whereas German is highest for race and religion. The evaluated API models score below the open-weight models throughout. Negative English-gender items exceed positive items; Australian and Mexican contexts exceed their Canadian and Spanish counterparts. These descriptive differences measure choice concentration, not stereotype endorsement or causal country effects. The crossed patterns show how aggregate scores can conceal category- and context-specific behaviour, motivating closer inspection before deployment.

Figures & tables

Explore similar work

CardsList
  1. Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit

    May 29, 2026Jiwoo Choi, Seonwoo Ahn, Tongxin Zhang +1Gender Bias in Language ModelsMultilingual Language Model Evaluation

  2. MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

    Jun 11, 2024Vera Neplenbroek, Arianna Bisazza, Raquel FernándezSocial Bias in Language ModelsLanguage Model Bias Evaluation