Language Model Bias Evaluation

Overview

Language model bias evaluation studies whether models systematically treat people, viewpoints, or information differently, and whether those differences cause harm in settings such as hiring, healthcare, education, and automated judging. Researchers use controlled counterfactual prompts and minimal-pair or stereotype benchmarks such as BBQ, alongside open-ended, multilingual, multimodal, and multi-turn tests that better reflect real interactions. Analyses range from output comparisons and human-validated benchmarks to probes of internal representations; mitigation includes prompting, fine-tuning, and activation steering. A central challenge is that measured bias changes with wording, answer format, language, and task, while reducing one disparity can hide or worsen another, so reliable evaluation must test multiple contexts and preserve task performance.

Latest papers

CardsList

529 papers44 papers in the last four weeks, down 8% on the four weeks before. 0.3% of all new papers.

  1. Once Upon a Bias: Gender Stereotyping in LLM-Generated Children's Stories

    Nov 18, 2026Sayumi Muthukumarana, Roshan Godaliyadda, Parakrama Ekanayake +1Gender Bias in Language ModelsLanguage Model Bias Evaluation

  2. What Does a Bias Head Measure? Non-Identifiability in Head-Level Bias Attribution

    Nov 17, 2026S. M Seefat AlamAttention Head AnalysisLLM Auditing

  3. AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

    Nov 17, 2026Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin +2Cross-Cultural Language Model EvaluationCultural Knowledge in Language Models

  4. Comparing Implicit and Explicit Language Attitudes toward the Dutch multiethnolect Straattaal in Humans and Large Language Models

    Nov 17, 2026Hedwig Oldenhof, Laura Rosseel, Anna Wegmann +1Linguistic Bias in Language ModelsDialect Bias in Language Models

  5. Corrupting Ties: Investigating Robustness to Bias Injection Across Multi-Agent Network Topologies

    Nov 16, 2026Cléa Chataigner, Mina Arzaghi, Florian Carichon +2Multi-Agent LLM SystemsMulti-Agent Systems

  6. Auditing a Moving Target: Evidence Anchoring and the Expiry of Fairness Audits of Provider-Served Language Models

    Nov 16, 2026Mika FacchiniLLM AuditingLanguage Model Bias Evaluation

  7. Revealing the Biases of Large Language Models By Inferring Prior Distributions

    Nov 13, 2026Joshua Ryo Cheng, Thomas L. Griffiths, Jake C. SnellSocial Bias in Language ModelsLanguage Model Bias Evaluation

  8. Label-Free Detection of Structural Bias in LLM-Based Recommender Systems via Randomized Paraphrasing: The Case of Verbosity

    Nov 13, 2026Carlos Peláez-González, David Herrera-Poyatos, Francisco Herrera +1Language Model Bias EvaluationRecommender Systems

  9. AI Reviews Are Biased Towards the Appendix

    Nov 13, 2026Saptarshi Ghosh, Mohammad Saim, Tianyu JiangAutomated Peer ReviewBias in LLM-as-a-Judge

  10. Loyalty Capture: Reporting Relationships and Structural Sycophancy in Frontier AI Models

    Nov 11, 2026Eric So, Alex ImasLLM SycophancyAI Accountability

  11. Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines

    Nov 10, 2026Mo Zhou, Yiwei ZhangFraming Effects in Language ModelsLLM Reliability

  12. BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs

    Nov 9, 2026Provat Saha, Ashika Habib Khan, Tahseen Tanha +2Social Bias in Language ModelsLanguage Model Bias Evaluation

  13. GRAP: Gender Rubric-Aware Probing for Analyzing Gender Bias in LLMs

    Nov 9, 2026Seona Seo, Geonhee Ahn, Hyunsoo Cho +1Gender Bias in Language ModelsLanguage Model Bias Evaluation

  14. WarBias: A Bilingual LLM Benchmark of Stereotypes About Internally Displaced People and Veterans

    Nov 8, 2026Viktoriia Makovska, Tetiana Zakharchenko, Oleksandra OstafiichukSocial Bias in Language ModelsLanguage Model Bias Evaluation

  15. Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training

    Nov 5, 2026Chao Peter Yang, Catherine Zehr, Pranav Bijith +1LLM SycophancyLLM Post-Training

  16. The Male Abstraction Bias in Internet Culture and Large Language Models

    Nov 3, 2026Douglas Guilbeault, Bhargav Srinivasa DesikanGender Bias in Language ModelsSocial Bias in Language Models

  17. User Attributes and Political Ideology Shape LLM Receptiveness in Online Political Debates

    Nov 2, 2026Akram Elbouanani, Yousra Taleb, Gianluca Demartini +1Political Bias in Language ModelsSocial Bias in Language Models

  18. LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

    Nov 2, 2026Danial Amin, Joni Salminen, Bernard J. JansenGender Bias in Language ModelsLanguage Model Bias Evaluation

  19. Architecting Fairness-Aware LLM Interactions

    Oct 31, 2026Alessandra Parziale, Valeria Pontillo, Fabio Palomba +1Algorithmic FairnessSocial Bias in Language Models

  20. Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

    Oct 31, 2026Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López ValverdeAlgorithmic BiasLanguage Model Bias Evaluation

  21. AI evaluators favor job applications written by certain AI models, independent of candidate quality

    Oct 30, 2026Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu +1Algorithmic FairnessLanguage Model Bias Evaluation

  22. Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines

    Oct 30, 2026Jonghyeon Choi, Yeonjun Choi, Beakcheol JangPolitical Bias in Language ModelsLanguage Model Bias Evaluation

  23. Language Models Show a "Women-Are-Wonderful" Effect in Blame Attribution That is Not Just Leniency

    Oct 30, 2026Anika Sharma, Snehalkumar 'Neil' S. Gaikwad, Snigdha Chaturvedi +1Gender Bias in Language ModelsSocial Bias in Language Models

  24. Illusory Truth or Salient Anomaly? Model-Dependent Responses to Repeated Poisoned Passages in RAG-Style Contexts

    Oct 29, 2026Mehrdad Ghassabi, Audrina Ebrahimi, Sadra Hakim +3Language Model Bias EvaluationNormative Conformity in Language Models

  25. Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models

    Oct 29, 2026Nawar S. Alseelawi, Mustafa S. AljumailyLanguage Model Bias EvaluationCross-Cultural Language Model Evaluation