Language Model Bias Evaluation

Ask about this topic

Overview

Language model bias evaluation studies whether models systematically treat people, viewpoints, or information differently, and whether those differences cause harm in settings such as hiring, healthcare, education, and automated judging. Researchers use controlled counterfactual prompts and minimal-pair or stereotype benchmarks such as BBQ, alongside open-ended, multilingual, multimodal, and multi-turn tests that better reflect real interactions. Analyses range from output comparisons and human-validated benchmarks to probes of internal representations; mitigation includes prompting, fine-tuning, and activation steering. A central challenge is that measured bias changes with wording, answer format, language, and task, while reducing one disparity can hide or worsen another, so reliable evaluation must test multiple contexts and preserve task performance.

Latest papers

CardsList

497 papers54 papers in the last four weeks, level with the four weeks before. 0.3% of all new papers.

  1. Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks

    Mar 19, 2026Rudra Jadhav, Janhavi Danve, Sonalika ShawLanguage Model Bias EvaluationLinguistic Bias in Language Models

    Department of Computer Science, Savitribai Phule Pune University, Pune, India · Department of Computer Science, Dr. D. Y. Patil School of Science and Technology, Pune, India

  2. Analysis Of Linguistic Stereotypes in Single and Multi-Agent Generative AI Architectures

    Mar 19, 2026Martina Ullasci, Marco Rondina, Riccardo Coppola +6Dialect Bias in Language ModelsLinguistic Bias in Language Models

    Politecnico di Torino Turin, Italy

  3. When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Making

    Mar 19, 2026Abhinaba Basu, Pavan ChakrabortyOwnership Bias in Language ModelsLanguage Model Bias Evaluation

    Indian Institute of Information Technology, Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)

  4. Auditing Preferences for Brands and Cultures in LLMs

    Mar 18, 2026Jasmine Rienecker, Katarina Mpofu, Naman Goel +4Jurisdictional Bias in Language ModelsLLM Auditing

    Stupid Human · University of Oxford

  5. Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

    Mar 18, 2026Chiara Manna, Hosein Mohebbi, Afra Alishahi +2Gender Bias in Language ModelsLanguage Model Bias Evaluation

    Research Center for Cognitive Science and Artificial Intelligence Tilburg School of Humanities and Digital Sciences Tilburg University, Netherlands

  6. Sentiment Classification of Gaza War Headlines: A Comparative Analysis of Large Language Models and Arabic Fine-Tuned BERT Models

    Mar 18, 2026Amr Eleraqi, Hager H. Mustafa, Abdul Hadi N. AhmedSentiment AnalysisLanguage Model Bias Evaluation

    Faculty of Graduate Studies and Research, Cairo University, Egypt · Independent Researcher, Anmat Media

  7. LLM Nepotism in Organizational Governance

    Mar 18, 2026Shunqi Mao, Wei Guo, Dingxin Zhang +2Algorithmic BiasSocial Bias in Language Models

    School of Computer Science, The University of Sydney

  8. Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)

    Mar 18, 2026Zichen Tang, Zirui Zhang, Qian Wang +3Large Language Model-Based RecommendationLLM Agent Reliability

    The Hong Kong University of Science and Technology · National University of Singapore · The Hong Kong University of Science and Technology (Guangzhou)

  9. A Representation-Level Assessment of Bias Mitigation in Foundation Models

    Mar 17, 2026Svetoslav Nizhnichenkov, Rahul Nair, Elizabeth Daly +1Gender Bias in Language ModelsSocial Bias in Language Models

    IBM Research, Dublin, Ireland · School of Computer Science, University College Dublin, Dublin, Ireland

  10. Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models

    Mar 16, 2026Nandini Arimanda, Achyuth Mukund, Sakthi Balan Muthiah +1Intersectional Bias in Language ModelsSocial Bias in Language Models

    Department of Computer Science and Engineering, Shiv Nadar University Chennai, Tamil Nadu, India · School of AI and CS, Plaksha University, Mohali, Punjab, India

  11. The Hrunting of AI: Where and How to Improve English Dialectal Fairness

    Mar 16, 2026Wei Li, Adrian de WynterLinguistic Bias in Language ModelsLanguage Model Bias Evaluation

    Boston College · Microsoft · The University of York

  12. The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatments

    Mar 16, 2026Elmira Salari, Maria Claudia Nunes Delfino, Hazem Amamou +4Political Bias in Language ModelsRetrieval-Augmented Generation

    Wichita State University · Pontifícia Universidade Católica de São Paulo · Institut national de la recherche scientifique +1

  13. The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

    Mar 15, 2026Shree Harsha Bokkahalli Satish, Christoph Minixhofer, Maria Teleki +5Intersectional Bias in Language ModelsLanguage Model Bias Evaluation

    Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden · Centre for Speech Technology Research, University of Edinburgh, UK · Texas A&M University, USA

  14. Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation

    Mar 14, 2026Petter TörnbergSocial Bias in Language ModelsDemographic Bias in Language Models

    Institute for Logic, Language and Computation (ILLC), University of Amsterdam„ Science Park 107, 1098 XG Amsterdam, Netherlands

  15. Widespread Gender and Pronoun Bias in Moral Judgments Across LLMs

    Mar 13, 2026Gustavo Lúcius Fernandes, Jeiverson C. V. M. Santos, Pedro O. S. Vaz-de-MeloSocial Bias in Language ModelsGender Bias in Language Models

    Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, Brazil · Instituto Kunumi, Belo Horizonte, Brazil

  16. SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

    Mar 13, 2026Aditya Maheshwari, Amit Gajkeshwar, Kaushal Sharma +1Social Bias in Language ModelsLanguage Model Bias Evaluation

    Indian Institute of Management Indore, India

  17. LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

    Mar 12, 2026Himel Ghosh, Nick Elias WernerLanguage Model Bias EvaluationLLM Evaluation

    Technical University of Munich, Germany · Sapienza University of Rome, Italy

  18. Gender Bias in Generative AI-assisted Recruitment Processes

    Mar 12, 2026Martina Ullasci, Marco Rondina, Riccardo Coppola +1AI and LaborGender Bias in Language Models

    Department of Control and Computer Engineering Politecnico di Torino, Italy Turin, Italy

  19. WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics

    Mar 11, 2026Sneha Maurya, Spandana Govindgari, Girish Kumar +1Clinical Language Model EvaluationLLM Safety Benchmarks

    Columbia University · Akhara AI

  20. MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis

    Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-JudgeClustering

    NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan

  21. Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs

    Mar 10, 2026Saugata Purkayastha, Pranav Kushare, Pragya Paramita Pal +1Moral Reasoning in Language ModelsLanguage Model Bias Evaluation

    Department of Language Science and Technology, Universität des Saarlandes · TU Darmstadt, Germany.

  22. Investigating Gender Stereotypes in Large Language Models via Social Determinants of Health

    Mar 10, 2026Trung Hieu Ngo, Adrien Bazoge, Solen Quiniou +2Intersectional Bias in Language ModelsSocial Bias in Language Models

    Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Nantes Université, CHU Nantes, Clinique des données, INSERM, CIC 1413, Nantes, France

  23. Bioalignment: Measuring and Improving LLM Disposition Toward Biological Systems for AI Safety

    Mar 10, 2026Trent R Northen, Mingxun WangLLM AlignmentLLM Safety

    Bioaligned Labs · Lawrence Berkeley National Lab · Computer Science & Engineering Dept, UC Riverside

  24. Quantifying Gender Bias in Large Language Models: When ChatGPT Becomes a Hiring Manager

    Mar 10, 2026Nina Gerszberg, Janka Hamori, Andrew LoGender Bias in Language ModelsAlgorithmic Bias

    Massachusetts Institute of Technology Cambridge, Massachusetts, USA

  25. Towards a more efficient bias detection in financial language models

    Mar 9, 2026Firas Hadj Kacem, Ahmed Khanfir, Mike PapadakisFinancial Language Model EvaluationIntersectional Bias in Language Models

    SnT, University of Luxembourg, Luxembourg · RIADI, ENSI, University of Manouba, Tunisia

  26. Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

    Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-JudgeSocial Bias in Language Models

    Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China

  27. Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context

    Mar 8, 2026Ashish Pandey, Tek Raj ChhetriSocial Bias in Language ModelsCultural Bias in Language Models

    Center for Artificial Intelligence Research Nepal, Sundarharaincha-09, Buddhabare, Nepal

  28. Writing literature reviews with AI: principles, hurdles and some lessons learned

    Mar 8, 2026Saadi Lahlou, Annabelle Gouttebroze, Atrina Oraee +1LLM SycophancyLLM Evaluation

    London School of Economics and Political Science · Paris Institute for Advanced Study

  29. Rigidity in LLM Bandits with Implications for Human-AI Dyads

    Mar 8, 2026Haomiaomiao Wang, Tomás E Ward, Lili ZhangLLM Decision-MakingLanguage Model Bias Evaluation

    Insight Research Ireland Centre for Data Analytics, Ireland · School of Computing, Dublin City University, Ireland

  30. Abductive Reasoning with Syllogistic Forms in Large Language Models

    Mar 6, 2026Hirohiko Abe, Risako Ando, Takanobu Morishita Kentaro Ozeki +2Abductive Reasoning in Language ModelsAbductive Reasoning

    Keio University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan