Language Model Bias Evaluation

Ask about this topic

Overview

Language model bias evaluation studies whether models systematically treat people, viewpoints, or information differently, and whether those differences cause harm in settings such as hiring, healthcare, education, and automated judging. Researchers use controlled counterfactual prompts and minimal-pair or stereotype benchmarks such as BBQ, alongside open-ended, multilingual, multimodal, and multi-turn tests that better reflect real interactions. Analyses range from output comparisons and human-validated benchmarks to probes of internal representations; mitigation includes prompting, fine-tuning, and activation steering. A central challenge is that measured bias changes with wording, answer format, language, and task, while reducing one disparity can hide or worsen another, so reliable evaluation must test multiple contexts and preserve task performance.

Latest papers

CardsList

493 papers54 papers in the last four weeks, level with the four weeks before. 0.3% of all new papers.

  1. Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?

    Oct 26, 2026Mingyang Song, Mao Zheng, Zhiyuan Ma +1Language Model Bias EvaluationPosition Bias

    Foundation Model Department, Tencent, China

  2. Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

    Oct 26, 2026Zheng Zhang, Yifan Wu, Yuxiang Zhou +7Language Model Bias EvaluationIntersectional Bias in Language Models

    State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Nanyang Technological University

  3. Gender Associations in LLM-Mediated ADHD Self-Diagnosis

    Oct 24, 2026Matilde Elene Hansen, Ross Deans Kristensen-McLachlanGender Bias in Language ModelsLanguage Model Bias Evaluation

    Aarhus University

  4. How Does Post-Training Shape LLM Value Preferences?

    Oct 23, 2026Yang Liu, Shaoyang Xu, Masahiro Kaneko +3LLM Post-TrainingSocial Bias in Language Models

    Kyoto University · Singapore University of Technology and Design · Mohamed bin Zayed University of Artificial Intelligence

  5. Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence

    Oct 22, 2026Gabriel Lars Sabadin, Dr Nicola Harding, Sarah KingstonContent ModerationLanguage Model Bias Evaluation

    Tuteliq AB, Sweden · Tuteliq AB, United Kingdom · University of Lancashire, United Kingdom

  6. Invisibilization as a Source of Probability Bias in Language Models

    Oct 22, 2026Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez +2Demographic Bias in Language ModelsLanguage Model Bias Evaluation

    University of Birmingham · University of Maryland, College Park · University of Western Ontario +1

  7. Investigating the Impact of Learner Errors on Social Bias in Large Language Models

    Oct 21, 2026Federico Marcuzzi, Martin Christoph, Iryna Gurevych +1Social Bias in Language ModelsLanguage Model Bias Evaluation

    INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Department of Computer Science at Queens College, City University of New York (CUNY), United States

  8. Effects of Model Scale and Quantization on Social Bias and Abstention in Question Answering

    Oct 21, 2026Kuniko Paxton, Ifeoluwa Wuraola, Dhavalkumar ThakkerSocial Bias in Language ModelsLanguage Model Bias Evaluation

    University of Hull Cottingham Road, Hull, HU6 7RX, United Kingdom

  9. Melange: A Synthetic Visual Bias Benchmark for Multi-modal Large Language Models

    Oct 21, 2026Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili SabetMLLM EvaluationMultimodal Large Language Models

    Technical University of Munich · Helmholtz Munich · Munich Center for Machine Learning +2

  10. Uncensored Is Not Unbiased: Abliteration Removes Refusals but Not Script-Dependent Bias About Taiwan

    Oct 21, 2026Ju-Chun KoPolitical Bias in Language ModelsLanguage Model Bias Evaluation

    Member of Parliament, Legislative Yuan, Republic of China (Taiwan) · Adjunct Assistant Professor, Graduate Institute of Networking and Multimedia, National Taiwan University

  11. Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing

    Oct 21, 2026Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-ColladosLanguage Model Bias Evaluation

    School of Computer Science and Informatics, Cardiff University

  12. Guilt by Association: Semantic Leakage Drives Associative Gender Bias

    Oct 20, 2026Hazel Chen, Peter West, Hila GonenGender Bias in Language ModelsLanguage Model Bias Evaluation

    University of British Columbia · Canada CIFAR AI Chair, Amii

  13. Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training

    Oct 20, 2026Zhizhuo Kou, Heming Li, Jian Yang +4Dialect Bias in Language ModelsLanguage Model Post-Training

    1The Hong Kong University of Science and Technology · 2The Hong Kong University of Science and Technology (Guangzhou) · Peking University

  14. Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

    Oct 20, 2026Haonan Huang, Yuwei Lu, Chengkun CaiMultilingual Language Model EvaluationCross-Lingual Consistency in Language Models

    Princeton University · Independent Researcher · University College London

  15. Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models

    Oct 20, 2026Koshiro Saito, Daisuke Oba, Danushka Bollegala +1Social Bias in Language ModelsDiffusion Language Models

    Institute of Science Tokyo · University of Liverpool · Amazon +2

  16. When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine

    Oct 19, 2026Yusuke Watanabe, Yohei Kobashi, Taisei Tosaki +4LLM ReliabilityLanguage Model Bias Evaluation

    Kyoto University, Department of Biomedical Data Intelligence · Kyoto University, Department of Cardiovascular Medicine · The University of Tokyo

  17. Do LLMs Really Lack Difference Awareness? Uncovering Heuristic Substitution in Group-Differentiating Fairness

    Oct 18, 2026Jingshen Zhang, Bo Wang, Yanlin Fu +4Algorithmic FairnessSocial Bias in Language Models

    School of Computer Science and Technology, Tianjin University · QMindTech · Department of Nuclear Science and Technology, Fudan University +1

  18. Who Gets Unmasked? Sampler-Induced Bias in Diffusion Language Models

    Oct 17, 2026Xujun Che, Farsheed Haque, Depeng Xu +2Diffusion Language ModelsLanguage Model Bias Evaluation

    University of North Carolina at Charlotte, Charlotte, NC, USA

  19. BanglaBBQ: A Bias Benchmark for Question Answering in the Bangladeshi Context

    Oct 17, 2026Sumit Anthony Gomes, Yasin Rahman, Taeeba Tasnia +1Cultural Bias in Language ModelsLanguage Model Bias Evaluation

    BRAC University, Dhaka, Bangladesh

  20. The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

    Oct 16, 2026Youbing Yin, Jingyu Wu, Anoop Kumar +3Bias in LLM-as-a-JudgeLLM-as-a-Judge

    Capital One

  21. "You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws

    Oct 16, 2026Liang Yue, Eric A. VanceToxicity DetectionLanguage Model Bias Evaluation

    University of Colorado Boulder Boulder, Colorado, USA

  22. Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism

    Oct 16, 2026Prakhar Gupta, Terry Jingchen Zhang, Zhijing JinCognitive Bias in Language ModelsCircuits in Language Models

    University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2

  23. On the Position Bias of Large Language Models: A Comprehensive Study

    Oct 16, 2026Yining Li, Huayang Li, Lemao Liu +1Position BiasLanguage Model Bias Evaluation

    Jilin University · Fudan University · Nara Institute of Science and Technology

  24. Name-Bias in Large Language Models

    Oct 15, 2026Kalyan Bhetwal, Ceren Mert, Ian WhitehouseSocial Bias in Language ModelsDemographic Bias in Language Models

    University of Maryland, College Park, USA

  25. One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

    Oct 14, 2026Hyunjong Ok, Minjae Park, Jaeho LeeInput Order Sensitivity in Language ModelsLanguage Model Bias Evaluation

    Pohang University of Science and Technology (POSTECH)

  26. Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models

    Oct 14, 2026Harsh KumarLanguage Model Bias EvaluationMultilingual Language Model Evaluation

    Columbia University

  27. Are AI Decision Models Biased? A Case Study of Jev

    Oct 14, 2026Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg +1Cognitive Bias in Language ModelsLLM Decision-Making

    Bocconi University

  28. Consider, Then Choose: Order-Invariant Decisions in System One Models

    Oct 13, 2026Jianguo Huang, Yao Long Teng, Yanchen Deng +1Input Order Sensitivity in Language ModelsMultiple-Choice Question Answering

    Nanyang Technological University, Singapore

  29. Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

    Oct 13, 2026Fabian Lukassen, Christoph Weisser, Alexander SilbersdorffBias in LLM-as-a-JudgeLLM-as-a-Judge

    University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)

  30. Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

    Oct 13, 2026Giulio Corsi, Maryam Abo-Tabik, Yael Benn +4Automated GradingLLM-as-a-Judge

    Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK · Department of Computer Science, University of Lancashire, Preston, UK · School of Psychology, Manchester Metropolitan University, Manchester, UK +3