Language Model Bias Evaluation

Ask about this topic

Overview

Language model bias evaluation studies whether models systematically treat people, viewpoints, or information differently, and whether those differences cause harm in settings such as hiring, healthcare, education, and automated judging. Researchers use controlled counterfactual prompts and minimal-pair or stereotype benchmarks such as BBQ, alongside open-ended, multilingual, multimodal, and multi-turn tests that better reflect real interactions. Analyses range from output comparisons and human-validated benchmarks to probes of internal representations; mitigation includes prompting, fine-tuning, and activation steering. A central challenge is that measured bias changes with wording, answer format, language, and task, while reducing one disparity can hide or worsen another, so reliable evaluation must test multiple contexts and preserve task performance.

Latest papers

CardsList

497 papers54 papers in the last four weeks, level with the four weeks before. 0.3% of all new papers.

  1. When Do Language Models Endorse Limitations on Human Rights Principles?

    Mar 4, 2026Keenan Samway, Nicole Miu Takagi, Rada Mihalcea +4LLM AlignmentLanguage Model Bias Evaluation

    Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · University of Michigan +2

  2. Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact

    Mar 1, 2026Michael Hardy, Yunsung KimLLM AlignmentLLM Evaluation

    Stanford University

  3. Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs

    Feb 26, 2026Phil Blandfort, Tushar Karayil, Alex McKenzie +3Language Model Bias EvaluationMoral Reasoning in Language Models

    Predictably Weird · Independent · AE Studio

  4. Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts

    Feb 25, 2026Jessica Y. Bo, Lillio Mok, Ashton AndersonHuman-AI Decision MakingLLM Decision-Making

    Computer Science University of Toronto

  5. Evaluating the Usage of African-American Vernacular English in Large Language Models

    Feb 25, 2026Deja Dunlap, R. Thomas McCoyDialect Bias in Language ModelsLinguistic Bias in Language Models

    Yale University, New Haven CT 06511, USA

  6. QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs

    Feb 24, 2026Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye +48Mathematical Reasoning BenchmarksLLM Evaluation

  7. Assessing Cognitive Biases in LLMs for Judicial Decision Support: Virtuous Victim and Halo Effects

    Feb 21, 2026Sierra S. LiuCognitive Bias in Language ModelsLanguage Model Bias Evaluation

    Millburn High School, New Jersey, USA

  8. ABCD: All Biases Come Disguised

    Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM EvaluationLanguage Model Bias Evaluation

    Dartmouth College

  9. Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

    Feb 19, 2026Kensuke Okada, Yui Furukawa, Kyosuke BunjiLanguage Model Bias EvaluationPersonality Modeling in Language Models

    The University of Tokyo · Kobe University

  10. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

    Feb 19, 2026Dusan BosnjakovicLanguage Model AuditingLanguage Model Bias Evaluation

  11. Intra-Fairness Dynamics: The Bias Spillover Effect in Targeted LLM Alignment

    Feb 18, 2026Eva Paraschou, Line Harder Clemmensen, Sneha DasLLM AlignmentLanguage Model Bias Evaluation

    Department of Applied Mathematics and Computer Science, Technical University of Denmark, 2800 Lyngby, Denmark · Department of Mathematical Sciences, University of Copenhagen, 2100 Copenhagen, Denmark

  12. Are LLMs Ready to Replace Bangla Annotators?

    Feb 18, 2026Md. Najib Hasan, Touseef Hasan, Souvika SarkarLLM-Assisted AnnotationLLM Reliability

    A2I Lab, School of Computing, Wichita State University

  13. In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations

    Feb 17, 2026Mohammad Aflah Khan, Mahsa Amani, Soumi Das +5Language Model Bias Evaluation

    Max Planck Institute for Software Systems · Microsoft, Hyderabad

  14. Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

    Feb 16, 2026Kawin Mayilvaghanan, Siddhant Gupta, Ayush KumarLanguage Model Bias EvaluationDemographic Bias in Language Models

    Observe.AI Bangalore, India

  15. Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

    Feb 16, 2026Lance Calvin Lim Gamboa, Yue Feng, Mark LeeSocial Bias in Language ModelsLanguage Model Bias Evaluation

    School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University

  16. FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

    Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM EvaluationLLM-as-a-Judge

    College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

  17. Investigating Social Bias Changes in Quantized Language Models

    Feb 5, 2026Stanley Z. Hua, Sanae Lotfi, Irene Y. ChenLLM QuantizationSocial Bias in Language Models

    UC Berkeley & UCSF · Meta Superintelligence Labs

  18. Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

    Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-JudgeLanguage Model Bias Evaluation

    OMRON SINIC X · NexaScience

  19. Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

    Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-JudgeLanguage Model Generation Evaluation

    Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2

  20. SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

    Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal UnderstandingMultimodal Grounding

    Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada

  21. White-Box Sensitivity Auditing with Steering Vectors

    Jan 23, 2026Hannah Cyberey, Yangfeng Ji, David EvansAlgorithmic AuditingLLM Auditing

    University of Virginia

  22. Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models

    Jan 21, 2026Shahar Ben-Natan, Oren TsurLLM EvaluationLLM Sycophancy

    Computer and Information Science Ben Gurion University

  23. Pro-AI Bias in Large Language Models

    Jan 20, 2026Benaya Trabelsi, Jonathan Shaki, Sarit KrausLanguage Model Bias EvaluationLLM Auditing

    Bar Ilan University

  24. A Scalable Entity-Based Framework for Auditing Bias in Large Language Models

    Jan 18, 2026Akram Elbouanani, Aboubacar Tuo, Adrian PopescuLLM AuditingPolitical Bias in Language Models

    Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

  25. The Language You Ask In: Language-Conditioned Ideological Divergence in LLM Analysis of Contested Political Documents

    Jan 17, 2026Oleg SmirnovMultilingual Language Model EvaluationPolitical Bias in Language Models

    Microsoft / Stockholm, Sweden

  26. Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection

    Jan 17, 2026Muhammad Alif Al Hakim, Alfan Farizki Wicaksono, Fajri KotoLLM QuantizationMultilingual Language Model Evaluation

    Faculty of Computer Science, Universitas Indonesia · Mohamed bin Zayed University of Artificial Intelligence

  27. Epistemic Constitutionalism Or: how to avoid coherence bias

    Jan 16, 2026Michele LoiAI GovernanceLanguage Model Bias Evaluation

    University of Milan

  28. Stylistic Evolution and LLM Neutrality in Singlish Language

    Jan 10, 2026Linus Tze En Foo, Weihan Angela Ng, Wenkai Li +1Dialect Bias in Language ModelsLanguage Model Bias Evaluation

    Independent Researcher, Singapore, Singapore · ETH Zürich, Zürich, Switzerland · Carnegie Mellon University, Pittsburgh, US

  29. Behavioral Coherence: A Method for Sensitive-Domain LLM Evaluation

    Dec 15, 2025Anika Sharma, Malavika Mampally, Chidaksh Ravuru +2Language Model Bias EvaluationDemographic Bias in Language Models

    Society-Centered AI Lab UNC Chapel Hill, USA · School of Nursing UNC Chapel Hill, USA

  30. Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles

    Nov 8, 2025Fatima Jahara, Mark Dredze, Sharon LevyGender Bias in Language ModelsLanguage Model Bias Evaluation

    Rutgers University · Johns Hopkins University