Language Model Bias Evaluation

Ask about this topic

Overview

Language model bias evaluation studies whether models systematically treat people, viewpoints, or information differently, and whether those differences cause harm in settings such as hiring, healthcare, education, and automated judging. Researchers use controlled counterfactual prompts and minimal-pair or stereotype benchmarks such as BBQ, alongside open-ended, multilingual, multimodal, and multi-turn tests that better reflect real interactions. Analyses range from output comparisons and human-validated benchmarks to probes of internal representations; mitigation includes prompting, fine-tuning, and activation steering. A central challenge is that measured bias changes with wording, answer format, language, and task, while reducing one disparity can hide or worsen another, so reliable evaluation must test multiple contexts and preserve task performance.

Latest papers

CardsList

497 papers54 papers in the last four weeks, level with the four weeks before. 0.3% of all new papers.

  1. Large Language Models Exhibit Normative Conformity

    Apr 21, 2026Mikako Bito, Keita Nishimoto, Kimitaka Asatani +1Normative Conformity in Language ModelsSocial Bias in Language Models

    The University of Tokyo

  2. Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs

    Apr 21, 2026Guy Mor-Lan, Omer Goldman, Matan Eyal +6Multilingual Language ModelsMultilingual Language Model Evaluation

    GGoogle Research · BBar-Ilan University · CUniversity of Cambridge +1

  3. Beyond Semantic Similarity: A Component-Wise Evaluation Framework for Medical Question Answering Systems with Health Equity Implications

    Apr 21, 2026Abu Noman Md Sakib, Md. Main Oddin Chisty, Zijie ZhangLLM Answer VerificationHealthcare

    University of Texas at San Antonio, USA · Khulna University of Engineering and Technology, Bangladesh

  4. More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models

    Apr 21, 2026Xiao WangMultiple-Choice Question AnsweringCoT Reasoning

  5. AlignCultura: Towards Culturally Aligned Large Language Models?

    Apr 21, 2026Gautam Siddharth Kashyap, Mark Dras, Usman NaseemCross-Cultural Language Model EvaluationCultural Alignment

    School of Computing, Macquarie University, Australia

  6. LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

    Apr 20, 2026Pedro Barbosa de Carvalho NetoLegal NLPLLM Evaluation

    Teresina, PI, Brazil

  7. Investigating Counterfactual Unfairness in LLMs towards Identities through Humor

    Apr 20, 2026Shubin Kim, Yejin Son, Junyeong Park +6Social Bias in Language ModelsLanguage Model Bias Evaluation

    ♡Yonsei University · ♠KAIST · ♣Seoul National University +1

  8. Six Llamas: Comparative Religious Ethics Through LoRA-Adapted Language Models

    Apr 20, 2026Chad Coleman, W. Russell Neuman, Manan Shah +5Moral Reasoning in Language ModelsCultural Bias in Language Models

  9. First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows

    Apr 20, 2026Sihao Xing, Zaur GoulievHealthcareSocial Bias in Language Models

    School of Enterprise Computing and Digital Transformation Technological University Dublin · School of Information and Communication Studies University College Dublin

  10. Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations

    Apr 19, 2026S. A. Desimone, L. Alonso AlemanySocial Bias in Language ModelsLanguage Model Bias Evaluation

    Universidad Nacional de C´ordoba · CONICET

  11. VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

    Apr 19, 2026Yi-Cheng Lin, Yusuke Hirota, Sung-Feng Huang +1Audio-Language Model EvaluationGender Bias in Language Models

    Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · NVIDIA Research NVIDIA Taipei, Taiwan · AI Center of Research Excellence National Taiwan University Taipei, Taiwan

  12. Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering

    Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware EngineeringLLM-as-a-Judge

    University of British Columbia, Canada

  13. Expressing Social Emotions: Misalignment Between LLMs and Human Cultural Emotion Norms

    Apr 18, 2026Sree Bhattacharyya, Manas Mehta, Leona Chen +4LLM EvaluationCultural Bias in Language Models

    The Pennsylvania State University · Duke University · Northeastern University +1

  14. Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineering

    Apr 18, 2026Francesco Sovrano, Gabriele Dominici, Alberto BacchelliSoftware EngineeringCognitive Bias in Language Models

    Collegium Helveticum at ETH Zurich, Switzerland · University of Zurich, Switzerland · University of Italian-speaking Switzerland, Switzerland

  15. Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

    Apr 17, 2026Nicolò Pagan, Christopher Barrie, Chris Andrew Bail +1LLM PromptingSocial Bias in Language Models

    University of Zurich, Department of Informatics, Zurich, CH · New York University, Department of Sociology, New York, USA · University of Oxford, Department of Sociology, Oxford, UK +2

  16. Context Over Content: Exposing Evaluation Faking in Automated Judges

    Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-JudgeLanguage Model Safety Evaluation

    1BITS Pilani, India · University of Michigan

  17. SPAGBias: Uncovering and Tracing Structured Spatial Gender Bias in Large Language Models

    Apr 16, 2026Binxian Su, Haoye Lou, Shucheng Zhu +4Gender Bias in Language ModelsLanguage Model Bias Evaluation

    School of Information Science, Beijing Language and Culture University, Beijing, China · 2Libraries, Renmin University of China, Beijing, China · School of Humanities, Tsinghua University, Beijing, China +2

  18. VoxSafeBench: Not Just What Is Said, but Who, How, and Where

    Apr 16, 2026Yuxiang Wang, Hongyu Liu, Yijiang Xu +9Language Model Safety EvaluationPrivacy Leakage in Language Models

    1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 3Amphion Technology Co., Ltd.

  19. TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

    Apr 15, 2026Yichuan Mo, Yukun Jiang, Yanbo Shi +4Diffusion Language Model BenchmarkingLLM Safety Benchmarks

    State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University. · CISPA Helmholtz Center for Information Security · School of EECS, Peking University +1

  20. The role of System 1 and System 2 semantic memory structure in human and LLM biases

    Apr 14, 2026Katherine Abramski, Giulio Rossetti, Massimo StellaDual-Process ReasoningCognitive Modeling

    Department of Computer Science, University of Pisa, Italy · Institute of Information Science and Technologies - National Research Council, Italy · CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento, Italy

  21. Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Models

    Apr 13, 2026Benjamin Maltbie, Shivam RavalIntersectional Bias in Language ModelsLLM Sycophancy

    Massachusetts Institute of Technology, Cambridge, MA, USA · Harvard University, Cambridge, MA, USA

  22. Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

    Apr 9, 2026Jiawei Chen, Ruoxi Xu, Boxi Cao +12Human Behavior SimulationUser Simulation

    Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Kuaishou Technology

  23. Language Bias under Conflicting Information in Multilingual LLMs

    Apr 8, 2026Robert Östling, Murathan KurfalıLinguistic Bias in Language ModelsLanguage Model Bias Evaluation

    Stockholm University Sweden · RISE Research Institutes of Sweden Sweden

  24. Digital Skin, Digital Bias: Uncovering Tone-Based Biases in LLMs and Emoji Embeddings

    Apr 8, 2026Mingchen Li, Wajdi Aljedaani, Yingjie Liu +5Demographic Bias in Language ModelsLanguage Model Bias Evaluation

    University of North Texas Denton, TX, United States · International Center for Artificial Intelligence Research and Ethics Riyadh, Saudi Arabia · Southern Methodist University Dallas, TX, United States +2

  25. The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission

    Apr 7, 2026Theodor SpiroCorrelated Errors in Language ModelsLLM Reliability

  26. Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning

    Apr 7, 2026Yanbei Jiang, Amr Keleg, Ryandito Diandaru +4LLM AlignmentLLM Fine-Tuning

    University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence · Oracle

  27. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

    Apr 7, 2026Xin Sun, Di Wu, Sijing Qin +3Bias in LLM-as-a-JudgeLLM-as-a-Judge

    National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · Hitotsubashi University, Japan +3

  28. Can We Trust a Black-box LLM? LLM Untrustworthy Boundary Detection via Bias-Diffusion and Multi-Agent Reinforcement Learning

    Apr 7, 2026Xiaotian Zhou, Di Tang, Xiaofeng Wang +1LLM AuditingLanguage Model Bias Evaluation

    Worcester Polytechnic Institute, USA · Indiana University Bloomington, USA

  29. Attribution Bias in Large Language Models

    Apr 6, 2026Eliza Berman, Bella Chang, Daniel B. Neill +1Claim Attribution in Language ModelsSocial Bias in Language Models

    New York University

  30. How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline

    Apr 5, 2026Mir Tafseer Nayeem, Davood RafieiLanguage Model PretrainingLLM Auditing

    Department of Computing Science University of Alberta