Language Model Bias Evaluation

Follow this topic Ask about this topic

Momentum

40 papers in the last four weeks, up 122% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 321

All topics

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
Aug 11, 2026Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie +1LLM Evaluation LLM-as-a-Judge
The University of Queensland · Brisbane, Australia
Status Association Does Not Reliably Predict Decision Leakage
Aug 10, 2026Abdullah XAI-Assisted Decision Making Social Bias in Language Models
Project AWARE
Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Access
Aug 9, 2026Lier Jin, Lan Hu, Binqi Shen +2Language Model Bias Evaluation Prompt Optimization
Fuqua School of Business Duke University Durham, USA · Department of Engineering Carnegie Mellon University Pittsburgh, USA · Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA +1
Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation
Aug 9, 2026Yu Wang, Zhe Zhou, Menglin Liu +1Language Model Bias Evaluation Classification
Cornell University · University of Washington · The Chinese University of Hong Kong, Shenzhen +1
Fair on the Surface? Benchmarking Hidden-Output Fairness Gaps in LLM Recommenders
Aug 8, 2026Chan Aristella Lu, Arya Fayyazi, Junhao Zhang +6LLM Auditing Language Model Bias Evaluation
University of Georgia · University of Southern California · Carnegie Mellon University +1
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
People Are Not Just Their Countries. Disentangling Social Determinants of LLM Value Alignment Across Europe
Aug 7, 2026Maria-Louisa Wightman, Guillaume Bied, Tijl De BieLLM Alignment Computational Social Science
Electronics and Information Systems (ELIS), Ghent University
Confirming Our Biases? Evaluating the Capabilities, Risks, and Societal Impact of Large Language Models
Aug 7, 2026Mudar Adas, Polina Tsvilodub, Michael Franke +1Prompt Sensitivity LLM Sycophancy
Neuro-Cognitive Modeling Group, University of Tübingen, Sand 14, 72076 Tübingen, Germany · Department of Linguistics, University of Tübingen, Keplerstraße 2, 72074 Tübingen
Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation
Aug 7, 2026Jonghyun Jee, Aaron ShawLLM Evaluation Cultural Bias in Language Models
Department of Communication Studies, Northwestern University
Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints
Aug 7, 2026Paul-Peter ArslanLLM Auditing Language Model Bias Evaluation
Institute for Future Technologies
Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests
Aug 7, 2026Seitaro Ono, Senna Ross, Jun SaikiNeural Representation Geometry Text Embedding Evaluation
Graduate School of Human and Environmental Studies, Kyoto University, Kyoto, Japan · Brock University, St. Catharines, Ontario, Canada
Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors
Aug 6, 2026Arya Labroo, Mengjie Qian, Kate KnillLanguage Model Bias Evaluation Automated Evaluation
University of Cambridge
Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts
Aug 6, 2026Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio +1LLM Sycophancy LLM Auditing
Université Côte d’Azur · Technical University Munich
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
Aug 6, 2026Lukas Twist, Twm Stone, Helen Yannakoudakis +1LLM Evaluation Language Model Bias Evaluation
King’s College London, London, UK · MATS Research
The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
Aug 4, 2026Irina Proskurina, Antoine Gourru, Julien VelcinSynthetic Data Pretraining Self-Training for Language Models
Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · Université Claude Bernard Lyon 1, Université Lumière Lyon 2, ERIC · École Centrale de Lyon, LIRIS, CNRS UMR 5205
VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs
Aug 4, 2026Andrei Chetvergov, Alexander Evseev, Timofei Sivoraksha +4LLM Evaluation Language Model Bias Evaluation
Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection
Aug 4, 2026Razieh Chalehchaleh, Reza Farahbakhsh, Noel CrespiGender Bias in Language Models Fake News Detection
SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France
Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili
Aug 4, 2026Ruolei Zhang, Teddy Njuguna, Yue FengMultilingual Language Model Evaluation Language Model Safety Evaluation
University of Birmingham
Who Should Be Generated? Justifying Demographic Targets in Open-Ended Generation
Aug 3, 2026Zeshen Zheng, Yujia He, Qianmian Lin +2Language Model Bias Evaluation Demographic Bias in Language Models
Sun Yat-sen University, China
Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese
Aug 3, 2026Naho Orita, Hayato Ogawa, Daisuke KawaharaLLM Auditing Social Bias in Language Models
Waseda University
TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs
Aug 1, 2026Jin Zhang, Linyu Li, Weili Jiang +7Cultural Bias in Language Models Language Model Bias Evaluation
University of Electronic Science and Technology of China · Peking University · Southwest Jiaotong University +1
Query Timing Produces Opposite Positional Biases Between LLMs and Humans
Aug 1, 2026Jasin Cekinmez, Addison J. Wu, Thomas L. GriffithsLanguage Model Bias Evaluation Belief Updating in LLMs
Princeton University
FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
Jul 31, 2026Martin LukkBenchmark Auditing LLM Auditing
University of Toronto
From Minds to Models: The Intersection of Psychology and LLM Behaviours
Jul 30, 2026Oliver Guidetti, Reza RyanSocial Bias in Language Models Implicit Bias
School of Psychology, Faculty of Science, Medicine and Health, University of Wollongong, Wollongong, NSW 2522, Australia. · School of Electrical Engineering, Computing and Mathematical Sciences, Faculty of Science and Engineering, Curtin University, Bentley, WA 6102, Australia.
Benchmarking LLM Competence on Logical Inference over Probability Operators
Jul 29, 2026Nayera Hasan, Jack Greff, Alvin GrissomProbabilistic Logic LLM Evaluation
Haverford College
Same Facts, Different Diagnosis: Measuring and Mitigating Narrative Anchoring in Clinical Language Models
Jul 29, 2026Prabhjot Singh, Pritam Deka, Vijay ChennareddySocial Bias in Language Models Clinical Reasoning
University of Texas at Austin Austin, Texas, USA · Queen’s University Belfast Belfast, United Kingdom · Middlesex University London, United Kingdom
Evaluating Regional Bias in LLMs From Abstract Stereotype to Concrete Social Decision-Making
Jul 29, 2026Jiayuan Di, Haoyi Yang, Yufei Luo +2Social Bias in Language Models Cultural Bias in Language Models
East China University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence · Beijing University of Posts and Telecommunications +2
OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
Jul 29, 2026Seonglae Cho, Adriano KoshiyamaLLM Alignment Language Model Bias Evaluation
1Holistic AI · University College London
Symphony of Bias: Exploring Gender Associations with Musical Instruments in Multimodal LLMs
Jul 29, 2026Farhan Farsi, Shayan Bali, Mohammad Heydari Rad +2Gender Bias in Language Models Multimodal Large Language Models
Amirkabir University of Technology · King’s College London · University of Tehran +1
Polistemics: Evaluating LLMs as Information Mediators in Politics & Elections
Jul 28, 2026Baran Peters, Gabor Hollbeck, Robert Jakob +1Political Bias in Language Models Language Model Bias Evaluation
ETH Agentic Systems Lab · CORDA

Aug 11, 2026, cs.IRJ/K move · Enter open · S save

Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation

Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie, Gianluca Demartini

The University of Queensland · Brisbane, Australia

LLM Evaluation LLM-as-a-Judge Language Model Bias Evaluation IR Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison. We study persona conditioning as a diagnostic mechanism for exposing LLM assessor sensitivity. Using task-oriented personas drawn from two complementary sources (PersonaHub and NVIDIA Nemotron-Personas-USA), we instantiate five assessor roles emphasizing intent interpretation, domain expertise, contrastive judgment, evidence verification, and global search-quality assessment, compared with a standard UMBRELA baseline. Across six LLM backbones on TREC DL20 and RAG24, our analyses reveal structured rather than uniform assessor sensitivity. Judgments usually remain close to the baseline while shifting assessment strictness, evidential threshold, or interpretation emphasis rather than producing widespread relevance reversals. At the system level, high-capacity models preserve system-ranking agreement, while smaller models amplify persona-induced instability. Local rank-displacement analysis shows sensitivity concentrates on particular retrieval systems and system types, especially neural ranking/reranking systems on DL20 and RAG-oriented pipelines on RAG24. Persona source matters less than assessor role and model capacity. These findings position persona-conditioned judging as a controlled sensitivity probe for stress-testing LLM-based IR evaluation pipelines and identifying systems whose evaluation outcomes are sensitive to assessor framing.

Momentum

Related topics

Latest papers 321