Language Model Bias Evaluation

Follow this topic Ask about this topic

Momentum

40 papers in the last four weeks, up 122% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 321

All topics

AI-Mediated Communication Can Steer Collective Opinion
May 15, 2026Stratis Tsirtsis, Kai Rawal, Chris Russell +2Opinion Dynamics LLM Auditing
1Hasso Plattner Institute · 2Oxford Internet Institute, University of Oxford · 3Weizenbaum Institute
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLLM-as-a-Judge Faithfulness of Language Model Explanations
Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.
Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
May 12, 2026Andreas Maier, Jeta Sopa, Gozde Gul Sahin +2LLM Auditing LLM Prompting
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
Pretraining Exposure Explains Popularity Judgments in Large Language Models
May 12, 2026Jamshid Mozafari, Bhawna Piryani, Adam JatowtLanguage Model Pretraining Language Model Bias Evaluation
University of Innsbruck Innsbruck, Austria
GKnow: Measuring the Entanglement of Gender Bias and Factual Gender
May 12, 2026Leonor Veloso, Hinrich SchützeGender Bias in Language Models LLM Interpretability
Center for Information and Language Processing, LMU Munich · Munich Center for Machine Learning (MCML)
Simulating Eating Disorder Patients with LLMs: Evaluating Psychological Persona Stability in Multi-Turn Conversations
May 12, 2026Jennifer Haase, Jana Gonnermann-Müller, See Heng Yim +3Persona Consistency Personality Modeling in Language Models
Weizenbaum Institute, Berlin, Germany · HU Berlin, Berlin, Germany · Zuse Institute Berlin, Berlin, Germany +1
Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science
May 12, 2026Varun KotteSocial Bias in Language Models LLM-Assisted Annotation
Independent Researcher
Causal Bias Detection in Generative Artificial Intelligence
May 12, 2026Drago PleckoAlgorithmic Fairness Language Model Bias Evaluation
Department of Statistics & Data Science UCLA
How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation
May 11, 2026Eduardo Tenorio, Karuna Bhaila, Xintao WuSocial Bias in Language Models Language Model Bias Evaluation
University of Arkansas
Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
May 11, 2026Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada +7LLM Auditing Multimodal Large Language Models
Department of Electrical & Computer Engineering · Department of Computer and Information Science and Engineering · Department of Clinical and Health Psychology +1
Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks
May 11, 2026Regina Gugg, Selina Niederländer, Andreas Stöckl +1Language Model Safety Evaluation Language Model Bias Evaluation
Dynatrace Research, Linz, Austria · University of Applied Sciences Upper Austria, Hagenberg, Austria
StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
May 11, 2026Pierre Le Jeune, Étienne Duchesne, Weixuan Xiao +4Open-Ended Generation Multilingual Language Model Evaluation
Giskard AI · École Normale Supérieure, INSERM, Paris
BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence
May 9, 2026Jialing Gan, Junhao Dong, Songze LiPrompt Sensitivity LLM Auditing
Southeast University, China · Nanyang Technological University, Singapore
Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups
May 9, 2026Gautam VeldandaLLM Auditing Algorithmic Fairness
Independent Researcher
Political Plasticity: An Analysis of Ideological Adaptability in Large Language Models
May 8, 2026Bruno Bianchi, Diego Tiscornia, Matias Travizano +1Political Bias in Language Models Language Model Bias Evaluation
Lab. de Inteligencia Artificial Aplicada, Instituto de Ciencias de la Computación · Dpto. de Computación, Facultad de Cs. Exactas y Naturales, UBA-Argentina · Disarmista
CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
May 8, 2026Taein Lim, Seongyong Ju, Munhyeok Kim +2AI Agent Security Benchmarks LLMs for Cybersecurity
Chung-Ang University · Myongji University
Is She Even Relevant? When BERT Ignores Explicit Gender Cues
May 8, 2026Jonas Klein, Chiara Manna, Eva VanmassenhoveLanguage Model Pretraining Gender Bias in Language Models
Department of Computational Cognitive Science, Tilburg University, Warandelaan 2, 5037 AB Tilburg, Netherlands
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
May 8, 2026Moaath Alshaikh, Tasneem Alshaher, Ricardo Vieira +7Software Engineering LLM Evaluation
Federal University of Bahia · State University of Feira de Santana · Federal University of Sergipe +1
MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media
May 7, 2026Souvik Pramanik, S. M. Riaz Rahman Antu, Shak Mohammad Abyad +2LLM Evaluation LLM-Assisted Annotation
North South University, Dhaka, Bangladesh
Are LLMs Ready for Conflict Monitoring? Empirical Evidence from West Africa
May 5, 2026Hoffmann Muki, Olukunle OwolabiText Classification Adversarial Robustness of Language Models
University of Maryland, College Park, USA · Independent Researcher, Seattle, USA
EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage
May 5, 2026Richard J. Young, Alice M. MatthewsGender Bias in Language Models Clinical Triage
University of Nevada, Las Vegas, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA · Concorde Career Colleges, Dept. of Cardiovascular and Medical Diagnostic Sonography
TriBench-Ko: Evaluating LLM Risks in Judicial Workflows
May 5, 2026Haesung Lee, Gyubin Choi, Eun-Ju Lee +5LLM Evaluation Language Model Safety Evaluation
Center for Trustworthy AI, Seoul National University · Graduate School of Data Science, Seoul National University · Responsible AI Team, KT +1
Measuring and Mitigating Bias in Code Generated by Large Language Models
May 5, 2026Yuxi Chen, Yutian Tang, Timothy StorerAlgorithmic Bias Language Model Bias Evaluation
School of Computing Science, University of Glasgow, United Kingdom.
Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability
May 4, 2026Yash Aggarwal, Atmika Gorti, Vinija Jain +3Moral Reasoning in Language Models Social Bias in Language Models
University of Maryland, College Park · 2Purdue University · 3Meta +5
Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
May 4, 2026Muneeb Ur Raheem KhanDebiased ML Language Model Decoding
Lahore University of Management Sciences (LUMS)
Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation
May 3, 2026Xuemei Tang, Xufeng Duan, Zhenguang G. CaiLanguage Model Bias Evaluation Position Bias
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
The Invisible Coalition Partner: How LLMs Vote When Democracy Gets Concrete
May 3, 2026Joel BarmettlerPolitical Bias in Language Models Language Model Bias Evaluation
Independent Researcher Zurich, Switzerland
Are LLMs More Skeptical of Entertainment News?
May 3, 2026Huiqian LaiFake News Detection Text Classification
School of Information Studies, Syracuse University
Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models
May 2, 2026William Guey, Wei Zhang, Pierrick Bougault +4Multilingual Language Model Evaluation LLM Auditing
Department of Industrial Engineering, Tsinghua University, Beijing, China · School of Social Sciences, Tsinghua University, Beijing, China · Department of Industrial Engineering, Federal University of Rio de Janeiro, Brazil
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
May 2, 2026Plawan Kumar Rath, Rahul MaliakkalLLM Quantization Social Bias in Language Models
Meta

May 15, 2026, cs.CYJ/K move · Enter open · S save

AI-Mediated Communication Can Steer Collective Opinion

Stratis Tsirtsis, Kai Rawal, Chris Russell, Brent Mittelstadt+1

1Hasso Plattner Institute · 2Oxford Internet Institute, University of Oxford · 3Weizenbaum Institute

Opinion Dynamics LLM Auditing Political Bias in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Generative artificial intelligence (AI) is increasingly integrated into the online platforms where humans exchange opinions; large language models (LLMs) now polish users' posts on LinkedIn and provide context for content shared on X. While prior work has shown that AI can express biased opinions and shape individuals' opinions during human-AI interactions, less attention has been paid to its influence on collective opinion formation when mediating human-to-human communication. We address this gap via a combination of empirical and theoretical analyses. We show empirically that LLMs from multiple popular families introduce directional biases when instructed to edit human-written texts on contested topics, for example, nudging texts in favor of gun control and against atheism. Building on this observation, we introduce a mathematical model of opinion dynamics in which an AI system sits between users on a social network, transforming the opinions they express and perceive. By analytically characterizing the equilibrium of this model and performing simulations on real social network data, we show that biases introduced by AI in human-to-human communication can be amplified through the network and shift collective opinion in their direction. In light of these findings, we investigate whether such biases are controllable by online platforms. We audit the "Explain this post" feature on X and find evidence of pro-life bias in Grok's outputs on abortion-related content, which we trace back to specific design choices. We conclude with a discussion of the broader implications of our findings in relation to ongoing legislative efforts in the European Union.

Momentum

Related topics

Latest papers 321