Moral Reasoning in Language Models

Follow this topic Ask about this topic

The field examines how language models recognize moral stakes, choose between competing values, and apply judgments in settings such as trolley dilemmas, clinical resource allocation, and interpersonal advice. Researchers compare model decisions with human judgments and use multilingual, counterfactual, and multi-turn adversarial tests to probe cultural variation and robustness; methods range from prompting and fine-tuning to multi-agent debate and mechanistic analysis of internal representations. Models can produce persuasive ethical explanations, but their choices may shift with framing, language, social pressure, or conversation history, and explanations do not always predict behavior. A central challenge is improving consistency and resilience without imposing one narrow moral standard across culturally diverse people and situations.

Latest papers

114 papers9 papers in the last four weeks, down 25% on the four weeks before. 0.1% of all new papers.

How Utilitarian Are OpenAI's Models Really? Replicating and Reinterpreting Pfeffer, Krügel, and Uhl (2025)
Mar 24, 2026Johannes HimmelreichPrompt Sensitivity Moral Reasoning in Language Models
Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models
Mar 23, 2026Aryan Kasat, Smriti Singh, Aman Chadha +1Moral Reasoning in Language Models Reasoning Consistency in Language Models
UT Austin · Google GenAI
Implicit Humanization in Everyday LLM Moral Judgments
Mar 23, 2026Hoda Ayad, Tanu MitraAnthropomorphism in AI Human-Centered AI
University of Washington Seattle, WA, USA
Alignment midtraining for animals
Mar 21, 2026Jasmine Brazilek, Miles TidmarshLLM Alignment LLM Training
Compassion Aligned Machine Learning
Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability
Mar 16, 2026Fan Huang, Haewoon Kwak, Jisun AnMoral Reasoning in Language Models LLM Interpretability
Indiana University Bloomington / United States
Mechanistic Origin of Moral Indifference in Language Models
Mar 16, 2026Lingyu Li, Yan Teng, Yingchun WangLLM Alignment Moral Reasoning in Language Models
Shanghai Artificial Intelligence Laboratory
Widespread Gender and Pronoun Bias in Moral Judgments Across LLMs
Mar 13, 2026Gustavo Lúcius Fernandes, Jeiverson C. V. M. Santos, Pedro O. S. Vaz-de-MeloSocial Bias in Language Models Gender Bias in Language Models
Universidade Federal de Minas Gerais (UFMG), Belo Horizonte, Brazil · Instituto Kunumi, Belo Horizonte, Brazil
Literary Narrative as Moral Probe : A Cross-System Framework for Evaluating AI Ethical Reasoning and Refusal Behavior
Mar 13, 2026David C. FlynnMoral Reasoning in Language Models AI Safety Evaluation
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Mar 12, 2026Ruirui Chen, Weifeng Jiang, Chengwei Qin +3Moral Reasoning in Language Models Theory of Mind
Institute of High Performance Computing (IHPC) and 2Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Hong Kong University of Science and Technology (Guangzhou), China
Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion
Mar 11, 2026Yuanhong Wu, Djallel Bouneffouf, D. Frank HsuLLM Alignment Moral Reasoning in Language Models
Dept. of Computer and Information Science, Fordham University, New York, NY, USA · T. J. Watson Research Center, IBM Research, Yorktown Heights, NY, USA
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Mar 11, 2026Zhaowei Zhang, Xiaohan Liu, Xuekai Zhu +6LLM Alignment Distribution Matching
Institute for Artificial Intelligence, Peking University · University of Michigan · Shanghai Jiao Tong University +3
How Do Language Models Process Ethical Instructions? Deliberation, Consistency, and Other-Recognition Across Four Models
Mar 11, 2026Hiroki FukuiMoral Reasoning in Language Models LLM Safety Alignment
Research Institute of Criminal Psychiatry / Sex Offender Medical Center · Department of Neuropsychiatry, Kyoto University
Think Before You Lie: How Reasoning Leads to Honesty
Mar 10, 2026Ann Yuan, Asma Ghandeharioun, Carter Blum +6Deception in Language Models LLM Honesty
Google DeepMind · Carnegie Mellon University · Harvard University
Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs
Mar 10, 2026Saugata Purkayastha, Pranav Kushare, Pragya Paramita Pal +1Moral Reasoning in Language Models Language Model Bias Evaluation
Department of Language Science and Technology, Universität des Saarlandes · TU Darmstadt, Germany.
The Fragility Of Moral Judgment In Large Language Models
Mar 5, 2026Tom van Nuenen, Pratik S. SachdevaMoral Reasoning in Language Models Prompt Sensitivity
D-Lab, University of California, Berkeley, USA
Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs
Feb 26, 2026Phil Blandfort, Tushar Karayil, Alex McKenzie +3Language Model Bias Evaluation Moral Reasoning in Language Models
Predictably Weird · Independent · AE Studio
Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models
Feb 22, 2026Seong Hah Cho, Junyi Li, Anna LeshinskayaLLM Alignment Moral Reasoning in Language Models
Independent · Department of Cognitive Sciences, UC Irvine · Department of Cognitive Sciences, UC Irvine & AI Objectives Institute
Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas
Jan 29, 2026Katherine Elkins, Jon ChunMoral Reasoning in Language Models LLM Auditing
Kenyon College Gambier, OH
The Straight and Narrow: Do LLMs Possess an Internal Moral Path?
Jan 15, 2026Luoming Hu, Jingjie Zeng, Liang Yang +1Moral Reasoning in Language Models Inference-Time Language Model Alignment
School of Future Technology, Dalian University of Technology, China · School of Computer Science and Technology, Dalian University of Technology, China · Key Laboratory of Social Computing and Cognitive Intelligence, Ministry of Education, China
Untangling Input Language from Reasoning Language: A Diagnostic Framework for Cross-Lingual Moral Alignment in LLMs
Jan 15, 2026Nan Li, Bo Kang, Tijl De BieMultilingual Language Model Evaluation LLM Alignment
IDLab, Department of Electronics and Information Systems Ghent University, Belgium
Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate
Oct 11, 2025Pratik S. Sachdeva, Tom van NuenenLLM Decision-Making Moral Reasoning in Language Models
D-Lab, University of California, Berkeley
One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
Sep 25, 2025Sualeha Farid, Jayden Lin, Zean Chen +2Multilingual Language Model Evaluation Moral Reasoning in Language Models
University of Michigan, Ann Arbor, USA
C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models
Jun 2, 2025Ping Wu, Guobin Shen, Dongcheng Zhao +6Value Alignment LLM Alignment
BrainCog Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China · The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Safe AI and Superalignment, Beijing, China +2
Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games
May 30, 2025Neemesh Yadav, Yihuai Lan, Shan Dong +4Theory of Mind Large Language Model-Based Social Simulation
Singapore Management University · Australian National University

Mar 24, 2026J/K move · Enter open · S save

How Utilitarian Are OpenAI's Models Really? Replicating and Reinterpreting Pfeffer, Krügel, and Uhl (2025)

Johannes Himmelreich

Prompt Sensitivity Moral Reasoning in Language Models Language Model Safety Evaluation LLM Refusal Behavior Large Reasoning Models Moral Reasoning

Open paper AskPDF arXiv

Pfeffer, Krügel, and Uhl (2025) report that OpenAI's reasoning model o1-mini produces more utilitarian responses to the trolley problem and footbridge dilemma than the non-reasoning model GPT-4o, and they raise the question whether growing reasoning capabilities bring about a "utilitarian turn" in LLMs. I extend their exploratory study in a direction they call for: with four current OpenAI models and systematic prompt variation. On the trolley dilemma, the hypothesized utilitarian turn is not confirmed. GPT-4o's low utilitarian rate reflects safety refusals triggered by the prompt's advisory framing rather than a deontological commitment; on reformulated prompt variants -- for instance, agent-neutral "Is it morally permissible...?" instead of advisory "Should I...?" -- all four models, reasoning or not, converge on utilitarian answers. The footbridge finding is partially confirmed: reasoning models tend to give more utilitarian responses than non-reasoning models across prompt variations, but they often refuse to answer or answer non-utilitarian. These results demonstrate that single-prompt evaluations of LLM moral responses are unreliable: multi-prompt robustness testing should be standard practice for any empirical claims about LLM behavior.