cs.AINov 7, 2026

How to make a moral agent: sentimentalism and rationalism as lenses on AI

Authors: David Gottlieb, Jared Moore

Organizations: Stanford University

Abstract

Can AI systems be moral agents? That is, can they undertake actions that can be correctly assessed as morally right or wrong? There is no settled normative criterion of moral agency that applies neatly outside paradigm human cases. Because of the difficulties with a normative account, we instead propose a research program to explore descriptive moral agency. Most basically, we ask, what are the capabilities we humans possess that underlie our moral agency? To do so, we draw on historical scholarship in philosophy and moral psychology to isolate candidate capabilities. Then we suggest refining candidate capabilities so that they are testable in the context of AI systems. Finally, we offer specific empirical recommendations to test AI systems, especially large language models (LLMs), for those refined capabilities. Our approach offers several benefits. Firstly, it may demonstrate whether the behaviors of AI systems are aligned, or in a broad sense desirable. Secondly, the debate may inform policy questions (e.g., what models should be deployed in which contexts?). Thirdly, our approach may complement engineering and design efforts: when capabilities can be precisely characterized, they can inform the development of models. Fourthly, applying our moral reasoning concepts in the new context of AI systems should improve our understanding of those concepts.

Figures & tables

Explore similar work

CardsList
  1. How do LLMs Evaluate Perceived Moral Agency? Investigating Moral Decision-Making in Human-Artificial Agents Interactions

    Sep 4, 2026Fernanda Mansilla, Aloysius Tok, Bahia Guellaï +2Moral Reasoning in Language ModelsHuman-AI Decision Making

  2. Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment

    Sep 4, 2026Arno Libert, Derck W. E. Prinzhorn, Daan R. HenselmansLLM Safety AlignmentBehavioral Consistency of LLM Agents

  3. Are LLMs Bad at Moral Reasoning?

    Jun 10, 2026Menghang Zhu, Seth LazarAutomatic Rubric GenerationLLM Evaluation