cs.AINov 6, 2026

Do large language models rank moral values? A transitivity test of moral competence

Authors: Katelyn A. Wasson, Kiwon Song, James M. Jennings, Brittney Currie, Konstantina Sokratous, Clintin P. Stober

Organizations: Department of Psychological Sciences, University of Missouri. · College of Engineering, Oregon State University. · Department of Philosophy, The City College of New York, City University of New York. · MU Institute for Data Science and Informatics, University of Missouri.

Abstract

LLMs are increasingly used to support decisions that require moral consideration. While LLMs perform well on morality benchmarks from psychology and philosophy, producing appropriate outputs on these benchmarks (moral performance) does not establish that an LLM weighs competing moral values at stake in a decision (moral competence). We test a minimal, necessary condition for such weighing: that an ordinal ranking can represent LLM moral choices. Such a ranking requires transitivity. To assess transitivity, we presented LLMs with pairwise trade-offs among moral value violations in vignette choice sets and extracted resulting choice probabilities from first-token distributions. Tested against three stochastic transitive choice models, most choice sets admitted weak or unstable orderings. Aggregating an LLM's responses, a common practice in LLM moral evaluation, produced transitive structure that matched no individual choice set. Evaluations of AI morality should establish that choices are structured before asking whether they are aligned.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Is LLM's Moral Decision Compositional?

    May 29, 2026Weijia Zhang, Ruiqi Chen, Yunze Xiao +1LLM EvaluationMoral Reasoning in Language Models

  2. Are LLMs Bad at Moral Reasoning?

    Jun 10, 2026Menghang Zhu, Seth LazarAutomatic Rubric GenerationLLM Evaluation

  3. On the Context Sensitivity of LLM Moral Judgment

    Mar 24, 2026Adrian Sauter, Mona SchirmerMoral Reasoning in Language ModelsMoral Reasoning