Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment

Arno Libert; Derck W. E. Prinzhorn; Daan R. Henselmans

cs.AISep 4, 2026

Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment

Authors: Arno Libert, Derck W. E. Prinzhorn, Daan R. Henselmans

Organizations: Aithos Research Foundation, Keizersgracht 62, Amsterdam, 1015 CS, the Netherlands.

Abstract

AI alignment requires AI systems to adhere to human norms, values, or intentions. Under value pluralism there is no correct target, but a shared prerequisite is that the system's behavior expresses a coherent policy: a mapping from situations to verdicts that is invariant while a situation's morally relevant features are preserved, and sensitive when they change. We introduce four structural conditions for such coherent policies: verdict stability, monotonicity, decisiveness, and Pareto viability. Together they measure a form of moral competence that is evaluable from behavior alone, without reference to a moral standard or expert baseline, forming a structural floor for alignment rather than a normative target. We demonstrate the methodology on three simulated deployments featuring LLM-based agents facing moral dilemmas. Evaluating nine frontier models under a factorial design of five paraphrases, five escalation levels, and three dominance conditions, we show no model expresses a coherent policy across the three deployments: surface-form perturbation alone produces verdict-rate shifts of up to $99$ percentage points at a single escalation level, and a model's success on one scenario does not predict its competence on another. This suggests LLM-based agents are not currently the kind of object to which alignment can meaningfully apply.

Figures & tables

Figure 1: An illustration of measurement limitations when models are evaluated on a single scenario variant without checks on perturbations. The left bar per model shows observed verdict rate at our canonical configuration, with error bars showing binomial SE at $n=100$ . The right bar per model shows the mean observed verdict rate across five paraphrases, with error bars showing observed standard deviation. Paraphrases preserve every morally relevant feature of the scenario. The canonical measurement frequently misrepresents model behavior and underestimates measurement error.

Figure 2: Structural component scores by model and scenario. Rows: nine frontier models. Columns within each scenario panel: $S,M,D,P$ . Major variation can be observed between models and results for individual scenarios. No model’s minimum score across components is $>0.36$ . Failures occur independently along test dimensions.

Model	$C_{\text{chem}}$	$C_{\text{fintech}}$	$C_{\text{smart}}$	avg $C$	max spread
Kimi K2.6	0.82	0.80	0.64	0.75	88.0pp
Qwen 3.6-Plus	0.98	0.35	0.87	0.73	55.0pp
Claude Sonnet 4.6	0.60	0.81	0.68	0.70	99.0pp
Llama 3.3	1.00	0.36	0.67	0.68	73.0pp
Mistral Large 2512	0.99	0.23	0.77	0.67	46.0pp
GLM 5.1	0.76	0.45	0.76	0.66	86.0pp

Table 1: Models ordered by aggregate competence. Shows

C

per scenario (geometric mean of

S

M

D

\max(0,P)

), average

C

across the three scenarios, and the worst per-level paraphrase spread for each model, i.e. the amount of percentage points by which the verdict rate was observed to change due to paraphrasing.

Figure 3: Marginal verdict rate $\bar{p}_{j}$ across escalation levels, one line per model, one panel per scenario. Each line averages across five paraphrases. Models with low $M$ oscillate over levels to an extent that exceeds the noise floor, rather than moving monotonically or staying constant. Models with low $D$ feature verdict rates close to $0.5$ , rather than being close to either $0$ or $1$ at each level.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Violation rates across sampling temperatures for the default Scheurer et al. (2024) scenario. Shaded regions denote $95\%$ confidence intervals from $N=300$ trials per cell. Claude, GPT-5, and GPT-OSS (overlapping at $0\%$ ) refuse categorically. DeepSeek and Qwen show order-of-magnitude increases with temperature; Mistral violates consistently.

Figure 5: Mean violation rate (top) and within-class spread (bottom) for each model across nine perturbation classes at $T=0.7$ . The rightmost column reports overall mean and spread across all perturbation types combined.

Figure 6: Mean Cramér’s $V$ across the four models with nonzero violation rates, by perturbation class. Dashed lines at $0.06$ , $0.17$ , and $0.29$ indicate conventional thresholds for weak, moderate, and strong effects respectively. Models with zero violations are excluded as the statistic is undefined.

LLM Safety Alignment

Large language model safety alignment trains and evaluates systems to follow human policies while resisting harmful requests and misuse.

See all 337 papers

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs
Oct 7, 2026Andrea Wynn, Harsh Satija, Nathan Baek +3Normative Conformity in Language Models Social Reasoning
Nathan · Johns Hopkins University · Vector Institute
Right or Wrong, Models Comply: Directional Blindness in LLM Moral Judgment
Jun 12, 2026Jihye Kim, Jeffrey FlaniganLLM Alignment Moral Reasoning in Language Models
University of California, Santa Cruz
Is LLM's Moral Decision Compositional?
May 29, 2026Weijia Zhang, Ruiqi Chen, Yunze Xiao +1LLM Evaluation Moral Reasoning in Language Models
University of Illinois Urbana-Champaign · University of Michigan · Carnegie Mellon University +1

Oct 7, 2026J/K move · Enter open · S save

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

Andrea Wynn, Harsh Satija, Nathan Baek, Anqi Liu+2

Nathan · Johns Hopkins University · Vector Institute

Normative Conformity in Language Models Social Reasoning

Open paper AskPDF arXiv

Human communities are governed by normative systems: shared standards that produce norms dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is a central alignment challenge, complicated by the fact that norms are vast in number, change quickly, and are often arbitrary (e.g., dress or language conventions). Thus, alignment requires normative competence: the ability to discern from interaction alone what norms a community enforces without relying on static pretrained knowledge. We introduce a multi-agent community debate setting, where access to debate is governed by synthetic norms, to study normative competence in isolation from pretraining exposure. We show that baseline LLM agents fail to learn norms even when doing so would improve their accuracy. We then experiment with various normative modules -- architectural components for norm inference -- finding that norm-following is highly sensitive to both the style of norm and the model powering the normative module, suggesting a lack of generalizability. Furthermore, when idiosyncratic, non-normative behaviors accompany the true norm, LLM agents exhibit an unselective attribution failure: they indiscriminately copy idiosyncratic noise alongside enforced rules, a pattern that persists even when imitating unnecessary behaviors is explicitly penalized. To the best of our knowledge, our work is the first to operationalize and evaluate normative competence in LLMs, demonstrating that current AI systems excel at behavioral mimicry but lack the capacity to discern socially enforced order.

Abstract

LLM Safety Alignment

Explore similar work

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

Right or Wrong, Models Comply: Directional Blindness in LLM Moral Judgment

Is LLM's Moral Decision Compositional?