cs.CLMar 5, 2026

The Fragility Of Moral Judgment In Large Language Models

Authors: Tom van Nuenen, Pratik S. Sachdeva

Organizations: D-Lab, University of California, Berkeley, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

People increasingly use large language models (LLMs) for everyday moral and interpersonal guidance, yet these systems cannot interrogate missing context and judge dilemmas as presented. We introduce a perturbation framework for testing the stability and manipulability of LLM moral judgments while holding the underlying moral conflict constant. Using 2,939 dilemmas from r/AmItheAsshole (January-March 2025), we generate three families of content perturbations: surface edits (lexical/structural noise), point-of-view shifts (voice and stance neutralization), and persuasion cues (self-positioning, social proof, pattern admissions, victim framing). We also vary the evaluation protocol (output ordering, instruction placement, and unstructured prompting). We evaluated all variants with four models (GPT-4.1, Claude 3.7 Sonnet, DeepSeek V3, Qwen2.5-72B) (N=129,156 judgments). Surface perturbations produce low flip rates (7.5%), largely within the self-consistency noise floor (4-13%), whereas point-of-view shifts induce substantially higher instability (24.3%). A large subset of dilemmas (37.9%) is robust to surface noise yet flips under perspective changes, indicating that models condition on narrative voice as a pragmatic cue. Instability concentrates in morally ambiguous cases; scenarios where no party is assigned blame are most susceptible. Persuasion perturbations yield systematic directional shifts. Protocol choices dominate all other factors: agreement between structured protocols is only 67.6% (kappa=0.55), and only 35.7% of model-scenario units match across all three protocols. These results show that LLM moral judgments are co-produced by narrative form and task scaffolding, raising reproducibility and equity concerns when outcomes depend on presentation skill rather than moral substance.

Figures & tables

PerturbationDescription
Baseline
noneOriginal AITA dilemma with no modifications (control condition)
Surface Perturbations
remove sentenceRemove one sentence ( ≤\leq 10% of tokens)
change trivial detailChange one morally irrelevant detail (location, weather, time of day)
add extraneous detailInsert one morally irrelevant sentence ( ≤\leq 20 tokens)
Table 1. Overview of Content Perturbations and Evaluation Protocols
Figure 1. Baseline Uncertainty Predicts Perturbation Instability. (a) Relationship between baseline normalized entropy (NE) and perturbation flip rate across four models. NE is computed from runs 2–15 using a split-sample approach; flip rates are computed against held-out run 1 to avoid leakage. Lines show quasibinomial GLM fits (logit link). (b) Flip rates by baseline verdict category, aggregated across all models. Two-panel figure. Panel A shows scatter plot with GLM regression lines for four models (GPT-4.1, Claude 3.7, DeepSeek, Qwen 2.5) plotting baseline normalized entropy (x-axis, 0 to 0.65) against perturbation flip rate (y-axis, 0 to 0.55). All lines show positive slopes with similar trends. Panel B shows bar chart of flip rates by baseline verdict category, with nuanced verdicts (AAF, NAF) showing approximately 45-50% flip rates compared to 15-20% for clear verdicts (OAF, SAF).
Figure 2. Verdict Instability and Asymmetric Blame Attribution by Model. (a) Flip rates by base verdict category and perturbation family. (b) Net blame direction by model and perturbation type. Values represent (flips toward blaming narrator−flips toward exonerating narrator)/(total directional flips)(\text{flips toward blaming narrator}-\text{flips toward exonerating narrator})/(\text{total directional flips}) , ranging from −1-1 (all flips exonerate narrator) to +1+1 (all flips blame narrator), with 00 indicating balanced effects. Red cells indicate perturbations that shift blame toward the narrator; blue cells indicate shifts toward exonerating the narrator. Two-panel figure. Panel A shows grouped bar chart of flip rates by base verdict across three perturbation families. Panel B shows heatmap of net blame direction with models as columns and perturbation types as rows, colored from blue (exonerates narrator) through white (balanced) to red (blames narrator), with category labels on the left side grouping perturbations into Push Self At Fault, Push Other At Fault, Point-of-view, and Surface.
Figure 3. Epistemic Stance and Cross-Model Agreement Under Verification. (a) Change in net epistemic stance (boosters −- hedges per 100 words) between baseline and perturbed explanations. Negative values indicate more hedged, tentative language; positive values indicate more confident, direct language. Error bars show 95% confidence intervals. (b) Cross-model agreement by scenario-level verification intensity on reasoning traces. Scenarios are binned by total verification count across all models and protocols. Two-panel figure. Panel A shows horizontal bar chart of epistemic stance change for 11 perturbation types, grouped by category (Persuasion, Surface, Point-of-view), with Push-Self-At-Fault perturbations showing negative values and Point-of-view perturbations showing positive values. Panel B shows bar chart of cross-model agreement percentage across four verification levels (None, Low, Medium, High), with a clear declining trend from 82% to 18% as verification intensity increases.
Figure 4. Protocol Instability and Distributed Blame Verdict Fate. (a) Verdict flip rates under three protocol perturbations (Explanation First, System Prompt, Unstructured) compared to the main study’s verdict-first protocol, grouped by base verdict category. (b) Fate of distributed blame verdicts ( All At Fault or No One At Fault ) under content and protocol perturbations. Stacked bars show the percentage of verdicts that were retained versus shifted to clear blame attribution ( Other At Fault or Self At Fault ). Two-panel figure. Panel A shows grouped bar chart comparing flip rates across five verdict categories for three protocols, with Unstructured showing highest instability and a distinct Unclear category. Panel B shows two horizontal stacked bars comparing content and protocol perturbations, illustrating that protocol changes cause more nuanced verdicts to resolve to clear blame.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Perturbation TypeExample Change
Perspective Shift Perturbations (Flip Rate: 21–27%)
thirdperson (27.3% flip rate)Base: “I dated my ex for around 7 months… he gave me his old gaming chair cuz I needed one… after we broke up he said keep the chair” Perturbed: “The main person dated their partner for around 7 months… their partner gave the main person an old gaming chair… the partner told the main person to keep the chair”
firstperson_atfault (21.3% flip rate)Base: “AITA for saying spousal love is conditional?” Perturbed: “Am I at fault for saying spousal love is conditional?”
Push-YTA Perturbations (Flip Rate: 10–12%)
push_yta_pattern_admission (12.1% flip rate)Added: “This isn’t the first time I’ve reacted this way when my expectations aren’t perfectly met” after narrator describes rejecting husband’s gift choice.
push_yta_social_proof (11.6% flip rate)Added: “Most people I talked to say I overreacted” after narrator describes confronting spouse about meeting an old friend.
Appendix
Table 3. Complete perturbation examples across all categories. Each example shows a scenario where the perturbation caused 3–4 models (out of 4) to flip their verdict. Flip rates are calculated across all scenarios. Even minimal changes (degree << 0.05) consistently cause verdict instability.
ConditionN FlipsPreservedReversed→\rightarrow Blame→\rightarrow ExonNet
Baseline Variance
Entropy (m=15)7855.1%44.9%1520−5-5
Content Perturbations
All content15,01642.0%58.0%5,4263,283+2,143+2,143
Push-YTA3,77532.9%67.1%1,931601+1,330+1,330
Push-NTA3,43131.7%68.3%1,404938+466+466
Appendix
Table 4. OP Blame Status Transitions Across Experimental Conditions
Dimension% Agreeκ\kappaα\alphaR1 Y%R2 Y%
Core conflict preserved100.0—1.00100100
Target feature manipulated98.00.00−- 0.0110098
Unintended changes75.0−- 0.04−- 0.07198
Semantic coherence96.00.320.319995
Overall pass94.0−- 0.02−- 0.039995
Average92.60.070.24——
Appendix
Table 5. Inter-rater reliability for perturbation validation ratings ( n=100n=100 ).
ComparisonCohen’s κ\kappa% Agreement
Human–Human
Rater 1 vs. Rater 20.58867.8%
Human–LLM
Rater 1 vs. GPT-4.1-mini0.35647.5%
Rater 2 vs. GPT-4.1-mini0.46555.9%
Appendix
Table 6. Inter-Rater Reliability for Verdict Classification
ModelnAll-3 AgreeAvg PairwiseType
o3-mini1,16278.4%85.1%Reasoning
GPT-4.11,20058.9%71.1%Standard
Δ\Delta +19.5pp*+14.0pp* 
Claude 3.7 (thinking)1,09269.2%78.8%Reasoning
Claude 3.7 (standard)1,20067.8%77.6%Standard
Δ\Delta +1.4pp+1.2pp 
Appendix
Table 7. Cross-protocol agreement for reasoning vs. standard models. *OpenAI and Alibaba comparisons are confounded (different architectures or model sizes); Anthropic and DeepSeek are clean same-base comparisons.
Labelκ\kappa% AgreementN
Early commitment (Y/N)0.7187%99
Verification present (Y/N)0.6589%99
Verification quality (S/W)0.5879%74 ∗
∗ Conditional on verification present.
Appendix
Table 8. Human–LLM agreement on reasoning-quality annotations (N=99).
 Early commitAny verifStrong verifWeak verif
Overall (N=9,819)58.3%85.6%28.4%57.1%
Claude-thinking51.2%85.1%50.0%35.1%
DeepSeek-R155.1%85.5%21.5%64.1%
QwQ-32B68.4%86.0%14.0%72.0%
Appendix
Table 9. Reasoning-quality patterns (LLM-annotated).
More in this topic

Moral Reasoning in Language Models

Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.

See all 114 papers

Explore similar work

CardsList
Jun 10, 2026

Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs

As LLMs increasingly serve in advisory and deliberative roles, users rely on them for non-verifiable reasoning in domains lacking objective ground truths. However, traditional evaluations of LLM reasoning focus almost exclusively on fact-based domains, such as mathematics and science, leaving uncertainty over whether and to what degree models can handle ambiguous, subjective, or value-laden problems over time. To address this concern, we propose moral reasoning as a paradigmatic subdomain of non-verifiable reasoning. We define moral robustness as a model's capacity to exhibit sound moral reasoning across time and contexts, and we introduce a scalable, adversarial, multi-turn evaluation framework to empirically measure this capability. We simulate 48,000 user-agent moral deliberations across four frontier LLMs, varying premise relevance, premise order, conversation duration, and the user's stated moral view. We find that models successfully ignore morally-irrelevant distractors, but shift their reasoning by up to 6.5%, on average, towards the user's stated preferred moral view, and varying their reasoning depending on factors such as order (altering moral judgments by order in 13-22% of the cases) and duration (altering moral judgments between single-turn and multi-turn in 10-24% of the cases). Our analysis indicates that models tailor not just their final verdicts but their underlying justifications to align with a user's moral viewpoint - a failure mode we characterize as moral deliberative sycophancy.

Elizaveta Tennant, Benjamin Henke, Anita Keshmirian +5

Google DeepMind · Imperial College London · Institute of Philosophy, School of Advanced Study, University of London +2

Moral Reasoning in Language ModelsLanguage Model Robustness
AskPDF
Mar 16, 2026

Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability

Large language models (LLMs) increasingly participate in morally sensitive decision-making, yet how they organize ethical frameworks across reasoning steps remains underexplored. We introduce moral reasoning trajectories, sequences of ethical framework invocations across intermediate reasoning steps, and analyze their dynamics across six models and three benchmarks. We find that moral reasoning involves systematic multi-framework deliberation: 55.4--57.7% of consecutive steps involve framework switches, and only 16.4--17.8% of trajectories remain framework-consistent. Unstable trajectories remain 1.29 times more susceptible to persuasive attacks (p=0.015). At the representation level, linear probes localize framework-specific encoding to model-specific layers (layer 63/81 for Llama-3.3-70B; layer 17/81 for Qwen2.5-72B), achieving 16.8--22.2% lower KL divergence than the step-prior baseline. Activation steering applied during generation moves the framework-consistency--accuracy relationship, widening it for Qwen2.5-72B and erasing it for Llama-3.3-70B, and a probe-space layer sweep bounds the attainable drift reduction at 6.7--8.9%. We further propose a Moral Representation Consistency (MRC) metric whose underlying framework attributions are validated by human annotators (mean cosine similarity = 0.859), and we report what an automated coherence rater does and does not establish about it.

Fan Huang, Haewoon Kwak, Jisun An

Indiana University Bloomington / United States

Moral Reasoning in Language ModelsLLM InterpretabilityRepresentation ProbingMoral Reasoning
AskPDF
Mar 24, 2026

On the Context Sensitivity of LLM Moral Judgment

A human's moral decision depends heavily on the context. Yet research on LLM morality has largely studied fixed scenarios. We address this gap by introducing Contextual MoralChoice, a dataset of moral dilemmas with systematic contextual variations known from moral psychology to shift human judgment: consequentialist, emotional, and relational. Evaluating 22 LLMs, we find that nearly all models are context-sensitive, shifting their judgments towards rule-violating behavior. Comparing with a human survey, we find that models and humans are most triggered by different contextual variations, and that a model aligned with human judgments in the base case is not necessarily aligned in its contextual sensitivity. This raises the question of controlling contextual sensitivity, which we address with an activation steering approach that can reliably increase or decrease a model's contextual sensitivity. Code and data: https://github.com/adrian-sauter/contextual_moralchoice.

Adrian Sauter, Mona Schirmer

Helmholtz Munich · University of Amsterdam

Moral Reasoning in Language ModelsMoral Reasoning
AskPDF