The Fragility Of Moral Judgment In Large Language Models

Tom van Nuenen; Pratik S. Sachdeva

cs.CLMar 5, 2026

The Fragility Of Moral Judgment In Large Language Models

Authors: Tom van Nuenen, Pratik S. Sachdeva

Organizations: D-Lab, University of California, Berkeley, USA

Abstract

People increasingly use large language models (LLMs) for everyday moral and interpersonal guidance, yet these systems cannot interrogate missing context and judge dilemmas as presented. We introduce a perturbation framework for testing the stability and manipulability of LLM moral judgments while holding the underlying moral conflict constant. Using 2,939 dilemmas from r/AmItheAsshole (January-March 2025), we generate three families of content perturbations: surface edits (lexical/structural noise), point-of-view shifts (voice and stance neutralization), and persuasion cues (self-positioning, social proof, pattern admissions, victim framing). We also vary the evaluation protocol (output ordering, instruction placement, and unstructured prompting). We evaluated all variants with four models (GPT-4.1, Claude 3.7 Sonnet, DeepSeek V3, Qwen2.5-72B) (N=129,156 judgments). Surface perturbations produce low flip rates (7.5%), largely within the self-consistency noise floor (4-13%), whereas point-of-view shifts induce substantially higher instability (24.3%). A large subset of dilemmas (37.9%) is robust to surface noise yet flips under perspective changes, indicating that models condition on narrative voice as a pragmatic cue. Instability concentrates in morally ambiguous cases; scenarios where no party is assigned blame are most susceptible. Persuasion perturbations yield systematic directional shifts. Protocol choices dominate all other factors: agreement between structured protocols is only 67.6% (kappa=0.55), and only 35.7% of model-scenario units match across all three protocols. These results show that LLM moral judgments are co-produced by narrative form and task scaffolding, raising reproducibility and equity concerns when outcomes depend on presentation skill rather than moral substance.

Figures & tables

Perturbation	Description
Baseline
none	Original AITA dilemma with no modifications (control condition)
Surface Perturbations
remove sentence	Remove one sentence ( $\leq$ 10% of tokens)
change trivial detail	Change one morally irrelevant detail (location, weather, time of day)
add extraneous detail	Insert one morally irrelevant sentence ( $\leq$ 20 tokens)

Table 1. Overview of Content Perturbations and Evaluation Protocols

Figure 1. Baseline Uncertainty Predicts Perturbation Instability. (a) Relationship between baseline normalized entropy (NE) and perturbation flip rate across four models. NE is computed from runs 2–15 using a split-sample approach; flip rates are computed against held-out run 1 to avoid leakage. Lines show quasibinomial GLM fits (logit link). (b) Flip rates by baseline verdict category, aggregated across all models. Two-panel figure. Panel A shows scatter plot with GLM regression lines for four models (GPT-4.1, Claude 3.7, DeepSeek, Qwen 2.5) plotting baseline normalized entropy (x-axis, 0 to 0.65) against perturbation flip rate (y-axis, 0 to 0.55). All lines show positive slopes with similar trends. Panel B shows bar chart of flip rates by baseline verdict category, with nuanced verdicts (AAF, NAF) showing approximately 45-50% flip rates compared to 15-20% for clear verdicts (OAF, SAF).

Figure 2. Verdict Instability and Asymmetric Blame Attribution by Model. (a) Flip rates by base verdict category and perturbation family. (b) Net blame direction by model and perturbation type. Values represent $(\text{flips toward blaming narrator}-\text{flips toward exonerating narrator})/(\text{total directional flips})$ , ranging from $-1$ (all flips exonerate narrator) to $+1$ (all flips blame narrator), with $0$ indicating balanced effects. Red cells indicate perturbations that shift blame toward the narrator; blue cells indicate shifts toward exonerating the narrator. Two-panel figure. Panel A shows grouped bar chart of flip rates by base verdict across three perturbation families. Panel B shows heatmap of net blame direction with models as columns and perturbation types as rows, colored from blue (exonerates narrator) through white (balanced) to red (blames narrator), with category labels on the left side grouping perturbations into Push Self At Fault, Push Other At Fault, Point-of-view, and Surface.

Figure 3. Epistemic Stance and Cross-Model Agreement Under Verification. (a) Change in net epistemic stance (boosters

-

hedges per 100 words) between baseline and perturbed explanations. Negative values indicate more hedged, tentative language; positive values indicate more confident, direct language. Error bars show 95% confidence intervals. (b) Cross-model agreement by scenario-level verification intensity on reasoning traces. Scenarios are binned by total verification count across all models and protocols. Two-panel figure. Panel A shows horizontal bar chart of epistemic stance change for 11 perturbation types, grouped by category (Persuasion, Surface, Point-of-view), with Push-Self-At-Fault perturbations showing negative values and Point-of-view perturbations showing positive values. Panel B shows bar chart of cross-model agreement percentage across four verification levels (None, Low, Medium, High), with a clear declining trend from 82% to 18% as verification intensity increases.

Figure 4. Protocol Instability and Distributed Blame Verdict Fate. (a) Verdict flip rates under three protocol perturbations (Explanation First, System Prompt, Unstructured) compared to the main study’s verdict-first protocol, grouped by base verdict category. (b) Fate of distributed blame verdicts ( All At Fault or No One At Fault ) under content and protocol perturbations. Stacked bars show the percentage of verdicts that were retained versus shifted to clear blame attribution ( Other At Fault or Self At Fault ). Two-panel figure. Panel A shows grouped bar chart comparing flip rates across five verdict categories for three protocols, with Unstructured showing highest instability and a distinct Unclear category. Panel B shows two horizontal stacked bars comparing content and protocol perturbations, illustrating that protocol changes cause more nuanced verdicts to resolve to clear blame.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Perspective Shift Perturbations (Flip Rate: 21–27%)
Perturbation Type	Example Change
thirdperson (27.3% flip rate)	Base: “I dated my ex for around 7 months… he gave me his old gaming chair cuz I needed one… after we broke up he said keep the chair” Perturbed: “The main person dated their partner for around 7 months… their partner gave the main person an old gaming chair… the partner told the main person to keep the chair”
firstperson_atfault (21.3% flip rate)	Base: “AITA for saying spousal love is conditional?” Perturbed: “Am I at fault for saying spousal love is conditional?”
Push-YTA Perturbations (Flip Rate: 10–12%)
push_yta_pattern_admission (12.1% flip rate)	Added: “This isn’t the first time I’ve reacted this way when my expectations aren’t perfectly met” after narrator describes rejecting husband’s gift choice.
push_yta_social_proof (11.6% flip rate)	Added: “Most people I talked to say I overreacted” after narrator describes confronting spouse about meeting an old friend.

Appendix

Table 3. Complete perturbation examples across all categories. Each example shows a scenario where the perturbation caused 3–4 models (out of 4) to flip their verdict. Flip rates are calculated across all scenarios. Even minimal changes (degree

<

0.05) consistently cause verdict instability.

Condition	N Flips	Preserved	Reversed	$\rightarrow$ Blame	$\rightarrow$ Exon	Net
Baseline Variance
Entropy (m=15)	78	55.1%	44.9%	15	20	$-5$
Content Perturbations
All content	15,016	42.0%	58.0%	5,426	3,283	$+2,143$
Push-YTA	3,775	32.9%	67.1%	1,931	601	$+1,330$
Push-NTA	3,431	31.7%	68.3%	1,404	938	$+466$

Appendix

Table 4. OP Blame Status Transitions Across Experimental Conditions

Dimension	% Agree	$\kappa$	$\alpha$	R1 Y%	R2 Y%
Core conflict preserved	100.0	—	1.00	100	100
Target feature manipulated	98.0	0.00	$-$ 0.01	100	98
Unintended changes	75.0	$-$ 0.04	$-$ 0.07	19	8
Semantic coherence	96.0	0.32	0.31	99	95
Overall pass	94.0	$-$ 0.02	$-$ 0.03	99	95
Average	92.6	0.07	0.24	—	—

Appendix

Table 5. Inter-rater reliability for perturbation validation ratings (

n=100

Human–Human
Comparison	Cohen’s $\kappa$	% Agreement
Rater 1 vs. Rater 2	0.588	67.8%
Human–LLM
Rater 1 vs. GPT-4.1-mini	0.356	47.5%
Rater 2 vs. GPT-4.1-mini	0.465	55.9%

Appendix

Table 6. Inter-Rater Reliability for Verdict Classification

Model	n	All-3 Agree	Avg Pairwise	Type
o3-mini	1,162	78.4%	85.1%	Reasoning
GPT-4.1	1,200	58.9%	71.1%	Standard
$\Delta$		+19.5pp*	+14.0pp*
Claude 3.7 (thinking)	1,092	69.2%	78.8%	Reasoning
Claude 3.7 (standard)	1,200	67.8%	77.6%	Standard
$\Delta$		+1.4pp	+1.2pp

Appendix

Table 7. Cross-protocol agreement for reasoning vs. standard models. *OpenAI and Alibaba comparisons are confounded (different architectures or model sizes); Anthropic and DeepSeek are clean same-base comparisons.

Label	$\kappa$	% Agreement	N
Early commitment (Y/N)	0.71	87%	99
Verification present (Y/N)	0.65	89%	99
Verification quality (S/W)	0.58	79%	74 ∗
∗ Conditional on verification present.

Appendix

Table 8. Human–LLM agreement on reasoning-quality annotations (N=99).

	Early commit	Any verif	Strong verif	Weak verif
Overall (N=9,819)	58.3%	85.6%	28.4%	57.1%
Claude-thinking	51.2%	85.1%	50.0%	35.1%
DeepSeek-R1	55.1%	85.5%	21.5%	64.1%
QwQ-32B	68.4%	86.0%	14.0%	72.0%

Appendix

Table 9. Reasoning-quality patterns (LLM-annotated).

Moral Reasoning in Language Models

Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.

See all 114 papers

Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs
Jun 10, 2026Elizaveta Tennant, Benjamin Henke, Anita Keshmirian +5Moral Reasoning in Language Models Language Model Robustness
Google DeepMind · Imperial College London · Institute of Philosophy, School of Advanced Study, University of London +2
Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability
Mar 16, 2026Fan Huang, Haewoon Kwak, Jisun AnMoral Reasoning in Language Models LLM Interpretability
Indiana University Bloomington / United States
On the Context Sensitivity of LLM Moral Judgment
Mar 24, 2026Adrian Sauter, Mona SchirmerMoral Reasoning in Language Models Moral Reasoning
Helmholtz Munich · University of Amsterdam

Jun 10, 2026J/K move · Enter open · S save

Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs

Elizaveta Tennant, Benjamin Henke, Anita Keshmirian, Murray Shanahan+4

Google DeepMind · Imperial College London · Institute of Philosophy, School of Advanced Study, University of London+2

Moral Reasoning in Language Models Language Model Robustness

Open paper AskPDF arXiv

As LLMs increasingly serve in advisory and deliberative roles, users rely on them for non-verifiable reasoning in domains lacking objective ground truths. However, traditional evaluations of LLM reasoning focus almost exclusively on fact-based domains, such as mathematics and science, leaving uncertainty over whether and to what degree models can handle ambiguous, subjective, or value-laden problems over time. To address this concern, we propose moral reasoning as a paradigmatic subdomain of non-verifiable reasoning. We define moral robustness as a model's capacity to exhibit sound moral reasoning across time and contexts, and we introduce a scalable, adversarial, multi-turn evaluation framework to empirically measure this capability. We simulate 48,000 user-agent moral deliberations across four frontier LLMs, varying premise relevance, premise order, conversation duration, and the user's stated moral view. We find that models successfully ignore morally-irrelevant distractors, but shift their reasoning by up to 6.5%, on average, towards the user's stated preferred moral view, and varying their reasoning depending on factors such as order (altering moral judgments by order in 13-22% of the cases) and duration (altering moral judgments between single-turn and multi-turn in 10-24% of the cases). Our analysis indicates that models tailor not just their final verdicts but their underlying justifications to align with a user's moral viewpoint - a failure mode we characterize as moral deliberative sycophancy.

Abstract

Moral Reasoning in Language Models

Explore similar work

Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs

Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability

On the Context Sensitivity of LLM Moral Judgment