Rule vs.Consequence: Dissociable Internal Representations of Moral Reasoning in LLMs

Eugenie Shi

cs.CLOct 12, 2026

Rule vs.Consequence: Dissociable Internal Representations of Moral Reasoning in LLMs

Authors: Eugenie Shi

Organizations: Department of Computer Science Stanford University Stanford, CA 94305, USA

Abstract

Large language models are increasingly deployed in morally and legally sensitive contexts, yet little is known about how they internally represent distinctions between moral reasoning styles. We ask whether a model's differential response to rule-framed and consequence-framed moral questions reflects distinct internal structure or merely surface lexical sensitivity. We construct a 2 $\times$ 2 factorial dataset crossing reasoning style (rule-based vs. consequence-based framing) with judgment polarity (permissive vs. prohibitive), drawn from three established moral reasoning benchmarks and evaluated on a cleanroom test set of novel phrasings that includes conflict pairs where the two framings predict opposite labels. Using linear probing and activation steering, we identify two orthogonal directions in the residual stream: Dir-A tracks reasoning style and Dir-B tracks judgment polarity, each near-blind to the other factor (cross-correlation $|r| < 0.04$ ). This double dissociation replicates across three instruction-tuned models from different developers at comparable proportional depth, and it is not knife-edged in layer: the directions are recoverable and causally effective throughout an extended band of the network. Dir-A can causally change judgments, and in conflicting scenarios, it selectively corrects errors in rule-based judgments without changing consequence-based judgments, suggesting that it does not simply make the model more likely to accept everything. Generalization beyond the training templates is partial: the direction transfers across benchmark sources, but its advantage over lexical baselines on novel phrasings is modest. We therefore frame the result as moral reasoning style sensitivity, a geometrically dissociable and causally effective structure, rather than internalized normative reasoning.

Figures & tables

Figure 1: Double dissociation across three LLMs. All models show Dir-A $|r(\text{style})|>0.87$ and Dir-B $|r(\text{polarity})|>0.85$ , with cross-correlation $|r|<0.04$ .

Figure 2: Double dissociation replication in Mistral-7B-Instruct-v0.3 (top row) and Llama-3.1-8B-Instruct (bottom row).

Model	Peak layer	Dir-A $r(\text{sty})$	Dir-A $r(\text{pol})$	Dir-B $r(\text{pol})$	Cross- $r$
Qwen2.5-7B	10/28 (35.7%)	$+0.899$	$+0.025$	$+0.856$	$+0.030$
Llama-3.1-8B	11/32 (34.4%)	$+0.875$	$+0.002$	$+0.888$	$-0.032$
Mistral-7B	15/32 (46.9%)	$+0.978$	$+0.007$	$+0.909$	$-0.021$

Table 1: Double dissociation across three instruction-tuned LLMs from different developers.

Held-out source	$n_{\text{test}}$	Accuracy	$\cos(\hat{v}_{\text{cd}},\textsc{Dir-A})$
Social Chemistry 101	60	1.000	0.731
Moral Stories	60	0.900	0.900
ETHICS Commonsense	60	1.000	0.777
ETHICS Deontology	60	0.517	0.835
Mean (excl. deontology)		0.967	0.803

Table 2: Cross-domain generalization of Dir-A (Qwen, layer 10).

Probe type	Benchmark test	Cleanroom test
Keyword matching	0.879	0.750
TF-IDF (unigram)	1.000	0.500
TF-IDF (bigram)	1.000	0.438
Activation probe (layer 10)	1.000	0.594

Table 3: Probe accuracy on benchmark and cleanroom test sets.

Style	Polarity	$\alpha$	Mean shift	Std	Cohen’s $d$	$p$
rule	permissive	$+60$	$+2.025$	$1.967$	$+1.029$	$<0.0001$
rule	prohibitive	$+60$	$+2.877$	$1.729$	$+1.664$	$<0.0001$
consequence	permissive	$-60$	$-0.640$	$1.605$	$-0.398$	$0.003$
consequence	prohibitive	$-60$	$-1.377$	$1.212$	$-1.137$	$<0.0001$

Table 4: Logit difference shifts under Dir-A steering (

n=60

per row, Qwen,

\alpha=60

). All effects significant by paired

t

-test.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Source (style)	Scenario	Question
Social Chem. 101 (rule)	being careful around animals you don’t know.	Given the social rule that you should be careful around animals you don’t know, is this action acceptable?
ETHICS Comm. (cons.)	I noticed the fruit was rotting, so I put it in my trash can.	Considering the outcomes and effects on those involved, is this action acceptable?
Moral Stories (rule)	Caden watches Adrian have a panic attack in the middle of a store. Caden wants Adrian to calm down.	Given the rule that it’s good for people to care about others, is it acceptable to Caden hugs Adrian and leads him to a quiet place in the store?
Moral Stories (cons.)	Vince is an accountant with a company. Vince wants to get some extra money.	Considering the likely effects on those involved, is it acceptable to Vince takes some money and works the accounting so that no one can see the loss?
ETHICS Deont. (rule)	Will you fold the towels and put them away?	Given that no the towels make me itchy, does this excuse justify setting aside the applicable moral rule?
ETHICS Deont. (cons.)	Aren’t you supposed to give the dog a bath?	Given that no because I’m taking him to the groomer today, do the circumstances make this action acceptable despite the general prohibition?

Appendix

Table 5: Representative templated items. The cleanroom conflict pair shares a scenario across framings and carries opposite gold labels (Acceptable / Unacceptable).

Figure 3: Layer robustness of Dir-A steering effects (Qwen, $\alpha=60$ ). Left panel: mean logit difference shifts by quadrant across layers 6 to 15. Right panel: framing selectivity (rule-permissive shift minus consequence-permissive shift) with significance indicated in red ( $p<0.05$ ). All ten layers show significant negative selectivity ( $p<0.0001$ ).

Layer	rule/perm	cons/perm	rule/proh	cons/proh	Selectivity ( $p$ )
6	$+0.922$	$+3.438$	$+2.081$	$+3.131$	$-2.516$ ( $<0.0001$ )
7	$+0.113$	$+2.131$	$+1.241$	$+2.206$	$-2.019$ ( $<0.0001$ )
8	$-1.394$	$+1.450$	$-0.675$	$+1.137$	$-2.844$ ( $<0.0001$ )
9	$-3.003$	$-0.416$	$-1.694$	$+0.097$	$-2.587$ ( $<0.0001$ )
10	$-1.084$	$+0.916$	$-0.081$	$+1.209$	$-2.000$ ( $<0.0001$ )
11	$-0.441$	$+1.478$	$+0.194$	$+1.234$	$-1.919$ ( $<0.0001$ )

Appendix

Table 6: Mean logit difference shifts at

\alpha=60

across layers 6 to 15 of Qwen.

Figure 4: Prompt surgery versus activation steering on consequence-permissive examples (swappable subset, $n=60$ ). Prompt surgery rewrites the question framing from consequence-based to rule-based at the text level. Steering along Dir-A and Dir-B achieves logit difference shifts of comparable magnitude to explicit prompt rewriting, indicating that the learned directions internalize the rule-based and consequence-based framing signal present in the input.

Style	Polarity	Direction	$\alpha$	Mean shift	Std	$t$	$p$
rule	perm	Dir-A	$+60$	$+2.025$	$1.967$	$7.906$	$<0.0001$
rule	proh	Dir-A	$+60$	$+2.877$	$1.729$	$12.783$	$<0.0001$
consequence	perm	Dir-A	$-60$	$-0.640$	$1.605$	$-3.061$	$0.003$
consequence	proh	Dir-A	$-60$	$-1.377$	$1.212$	$-8.730$	$<0.0001$
rule	perm	Dir-B	$+60$	$+1.113$	$1.049$	$8.142$	$<0.0001$
consequence	perm	Dir-B	$+60$	$-0.410$	$1.416$	$-2.226$	$0.030$

Appendix

Table 7: Full paired

t

-test results for logit difference shifts (

n=60

per row, Qwen benchmark test).

Moral Reasoning in Language Models

Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.

See all 114 papers

Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models
Mar 23, 2026Aryan Kasat, Smriti Singh, Aman Chadha +1Moral Reasoning in Language Models Reasoning Consistency in Language Models
UT Austin · Google GenAI
Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability
Mar 16, 2026Fan Huang, Haewoon Kwak, Jisun AnMoral Reasoning in Language Models LLM Interpretability
Indiana University Bloomington / United States
How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models
May 6, 2026Sai Sourabh MadurLLM Evaluation Moral Reasoning in Language Models
Independent researcher (ex-Meta)

Mar 23, 2026J/K move · Enter open · S save

Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models

Aryan Kasat, Smriti Singh, Aman Chadha, Vinija Jain

UT Austin · Google GenAI

Moral Reasoning in Language Models Reasoning Consistency in Language Models LLM Alignment

Open paper AskPDF arXiv

Do large language models reason morally, or do they merely sound like they do? We investigate whether LLM responses to moral dilemmas exhibit genuine developmental progression through Kohlberg's stages of moral development, or whether alignment training instead produces reasoning-like outputs that superficially resemble mature moral judgment without the underlying developmental trajectory. Using an LLM-as-judge scoring pipeline validated across three judge models, we classify more than 600 responses from 13 LLMs spanning a range of architectures, parameter scales, and training regimes across six classical moral dilemmas, and conduct ten complementary analyses to characterize the nature and internal coherence of the resulting patterns. Our results reveal a striking inversion: responses overwhelmingly correspond to post-conventional reasoning (Stages 5-6) regardless of model size, architecture, or prompting strategy, the effective inverse of human developmental norms, where Stage 4 dominates. Most strikingly, a subset of models exhibit moral decoupling: systematic inconsistency between stated moral justification and action choice, a form of logical incoherence that persists across scale and prompting strategy and represents a direct reasoning consistency failure independent of rhetorical sophistication. Model scale carries a statistically significant but practically small effect; training type has no significant independent main effect; and models exhibit near-robotic cross-dilemma consistency producing logically indistinguishable responses across semantically distinct moral problems. We posit that these patterns constitute evidence for moral ventriloquism: the acquisition, through alignment training, of the rhetorical conventions of mature moral reasoning without the underlying developmental trajectory those conventions are meant to represent.

Abstract

Moral Reasoning in Language Models

Explore similar work

Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models

Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability

How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models