Disambiguating Moral Circuits and Moral Features in Language Models: Moral content, normative judgment or cue words? Evidence from 14 models
Organizations: University of Luxembourg · Luxembourg Institute of Science and Technology (LIST)
Abstract
Interpretability methods can locate the attention heads and sparse autoencoder (SAE) features that change the answer a language model gives to a moral prompt, e.g., whether "Stealing medicine for a dying child is morally" continues with "justified" or "wrong". Such a component could respond to moral content, to the judgment of conduct by any norm, or to the word "morally" in the prompt. We built 227 minimal pairs of short phrases, validated by five annotators, whose two versions differ in a moral standard, in another kind of norm (prudence, etiquette, aesthetics or evidence) or in a descriptive quality, and showed them to 14 open-weight models (Llama, Gemma and Qwen, 1B to 14B, base and instruction-tuned), with and without the word "morally". In every model, the heads that change the moral answer belong to an evaluation circuit the models use for every kind of normative judgment we tested: these heads carry the same share of the effect for moral as for other normative pairs (pooled ratio 1.00, 97.5% confidence interval [0.92, 1.07], with the word; 0.90, [0.84, 0.96], without it), and the models need them as much for both. The models need these heads also when nothing in the prompt asks for a judgment, and on 400 scenarios of the ETHICS benchmark, where removing them takes away 37 to 39% of the difference between acceptable and unacceptable scenarios, against 1 to 3% for ten random heads. The SAE features we located respond mostly to the word "morally" rather than to what the phrase is about. The models represent the category of a pair (the standard the annotators assigned to it): it can be read from their hidden states, and, in the eight models we tested, removing the direction that carries it weakens the moral verdict by a few percent, and more than the other verdicts in seven of them. We release the pairs, the ratings, the code, the trained SAE dictionaries and the result files.
Figures & tables
| Category | Pair |
|---|---|
| Moral | Comforting a crying child at the playground / Mocking a crying child at the playground |
| Prudential | Inspecting the tent before the first camping trip / Forgetting the tent before the first camping trip |
| Etiquette | Greeting the host when you arrive at a party / Avoiding the host when you arrive at a party |
| Aesthetic | Trimming the hedges in front of a historic mansion / Hacking the hedges in front of a historic mansion |
| Epistemic | Verifying a surprising statistic before citing it / Trusting a surprising statistic before citing it |
| Descriptive | The harvest after a season of steady rain / The harvest after a season of severe drought |
| What we examined | What we found | What it means |
|---|---|---|
| The located heads | The same share of the effect for moral and for normative pairs (ratio 1.00 with the cue, 0.90 without), and about a fifth more for normative than for descriptive pairs about a thing (on descriptive pairs about a person: the share of normative pairs without the cue, a share halfway with it) | They belong to a general evaluation circuit that every standard uses, not to moral judgment in particular |
| Removing the located heads | A fifth to a quarter of the preference disappears, at least as much on normative as on moral pairs | The models need them for moral, other normative and descriptive pairs alike |
| The located heads in other prompts and on other text | With “ [stem] is” alone, with a plain question and with “socially” as the cue, they carry 4 to 5 times the share of ten random heads on moral and normative pairs, and removing them takes away 22 to 37 % of the preference; on ETHICS scenarios, 37 to 39 % of the difference between acceptable and unacceptable; in a choice between two options they exceed every random set in 6 and 7 of 14 models | The circuit does not depend on a prompt that asks for a verdict, on the word “morally” or on our pairs; a choice between two options needs it in some models only |
| Heads chosen on the new pairs | Heads chosen on moral pairs favour moral pairs no more than heads chosen on normative pairs favour normative pairs | Their moral selectivity does not exceed what a split between two kinds of norm shows |
| The located features | Active with “morally” (past the first layer also “ethically”) ending the prompt, much less dependent on the content (descriptive phrases reach 70 to 99 % of the activation of moral phrases); nearly silent without a cue or when “Morally speaking” opens the sentence, half as active with “socially”, a third with “logically” | They respond mostly to the word where they are read, most of all a moral one, and much less to moral content |
| Features found on the new pairs | More features separate moral from normative pairs than separate kinds of norm in 12 of 14 dictionaries with the cue, 9 with the cue at the start of the sentence, 3 without a cue, 2 with “socially” and none with “logically”; removed, they lower the moral preference by 5 to 10 % in some small models | With the moral cue, the dictionaries mark moral pairs as a class of their own, possibly in part because that cue suits them |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| All pairs | Same token count | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Head | Heads M C | All heads | Raw | Share [95 % CI ] | Rank | Share | ||
| Llama-3.2-1B | L7H8 | 97.7 % | 2.25 | 4.64 | 2.41 [1.69, 3.65] | 0.003 | 1 | 2.63 | 0.006 |
| Llama-3.2-1B- IT | L13H25 | 62.3 % | 0.92 | 6.63 | 6.71 [3.94, 12.29] | 0.003 | 1 | 5.86 | 0.005 |
| Llama-3.2-3B | L11H17 | 90.0 % | 1.60 | 5.12 | 3.43 [2.52, 4.73] | 0.004 | 1 | 4.32 | 0.013 |
| Llama-3.2-3B- IT | L11H17 | 79.2 % | 1.17 | 3.44 | 3.57 [2.46, 5.37] | 0.003 | 1 | 3.83 | 0.003 |
| Gemma-2B | L3H6 | 86.1 % | 1.51 | 2.98 | 1.31 [0.81, 2.23] | 0.249 | 7 | 1.80 | 0.030 |
| Base model | Base top head | Shared pairs | Pearson | Spearman | Share ratio base / IT | Rank in IT | Raw change |
|---|---|---|---|---|---|---|---|
| Llama-3.2-1B | L7H8 | 57 | 0.913 | 0.954 | 2.38 / 1.78 | 3 | % |
| Gemma-2B | L3H6 | 50 | 0.899 | 0.913 | 1.29 / 1.66 | 3 | % |
| Qwen2.5-3B | L20H12 | 54 | 0.868 | 0.747 | 3.78 / 2.32 | 2 | % |
| Llama-3.2-3B | L11H17 | 68 | 0.967 | 0.966 | 3.46 / 3.41 | 1 | % |
| Qwen2.5-7B | L0H24 | 66 | 0.940 | 0.943 | 2.25 / 1.25 | 6 | % |
| Llama-3.1-8B | L12H22 | 70 | 0.969 | 0.965 | 3.03 / 3.30 | 1 | % |
| Model | Site / | Width | Moral norm | ||
|---|---|---|---|---|---|
| Llama-3.2-1B | R7 / 64 | 16,384 | 62 | 172 | 32 |
| Llama-3.2-1B | R7 / 128 | 16,384 | 62 | 172 | 66 |
| Llama-3.2-1B- IT | R7 / 64 | 16,384 | 54 | 175 | 20 |
| Llama-3.2-1B- IT | R7 / 128 | 16,384 | 54 | 175 | 52 |
| Llama-3.2-1B | MLP 0 / 64 | 16,384 | 62 | 172 | 16 |
| Llama-3.2-3B | R11 / 64 | 24,576 | 68 | 190 | 56 |
| Set | Mean edit (tokens) | Pairs with no shared prefix | Same token count |
|---|---|---|---|
| Moral (73) | 4.2 | 1 % | 64 % |
| Norm, original 32 | 5.5 | 31 % | - |
| Norm, 189 added | 7.7 | 81 % | 15 % of all 221 |
| Neutral (105) | 3.4 | - | 38 % |
| Frame | Role | Prompt |
|---|---|---|
| A1 | primary, cue present | A sunny day at the beach is good. A flat tire on the highway is bad. Comforting a crying child at the playground is morally |
| B | primary, no cue | A sunny day at the beach is good. A flat tire on the highway is bad. Comforting a crying child at the playground is |
| A0 | secondary, the original frame, as in Liga and Yu,2025 | Comforting a crying child at the playground is morally |
| Interval of | Reading | |
|---|---|---|
| 1 | entirely above 1.5 | strong moral preference of the located heads |
| 2 | entirely inside 0.8 to 1.25 | no meaningful content specificity: the located heads serve normative evaluation in general (if the interval also excludes 1, this is said, and the difference is described as within the margin) |
| 3 | entirely above 1.0 | weak to moderate moral preference |
| 4 | entirely below 1.0 | the located heads carry a smaller share for moral than for norm content |
| 5 | anything else | inconclusive |
| Included pairs (moral/norm/descr.) | [95 % CI ]; | |||||
|---|---|---|---|---|---|---|
| Model | A1 | B | A0 | A1 | B | A0 |
| Llama-3.2-1B | 78/97/39 | 73/91/35 | 74/82/27 | 1.07 [0.99, 1.15]; 0.216 | 1.08 [0.99, 1.18]; 0.130 | 1.01 [0.94, 1.10]; 0.768 |
| Llama-3.2-1B- IT | 69/92/38 | 72/95/37 | 72/86/24 | 1.13 [1.01, 1.27]; 0.135 | 1.03 [0.91, 1.18]; 0.680 | 1.23 [1.12, 1.35]; 0.001 |
| Llama-3.2-3B | 79/104/40 | 78/100/40 | 78/90/25 | 0.93 [0.83, 1.04]; 0.304 | 0.88 [0.77, 1.00]; 0.123 | 1.11 [0.99, 1.24]; 0.154 |
| Llama-3.2-3B- IT | 72/103/39 | 69/103/39 | 77/95/30 | 0.97 [0.86, 1.09]; 0.657 | 0.81 [0.72, 0.91]; 0.009 | 1.03 [0.93, 1.13]; 0.768 |
| Gemma-2B | 74/99/39 | 72/95/40 | 75/93/34 | 0.93 [0.86, 1.00]; 0.135 | 0.91 [0.83, 0.99]; 0.063 | 1.01 [0.94, 1.09]; 0.828 |
| Profile | Top head of the base model | Split-half | ||
|---|---|---|---|---|
| Base model | base vs IT | in the IT top 10 (its rank) | base | IT |
| Llama-3.2-1B | 0.90 | no (12) | 0.99 | 0.98 |
| Llama-3.2-3B | 0.91 | yes (4) | 0.99 | 0.99 |
| Llama-3.1-8B | 0.95 | yes (4) | 0.98 | 0.99 |
| Gemma-2B | 0.89 | no (19) | 0.98 | 0.98 |
| Qwen2.5-3B | 0.88 | yes (1) | 0.99 | 0.98 |
| Analysis | A1 | B |
|---|---|---|
| Primary analysis: located heads, p2r, all positions, , threshold 0.02, retained pairs | 1.00 [0.92, 1.07] | 0.90 [0.84, 0.96] |
| Frame A0 in place of A1 | 1.12 [1.05, 1.18] | - |
| Direction r2p | 1.05 [1.00, 1.12] | 1.03 [0.95, 1.11] |
| Patching at the final position only | 1.04 [0.98, 1.10] | 0.86 [0.80, 0.91] |
| Score LD | 1.02 [0.97, 1.06] | 0.98 [0.93, 1.03] |
| Original score (10 + 10 words) | 1.00 [0.93, 1.06] | 0.91 [0.86, 0.96] |
| A1 | B | |||||
|---|---|---|---|---|---|---|
| Model | moral | norm | rank/20 (moral, norm) | moral | norm | rank/20 (moral, norm) |
| Llama-3.2-1B | 0.14 | 0.17 | 0, 0 | 0.29 | 0.30 | 0, 0 |
| Llama-3.2-1B- IT | -0.03 | -0.01 | 18, 15 | 0.07 | 0.10 | 0, 0 |
| Llama-3.2-3B | 0.31 | 0.44 | 0, 0 | 0.22 | 0.30 | 0, 0 |
| Llama-3.2-3B- IT | 0.08 | 0.16 | 0, 0 | 0.06 | 0.10 | 0, 0 |
| Gemma-2B | 0.04 | 0.03 | 6, 9 | 0.28 | 0.22 | 0, 0 |
| Multiple of the random share | Gap removed, % (random sets) | Above every random set | |||||||
| Prompt | M | N | D | M | N | D | M | N | D |
| Priming, “ [stem] is morally” (A1) | 6.00 | 5.92 | 4.86 | 28 (2) | 34 (3) | 28 (2) | 11 | 11 | 8 |
| Priming, “ [stem] is” (B) | 5.21 | 5.31 | 4.39 | 26 (2) | 30 (2) | 23 (1) | 9 | 11 | 9 |
| “ [stem] is morally” (A0) | 5.60 | 5.02 | 4.32 | not run | not run | ||||
| “ [stem] is socially” | 5.25 | 4.75 | 4.24 | 37 (4) | 35 (3) | 22 (3) | 14 | 12 | 9 |
| “ [stem] is” | 5.01 | 4.78 | 4.03 | 24 (3) | 22 (3) | 14 (2) | 13 | 12 | 8 |
| Capture | H5: descr./moral | H6 counts: moral vs norm the three splits among norms | ||||
|---|---|---|---|---|---|---|
| Model | Site / | A1 | B | A1 [95 % CI ] | A1 | B |
| Llama-3.2-1B | R7 / 64 | -0.23* | 0.11* | 0.70 [0.68, 0.72] f | 14.3 0.3, 4.0, 6.0 | 6.2 8.8, 2.3, 9.7 |
| Llama-3.2-1B | R7 / 128 | -0.18* | 0.21* | 0.73 [0.70, 0.75] f | 23.4 3.0, 11.6, 6.0 | 16.4 8.2, 13.0, 18.6 |
| Llama-3.2-1B- IT | R7 / 64 | -0.25* | 0.14* | 0.79 [0.77, 0.81] f | 15.0 0.0, 0.1, 0.7 | 14.3 6.8, 3.3, 13.3 |
| Llama-3.2-1B- IT | R7 / 128 | -0.21* | 0.24* | 0.78 [0.76, 0.80] f | 30.9 1.5, 6.8, 5.8 | 16.9 8.7, 15.6, 20.0 |
| Llama-3.2-1B | MLP 0 / 64 | -1.54* | 0.37* | 0.99 [0.99, 1.00] f | 0.2 0.0, 0.9, 0.5 | 0.1 0.7, 0.1, 0.8 |
| Frame | End of the prompt | Located features | Moral pairs stand apart (H6) |
|---|---|---|---|
| Moral cue (A1) | [stem] is morally | 1 | 12 of 14 |
| Synonym | [stem] is ethically | 0.79 (0.64 to 0.90) | 7 of 14 |
| Moral cue before the stem | Morally speaking, [stem] is | 0.18 (0.10 to 0.49) | 9 of 14 |
| Social cue | [stem] is socially | 0.50 (0.33 to 0.54) | 2 of 14 |
| Logical cue | [stem] is logically | 0.31 (0.18 to 0.47) | 0 of 14 |
| No cue (B) | [stem] is | 0.16 (0.03 to 0.33) | 3 of 14 |
| With five example sentences | Without examples | |||||
|---|---|---|---|---|---|---|
| Model | AUC | accuracy | covered | AUC | accuracy | covered |
| Llama-3.2-1B | 0.97 | 0.84 | 9 % | 0.93 | 0.54 | 6 % |
| Llama-3.2-1B- IT | 0.95 | 0.85 | 8 % | 0.91 | 0.66 | 3 % |
| Llama-3.2-3B | 0.98 | 0.90 | 9 % | 0.91 | 0.60 | 5 % |
| Llama-3.2-3B- IT | 0.97 | 0.87 | 9 % | 0.91 | 0.71 | 3 % |
| Gemma-2B | 0.96 | 0.85 | 9 % | 0.90 | 0.61 | 5 % |
| Moral-versus-normative lists | Stance lists | |||||||||||
| with cue | without cue | with cue | without cue | |||||||||
| Model | M | N | D | M | N | D | M | N | D | M | N | D |
| Llama-3.2-1B | † | ∗ | ∗ | |||||||||
| Llama-3.2-1B-Instruct | ∗∗ | |||||||||||
| Llama-3.2-3B | † | ∗∗ | ||||||||||
| Llama-3.2-3B-Instruct | ∗∗ | ∗ | ||||||||||
| Category | Rated | Retained | Subtype confirmed | Minimum | Rejections by criterion |
|---|---|---|---|---|---|
| Moral | 89 | 80 | - | 60 | class 8 |
| Prudential | 41 | 37 | 15 | 10 confirmed | class 4 |
| Etiquette | 24 | 22 | 18 | 10 confirmed | class 2; moral rating 1 |
| Aesthetic | 24 | 24 | 22 | 10 confirmed | - |
| Epistemic | 24 | 24 | 24 | 10 confirmed | - |
| All normative | 113 | 107 | - | 60 |
Moral Reasoning in Language Models
Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.
See all 126 papers