Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable

Marius De Kuthy Meurers; Aaron Parisi; Bernd Bohnet

cs.CLOct 21, 2026

Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable

Authors: Marius De Kuthy Meurers, Aaron Parisi, Bernd Bohnet

Organizations: Google DeepMind

Abstract

LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well. We train a reinforcement learning policy initialized from a state-of-the-art language model in a self-judged tournament with masked thinking: the generated arguments are judged pairwise on argument quality by the policy itself. As an empirical basis, we extract authentic dilemmas from classic novels, using a logic solver to verify that each constitutes a conflict. We show that our judge rates argument quality like human annotators, and the policy then adopts the language register that it rewards. Testing on the external benchmarks MoralChoice, ELEPHANT, and MORABLES, we find that the policy becomes more decisive and invariant to the order in which options are presented, but sycophancy and reframing remain a challenge to validity. Two ablation studies with synthetic dilemmas and unmasked thinking match the reordering gains, but show a weaker register shift and higher sycophancy. We conclude that consistency is not validity: consistency under reordering can be optimized without improving the moral judgment it is meant to signal.

Figures & tables

Panel of Figure 1 — Figure 1: 4B Base, 4B after GRPO (step 48), and 31B judges on IBM-ArgQ-30k across the human preference gap $\Delta$ (bands: 95% Wilson CIs). Top: resolved accuracy. Bottom: presentation-flip rate (dashed), explicit tie rate (dotted), and pair density $p(\Delta)$ (histogram).

Panel of Figure 2 — Figure 2: Resolved accuracy and abstention rate of the 4B Base and 31B judges per aspect and overall on IBM-ArgQ-30k (95% Wilson CIs). Abstention: share of all pairs, stacked as presentation flips and ties.

Panel of Figure 3 — Figure 3: Paired change in replicated register features, epoch 2 vs. epoch 1 (ratio, log axis; 95% dilemma-clustered bootstrap CIs). Faded: Unmasked and Synthetic ablations.

Panel of Figure 4 — Figure 4: Training reward per within-group SD of replicated features and the scholar persona ( $x$ ) against paired change, epoch 2 vs. epoch 1, in GRPO ( $y$ , in SD). Shaded quadrants: change in the rewarded direction. Filled: $q<.05$ (BH).

Panel of Figure 5 — Figure 5: Held-out benchmark change, GRPO (step 48) vs. Base (percentage points; item-paired 95% bootstrap CIs). Crosses: targeted 4-shot ICL vs. Base (no matched MORABLES run; arrowheads: values past the axis). Squares: the same GRPO $-$ Base difference re-scored by a frozen 31B judge (hollow: CI includes zero). $\uparrow$ / $\downarrow$ : higher/lower is better. ‡ Scored by the evaluated model (for ICL, by Base).

Panel of Figure 6 — Figure 6: Held-out benchmark change, Unmasked and Synthetic vs. GRPO (percentage points; item-paired 95% bootstrap CIs; rows as in Figure 5 ). Squares: the same Unmasked $-$ GRPO difference re-scored by a frozen 31B judge (Synthetic not re-scored). Hollow: CI includes zero.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Author	Source Work	Extracted
Hugo	Les Misérables (#135)	88
Austen	Pride and Prejudice (#1342)	39
Melville	Moby-Dick (#2701)	33
Dostoevsky	Crime and Punishment (#2554)	25
Total		185

Appendix

Table 1: Curriculum dilemma pool ( LitPool ). Top: dilemmas extracted per PG19 source work (Gutenberg ID). Bottom: most frequent competing value-concepts among the 185 extracted dilemmas, of which 156 pass the Z3 gate and are trained on.

$Z{=}0$ failure modes
Z3 gate outcome	Dilemmas
Generated ( $N$ )	130
$Z{=}2$ (exclusive)	101 (77.7%)
$Z{=}1$ (co-satisfiable)	20 (15.4%)
$Z{=}0$ (degenerate)	9 (6.9%)
One duty UNSAT	8

Appendix

Table 2: Outcomes of the Z3 gate on the first generation batch of the synthetic dilemmas the Synthetic run trains on.

Z{=}2

: the duties are mutually exclusive (a genuine dilemma).

Z{=}1

: the duties are co-satisfiable.

Z{=}0

: the encoding is degenerate.

Source	$n$	Vendi $\uparrow$			Eff. rank $\uparrow$			NN $\downarrow$	Conc.
		bge	uc	gem	bge	uc	gem	bge
Hugo	88	3.79	7.02	2.75	22.40	23.12	21.97	0.833	16
Dostoevsky	25	3.89	6.67	2.71	22.02	22.96	21.22	0.839	20
Austen	39	3.51	6.29	2.39	22.53	23.02	21.95	0.839	24
Melville	33	4.41	6.62	3.10	22.71	23.10	22.31	0.795	25
LitPool	185	4.39	6.97	3.27	22.63	23.19	22.20	0.806	25

Appendix

Table 3: Intrinsic diversity of the four-author literature pool ( LitPool ), its source works, and synthetic dilemmas from the generator of the Synthetic run, on three embedders (bge, unixcoder, gemini-embedding) at

n{=}25

per source (seed 42).

n

: dilemmas available to this analysis. NN: mean nearest-neighbor cosine. Conc.: distinct value-concepts. Vendi is embedder-scaled: compare within a column.

Model and hardware
Component	Setting
Policy	Gemma-4 4B, instruction-tuned
Hardware	32 accelerator chips, FSDP $16\times$ TP $2$
Sequence length	1,024 prompt + 3,072 generation tokens
Data and schedule
Training data	156 Z3-verified LitPool dilemmas, 2 epochs

Appendix

Table 4: Training configuration of the GRPO run.

Logic
Dimension	Question in the judge prompt
Cogency	Are the premises acceptable, relevant to the conclusion, and sufficient to draw it?
Local acceptability	Are the premises rationally worthy of belief?
Local relevance	Do the premises actually bear on the conclusion?
Local sufficiency	Do the premises together give enough support?
Rhetoric

Appendix

Table 5: The 15 argument-quality dimensions of Wachsmuth et al. (2017) as phrased in the judge prompt (Appendix G.3 ), grouped by aspect; sub-dimensions are indented under their aspect. The judge reasons over all 15 but emits only four verdicts: logic, rhetoric, dialectic, and overall quality.

Panel of Figure 7 — Figure 7: Kendall $\tau_{b}$ agreement on Dagstuhl-15512 between three human experts ( $A_{1}$ – $A_{3}$ ) and the 31B judge, per quality dimension (diagonal omitted). Outlined: the 31B judge’s row and column.

	Aspect	Same	Flip	Tie	Other
4B	Logic	82.4	15.0	1.8	0.9
	Rhetoric	93.7	5.2	0.0	1.1
	Dialectic	90.0	9.1	0.8	0.1
31B	Logic	98.6	1.1	0.4	0.0
	Rhetoric	88.7	9.0	1.1	1.2
	Dialectic	96.7	2.7	0.6	0.0

Appendix

Table 6: Aspect verdicts against the reward on IBM-ArgQ-30k, over the pairs with a decisive overall verdict (%). Same: identical verdict. Mismatches: presentation flip (Flip), tie (Tie), or the other argument (Other).

Feature group	$n$	Core
Allocation : response words, thinking words, thinking share, thinking–response overlap	4	4
Formatting : list items, numbered-list items, bullet-list items, horizontal rules, headings, bold spans	6	5
Discourse and argument : causal connectives, forward indicators, backward indicators, rebuttal indicators, backward/forward ratio	5	2
Syntax (full prose sentences) : sentence length, max dependency depth, mean dependency depth, clausal embedding depth, subordination index, coordinators, fronted adverbial clauses	7	6
Conditionals : concessive conditionals, counterfactual sentences, conditional sentences	3	2
Lexical semantics : nominalizations, verb agentivity, necessity modals	3	3

Appendix

Table 7: Register features compared between epoch 2 and epoch 1, by group.

n

: features in the group. Core: same-direction change in all three trained runs (

q<.05

; thinking-channel features: GRPO and Synthetic only). Biber dimensions: composites of the Biber features, not in the total. Self-similarity and length variation: no per-completion reward attribution, not in Figure 4 .

Panel of Figure 8 — Figure 8: Paired change (%) in GRPO on the same prompt, epoch 2 vs. epoch 1, by assigned framework (95% dilemma-clustered bootstrap CIs). Rows as in Figure 3 , plus two content rows. Hollow cells: CI includes zero (12 of 110).

MoralChoice
		ICL ( $k{=}4$ )
	Base	Rand.	Targ.	GRPO
Fragility, high amb. $\downarrow$	14.3	4.3	2.1	6.6
Ambiguity recognized $\uparrow$	16.0	30.3	22.1	19.1
Fragility, low amb. $\downarrow$	1.2	4.4	0.9	0.7
ELEPHANT

Appendix

Table 8: In-context baselines on the matched items of Figure 5 (%). Base without demonstrations, with

k{=}4

random in-domain demonstrations (Rand.), or with

k{=}4

targeted demonstrations (Targ.; anti-sycophancy for ELEPHANT, cross-framing for MoralChoice), and GRPO. Each arm is scored on the items it shares with Base; MORABLES is omitted because its ICL sweep used another protocol. ‡ Scored by the evaluated model (for ICL, by Base). Last block: the same Base and GRPO completions re-scored by a frozen 31B judge (items scored for both; ICL not re-scored, –).

Panel of Figure 9 — Figure 9: Answer (solid) and thinking (dashed) words across training for GRPO and Unmasked, as change from the first four steps net of prompt fixed effects (bands: 95% dilemma-clustered bootstrap CIs; 1 epoch = 24 steps).

Panel of Figure 10 — Figure 10: Mean thinking words per completion on held-out benchmarks for Base, GRPO, and Unmasked (95% bootstrap CIs).

	F ${=}0$	F ${=}1$	$\Delta$	OR [95% CI]
Base	368.8	364.3	$-$ 4.5	0.98 [0.92, 1.04]
GRPO	319.2	406.7	+87.5	1.48 [1.39, 1.57]
Unmasked	462.0	461.6	$-$ 0.4	0.99 [0.86, 1.13]

Appendix

Table 9: Thinking words per completion on ELEPHANT, split by framing acceptance (F

{=}1

: the answer accepts the user’s framing), on the 2,782 completions per model of the items GRPO completed (each item in its original and flipped version).

\Delta

: F

{=}1

minus F

{=}0

. OR: odds ratio of framing acceptance per 100 thinking words from a logistic regression (95% Wald CI).

Moral Reasoning in Language Models

Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.

See all 114 papers

On the Context Sensitivity of LLM Moral Judgment
Mar 24, 2026Adrian Sauter, Mona SchirmerMoral Reasoning in Language Models Moral Reasoning
Helmholtz Munich · University of Amsterdam
The Fragility Of Moral Judgment In Large Language Models
Mar 5, 2026Tom van Nuenen, Pratik S. SachdevaMoral Reasoning in Language Models Prompt Sensitivity
D-Lab, University of California, Berkeley, USA
Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs
Jun 10, 2026Elizaveta Tennant, Benjamin Henke, Anita Keshmirian +5Moral Reasoning in Language Models Language Model Robustness
Google DeepMind · Imperial College London · Institute of Philosophy, School of Advanced Study, University of London +2

Mar 24, 2026J/K move · Enter open · S save

On the Context Sensitivity of LLM Moral Judgment

Adrian Sauter, Mona Schirmer

Helmholtz Munich · University of Amsterdam

Moral Reasoning in Language Models Moral Reasoning

Open paper AskPDF arXiv

A human's moral decision depends heavily on the context. Yet research on LLM morality has largely studied fixed scenarios. We address this gap by introducing Contextual MoralChoice, a dataset of moral dilemmas with systematic contextual variations known from moral psychology to shift human judgment: consequentialist, emotional, and relational. Evaluating 22 LLMs, we find that nearly all models are context-sensitive, shifting their judgments towards rule-violating behavior. Comparing with a human survey, we find that models and humans are most triggered by different contextual variations, and that a model aligned with human judgments in the base case is not necessarily aligned in its contextual sensitivity. This raises the question of controlling contextual sensitivity, which we address with an activation steering approach that can reliably increase or decrease a model's contextual sensitivity. Code and data: https://github.com/adrian-sauter/contextual_moralchoice.

Abstract

Moral Reasoning in Language Models

Explore similar work

On the Context Sensitivity of LLM Moral Judgment

The Fragility Of Moral Judgment In Large Language Models

Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs