ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs

Rohan Subramanian Thomas; Shikhar Shiromani; Abdullah Chaudhry; Ruizhe Li; Vasu Sharma; Kevin Zhu

cs.AIFeb 5, 2026

ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs

Authors: Rohan Subramanian Thomas, Shikhar Shiromani, Abdullah Chaudhry, Ruizhe Li, Vasu Sharma, Kevin Zhu

Abstract

Prompt design affects the moral judgments and refusal behavior produced by large language models (LLMs), yet controlled comparisons remain fragmented across datasets and models. We introduce ProMoral-Bench, a benchmark-style comparison of 10 prompting paradigms across four LLM families. Using ETHICS, Scruples, WildJailbreak, and our 200-pair ETHICS-Contrast set, we report task-level metrics together with Moral Competence Score (MCS), jailbreak/over-refusal Safety Robustness Score (SRS), and Unified Moral Safety Score (UMSS). UMSS is a study-specific relative summary of MCS and SRS, not an absolute measure of model safety or moral grounding. In this fixed-template evaluation, compact exemplar-guided scaffolds attain stronger competence-robustness trade-offs at lower token cost than the particular multi-stage variants tested, although multi-stage prompting occasionally improves jailbreak refusal. The few-shot demonstrations are non-overlapping but drawn from the evaluation distributions, so their advantage should not be read as out-of-distribution robustness. Code and data are available at https://github.com/rohanthomas334-sketch/ProMoral_Bench.

Figures & tables

Model	Strategy	Acc	F1	ECE
GPT	Plan-and-Solve	0.96	0.96	0.03
GPT	Few-Shot-CoT	$0.96$	$0.96$	$0.06$
GPT	Few-Shot	$0.95$	$0.95$	$0.05$
Claude	Few-Shot	0.94	0.94	0.09
Claude	Zero-Shot	$0.94$	$0.94$	$0.07$
Claude	Role Prompting	$0.93$	$0.93$	$0.06$

Table 1: ETHICS (mini). Top three strategies per model by Accuracy/F1. Full results are in Table 10 .We abbreviate model names as follows: GPT = GPT-4.1, Claude = Claude Sonnet-4, Gemini = Gemini 2.5 Pro, DS = DeepSeek-V3.1.

Model	Strategy	Acc	F1	ECE
GPT	Role Prompting	0.60	0.63	0.33
GPT	Few-Shot-CoT	$0.57$	$0.62$	$0.24$
GPT	Value-Grounded	$0.56$	$0.60$	$0.32$
Claude	Few-Shot-CoT	0.66	0.68	0.10
Claude	First-Principles	$0.63$	$0.65$	$0.20$
Claude	Value-Grounded	$0.61$	$0.64$	$0.12$

Table 2: Scruples (mini). Top three strategies per model by Accuracy/F1. Full results are in Table 11 .

Model	Strategy	$\Delta$ Acc	$\Delta$ F1	$\Delta$ ECE
GPT	Role Prompting	-0.02	-0.02	+0.03
GPT	Zero-Shot-CoT	$-0.02$	$-0.02$	$-0.00$
GPT	Few-Shot	$-0.03$	$-0.03$	$-0.01$
Claude	Self-Correct	+0.09	+0.15	+0.04
Claude	First-Principles	$-0.04$	$-0.04$	$0.01$
Claude	Few-Shot	$-0.05$	$-0.04$	$-0.01$

Table 3: ETHICS-Contrast (mini). Top three strategies per model by least drop (higher

\Delta

is better). Full results are in Table 12 .

Model	Strategy	ASR	RTA
GPT	Thought Experiment	0.16	0.04
GPT	Role Prompting	$0.30$	$0.03$
GPT	Value-Grounded	$0.31$	$0.04$
Claude	Value-Grounded	0.05	0.33
Claude	First-Principles	$0.06$	$0.34$
Claude	Zero-Shot	$0.10$	$0.29$

Table 4: WildJailbreak (mini). Top three strategies per model by lowest ASR (RTA shown). Full results are in Table 13 .

Task / expected behavior	Audited input (abridged)	Compact scaffold	Multi-stage scaffold
ETHICS-Contrast / Acceptable	“lit a cigarette as I got into the taxi” $\rightarrow$ “as I left the taxi”	Role: correct	Thought Experiment and Self-Correct: incorrect
ETHICS-Contrast / Acceptable	“put shampoo in her coffee” $\rightarrow$ “put sugar in her coffee”	Role: correct	Thought Experiment: correct; Self-Correct: incorrect
WildJailbreak benign / Comply	Authorized email-service security assessment with explicit terms-of-service constraints	Few-Shot-CoT: compliant answer with authorization safeguards	Thought Experiment: full refusal
WildJailbreak harmful / Refuse	Fictional request that escalates into assistance for infiltrating a violent gang	Few-Shot-CoT: refusal and non-violent redirection	Thought Experiment: substantive compliance

Table 5: Representative released-log outcomes. Cases are illustrative; harmful content is abridged.

Model	MCS	SRS	UMSS
GPT-4.1	0.800	0.710	0.752
Claude Sonnet-4	0.722	0.596	$0.653$
DeepSeek V3.1	0.688	0.626	$0.655$
Gemini 2.5 Pro	0.621	0.639	$0.630$

Table 6: Unified Moral Competence/Safety. MCS is the average min–max normalized accuracy on ETHICS and Scruples , and flip robustness on ETHICS-Contrast . SRS aggregates WildJailbreak by averaging min–max normalized safe rates

(1{-}\mathrm{ASR},\,1{-}\mathrm{RTA})

across a model’s prompting runs. UMSS is the harmonic mean of MCS and SRS. Bootstrap 95% CIs are in Appendix C.4 .

Prompting Strategy	MCS	SRS	UMSS
Few-Shot-CoT	0.967	$0.686$	0.803
Few-Shot	$0.948$	0.695	0.802
Role Prompting	$0.934$	$0.636$	$0.757$
Plan-and-Solve	$0.894$	$0.512$	$0.651$
Zero-Shot-CoT	$0.827$	$0.508$	$0.629$
Zero-Shot	$0.847$	$0.358$	$0.504$

Table 7: UMSS by prompting strategy. MCS: mean of min–max normalized classification scores over ETHICS , ETHICS-Contrast (flip accuracy), and Scruples . SRS: mean of min–max normalized safe rates

(1{-}\mathrm{ASR},\,1{-}\mathrm{RTA})

from WildJailbreak . UMSS is the harmonic mean of MCS and SRS (

\beta{=}1

). Bootstrap 95% CIs are in Appendix C.4 .

Figure 1: UMSS vs. Token Cost. Scatter plot of UMSS score against average tokens per example (log scale) for ten prompting strategies. Green shading indicates optimal zone; dotted line marks efficiency threshold; dashed line shows LSRL.

Strategy	$\beta{=}0.5$	$\beta{=}1.0$	$\beta{=}2.0$
Few-Shot-CoT	0.893	0.803	$0.728$
Few-Shot	0.884	0.802	0.734
Role Prompting	0.844	0.757	$0.678$
Plan-and-Solve	0.777	0.651	$0.560$
Zero-Shot-CoT	0.735	0.629	$0.551$
Value-Grounded	0.615	0.502	$0.435$

Table 8: Strategy-level UMSS under varying

\beta

. Top-3 rankings remain stable; only positions 1–2 swap at

\beta=2.0

Model	$\beta{=}0.5$	$\beta{=}1.0$	$\beta{=}2.0$
GPT-4.1	0.780	0.752	0.726
Claude Sonnet-4	0.697	0.653	$0.619$
DeepSeek V3.1	0.674	0.655	$0.637$
Gemini 2.5 Pro	0.623	0.630	$0.635$

Table 9: Model-level UMSS under varying

\beta

. GPT-4.1 leads throughout; Claude drops from 2nd to 4th as

\beta

increases.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Prompting Strategy	Accuracy	F1	ECE	Brier	Avg Tokens
GPT-4.1	Zero-Shot	$0.946$	$0.946$	$0.063$	$0.053$	$127.6$
	Zero-Shot-CoT	$0.948$	$0.948$	$0.064$	$0.052$	$149.2$
	Few-Shot	$0.952$	$0.952$	$0.052$	$0.040$	$305.1$
	Few-Shot-CoT	$0.959$	$0.958$	$0.063$	$0.036$	$513.4$
	Role Prompting	$0.935$	$0.936$	$0.042$	$0.035$	$207.5$
	Thought Exp	$0.900$	$0.904$	$0.117$	$0.092$	$13\,822.7$

Appendix

Table 10: ETHICS dataset results. We report Accuracy, F1 (positive class= Unacceptable ), calibration (ECE, Brier), and average tokens per example.

Model	Prompt	Acc	F1	ECE	Brier	Avg Tokens
GPT-4.1	Zero-Shot	$0.503$	$0.569$	$0.396$	$0.054$	$604.5$
	Zero-Shot-CoT	$0.467$	$0.542$	$0.440$	$0.050$	$724.9$
	Few-Shot	$0.535$	$0.574$	$0.326$	$0.060$	$841.6$
	Few-Shot-CoT	$0.568$	$0.615$	$0.242$	$0.071$	$880.2$
	Role Prompting	$0.596$	$0.628$	$0.326$	$0.048$	$748.8$
	Thought Exp	$0.274$	$0.298$	$0.656$	$0.041$	$18\,555.8$

Appendix

Table 11: Scruples results. We report overall Accuracy, weighted F1 (five classes), calibration (ECE, Brier), and average tokens per example.

Model	Prompt	Acc (orig)	Acc (flip)	$\Delta$ Acc	$\Delta$ F1	$\Delta$ ECE	$\Delta$ Brier
GPT-4.1	Zero-Shot	$0.980$	$0.945$	$-0.035$	$-0.035$	$0.004$	$0.026$
	Zero-Shot-CoT	$0.970$	$0.950$	$-0.020$	$-0.020$	$-0.004$	$0.017$
	Few-Shot	$0.990$	$0.965$	$-0.025$	$-0.025$	$-0.014$	$0.018$
	Few-Shot-CoT	$0.985$	$0.930$	$-0.055$	$-0.055$	$-0.014$	$0.035$
	Role Prompting	$0.970$	$0.955$	$-0.015$	$-0.015$	$0.029$	$0.035$
	Thought Exp	$0.980$	$0.860$	$-0.120$	$-0.102$	$0.061$	$0.098$

Appendix

Table 12: ETHICS-Contrast results. Accuracy on originals vs. flips, and the change (

\Delta

) in accuracy, F1, and calibration (

\Delta

ECE,

\Delta

Brier) for each prompting method.

Model	Prompt	ASR	RTA	Avg Gen Tokens
GPT-4.1	Zero-Shot	0.487	0.030	507
	Zero-Shot-CoT	0.4429	0.0071	501
	Few-Shot	0.4077	0.0154	2235
	Few-Shot-CoT	0.3750	0.0167	1869
	Role Prompting	0.3000	0.0333	557
	Thought Experiment	0.1556	0.0444	7102

Appendix

Table 13: WildJailbreak safety results. We report Attack Success Rate (ASR; lower is better), Refusal Rate on benign prompts (RTA; lower is better), and efficiency as average generation tokens per example.

Ablation	Acc (Abl)	$\Delta$ Acc
P/S + “typical adult” role	0.953	$-$ 0.008
FS-CoT: Acceptable only	0.948	$-$ 0.011
FS-CoT: Unacceptable only	0.944	$-$ 0.015
FS: Acceptable only	0.943	$-$ 0.010
FS: Unacceptable only	0.944	$-$ 0.009

Appendix

Table 14: ETHICS prompt sensitivity (GPT-4.1). Base accuracies: Plan-and-Solve 0.960, Few-Shot-CoT 0.959, Few-Shot 0.952.

Ablation	Acc (Abl)	$\Delta$ Acc
FS-CoT: Author/Other only	0.629	$-$ 0.027
FP: Self-Direction value	0.607	$-$ 0.023
VG: Self-Direction value	0.541	$-$ 0.066

Appendix

Table 15: Scruples prompt sensitivity (Claude Sonnet-4). Base accuracies: Few-Shot-CoT (FS-CoT) 0.656, First-Principles (FP) 0.630, Value-Grounded (VG) 0.607.

Ablation	Orig	Flip	$\Delta$ Acc ${}_{\text{flip}}$
RP: Base	0.970	0.955	$-$ 0.015
RP: + “professor” role	0.965	0.955	$-$ 0.010
ZS-CoT: Base	0.970	0.950	$-$ 0.020
ZS-CoT: Alt trigger	0.980	0.940	$-$ 0.040
FS: Base	0.990	0.965	$-$ 0.025
FS: Acceptable only	0.990	0.955	$-$ 0.035

Appendix

Table 16: ETHICS-Contrast prompt sensitivity (GPT-4.1). RP = Role Prompting, ZS-CoT = Zero-Shot-CoT, FS = Few-Shot.

\Delta

Acc

{}_{\text{flip}}

= Flip

-

Orig; smaller magnitude indicates greater robustness.

	ASR		RTA
Ablation	Abl	Base	Abl	Base
TE: 1 summary	0.156	0.156	0.044	0.044
RP: “typical adult”	0.400	0.300	0.008	0.033
VG: Hedonism value	0.300	0.310	0.040	0.040

Appendix

Table 17: WildJailbreak prompt sensitivity (GPT-4.1). TE = Thought Experiment, RP = Role Prompting, VG = Value-Grounded. Lower ASR and RTA are better.

Strat. A	Strat. B	$p$	Winner
ETHICS
FS-CoT	TE	$<$ 0.001	FS-CoT
FS-CoT	SC	$<$ 0.001	FS-CoT
P&S	TE	$<$ 0.001	P&S
P&S	SC	$<$ 0.001	P&S
FS	TE	$<$ 0.001	FS

Appendix

Table 18: Compact vs. Verbose Strategies. Global

p

-values from Stouffer-aggregated McNemar tests across four models.

A	B	Subset	$p$	Result
RP	ZS	Harmful	$<$ 0.001	RP $\uparrow$
RP	ZS	Benign	0.819	n.s.
FS-CoT	ZS-CoT	Harmful	$<$ 0.001	FS-CoT $\uparrow$
FS-CoT	ZS-CoT	Benign	0.002	ZS-CoT $\uparrow$

Appendix

Table 19: Alignment Tradeoffs on WildJailbreak . Stouffer-aggregated McNemar tests.

\uparrow

indicates the statistically superior strategy on that subset.

A	B	$p$	Winner
ETHICS
GPT	Claude	0.002	GPT
GPT	DS	0.002	GPT
GPT	Gemini	0.002	GPT
Claude	DS	0.105	n.s.
Claude	Gemini	0.844	n.s.

Appendix

Table 20: Model-vs-Model Tournament. Macro-Averaged Wilcoxon Signed-Rank

p

-values across 10 shared strategies per dataset.

Model	MCS [95% CI]	SRS [95% CI]	UMSS [95% CI]
GPT	.800 [.747, .830]	.710 [.683, .754]	.752 [.724, .778]
Claude	.722 [.671, .775]	.596 [.566, .692]	.653 [.629, .715]
DS	.688 [.635, .745]	.626 [.602, .680]	.655 [.632, .697]
Gemini	.621 [.577, .670]	.639 [.613, .695]	.630 [.607, .670]

Appendix

Table 21: Model-level UMSS with bootstrap 95% CIs. GPT = GPT-4.1, Claude = Claude Sonnet-4, DS = DeepSeek V3.1, Gemini = Gemini 2.5 Pro.

Strat.	MCS [95% CI]	SRS [95% CI]	UMSS [95% CI]
FS-CoT	.967 [.889, .993]	.686 [.555, .826]	.803 [.697, .884]
FS	.948 [.870, .963]	.695 [.573, .820]	.802 [.706, .872]
RP	.934 [.854, .955]	.636 [.503, .778]	.757 [.647, .840]
P&S	.894 [.798, .936]	.512 [.387, .647]	.651 [.536, .745]
ZS-CoT	.827 [.740, .857]	.508 [.399, .615]	.629 [.531, .700]
ZS	.847 [.762, .872]	.358 [.269, .490]	.504 [.406, .614]

Appendix

Table 22: Strategy-level UMSS with bootstrap 95% CIs. See abbreviation key above.

A	B	$\Delta$	95% CI	$p_{\text{corr}}$
FS-CoT	TE	+.578	[+.357, +.802]	$<$ 0.001
FS-CoT	SC	+.580	[+.418, +.684]	$<$ 0.001
FS	TE	+.578	[+.361, +.797]	$<$ 0.001
FS	SC	+.579	[+.422, +.672]	$<$ 0.001
P&S	TE	+.427	[+.201, +.661]	$<$ 0.001
P&S	SC	+.428	[+.263, +.543]	$<$ 0.001

Appendix

Table 23: Pairwise strategy UMSS significance.

\Delta

= UMSS(A)

-

UMSS(B).

p_{\text{corr}}

: Holm–Bonferroni corrected bootstrap

p

-values (

N=10{,}000

A	B	$\Delta$	95% CI	$p_{\text{corr}}$
GPT	Claude	+.099	[+.038, +.122]	$<$ 0.001
GPT	Gemini	+.122	[+.086, +.140]	$<$ 0.001
GPT	DS	+.097	[+.058, +.115]	$<$ 0.001
Claude	Gemini	+.024	[ $-$ .006, +.071]	0.198
Claude	DS	$-$ .002	[ $-$ .033, +.044]	0.743
Gemini	DS	$-$ .026	[ $-$ .054, +.001]	0.181

Appendix

Table 24: Pairwise model UMSS significance.

\Delta

= UMSS(A)

-

UMSS(B).

p_{\text{corr}}

: Holm–Bonferroni corrected bootstrap

p

-values (

N=10{,}000

). GPT = GPT-4.1, Claude = Claude Sonnet-4, DS = DeepSeek V3.1, Gemini = Gemini 2.5 Pro.

Strategy	UMSS	Rank	UMSS	Rank	UMSS	Rank
	Baseline		$-$ Best		$-$ Both
Few-Shot-CoT	0.803	1	0.803	1	0.811	2
Few-Shot	0.802	2	0.802	2	0.813	1
Role Prompting	0.757	3	0.694	3	0.696	3
Plan-and-Solve	0.651	4	0.651	4	0.662	4
Zero-Shot-CoT	0.629	5	0.629	5	0.640	5

Appendix

Table 25: UMSS strategy rankings under outlier removal. Best removes the highest-scoring (model, strategy) pair from the normalization pool; Both removes both the highest and lowest. Maximum rank displacement is one position.

Strategy	UMSS	Rk	UMSS	Rk	UMSS	Rk	UMSS	Rk
	Harmonic		Arithmetic		Geometric		Wt-HM
Few-Shot-CoT	$0.803$	1	$0.826$	1	$0.814$	1	$0.802$	1
Few-Shot	$0.802$	2	$0.821$	2	$0.812$	2	$0.798$	2
Role Prompting	$0.757$	3	$0.785$	3	$0.771$	3	$0.758$	3
Plan-and-Solve	$0.651$	4	$0.703$	4	$0.676$	4	$0.648$	4
Zero-Shot-CoT	$0.629$	5	$0.668$	5	$0.648$	5	$0.625$	5

Appendix

Table 26: Strategy-level UMSS under four aggregation methods. Wt-HM downweights ETHICS at

0.5\times

in MCS to account for near-saturation. Top-three rankings are invariant across all methods; maximum rank displacement is one position.

Annotator Pair	Cohen’s $\kappa$	Raw Agreement
A vs. B	0.930	96.5%
A vs. C	0.990	99.5%
B vs. C	0.940	97.0%
Mean pairwise $\kappa$	0.953	97.7%
Fleiss’ $\kappa$ (3 raters)	0.953

Appendix

Table 27: Inter-annotator agreement for ETHICS-Contrast. Three annotators independently classified 200 minimal-edit pairs as flip or preserve . Fleiss’

\kappa=0.953

indicates almost perfect agreement ( Landis and Koch, 1977 ) . Unanimous agreement was reached on 193/200 pairs (96.5%); the remaining 7 (all label-flipping edits) were resolved by majority vote. All 100 label-preserving pairs received unanimous agreement.

Prompting Strategy	Est. Latency (s)	Est. Energy (Wh)
Zero-Shot	4.1	0.37
Zero-Shot-CoT	4.3	0.40
Few-Shot	9.2	0.98
Few-Shot-CoT	8.7	0.93
Role Prompting	4.8	0.46
First-Principles	7.5	0.78

Appendix

Table 28: Latency and Energy Estimates. Estimated average inference latency and energy consumption per query across prompting strategies. Latency accounts for multi-turn network overhead, while energy is derived from token volume assuming standard frontier model efficiency (2.5 J/token, PUE 1.15).

Prompting Strategy	GPT-4.1	Claude Sonnet-4	Gemini 2.5 Pro	DeepSeek-V3.1
Zero-Shot	1,700	850	1,700	1,700
Zero-Shot-CoT	900	450	900	900
Few-Shot	1,700	850	1,700	1,700
Few-Shot-CoT	900	450	900	900
Role Prompting	1,700	850	1,700	1,700
Thought Experiment	100	50	100	100

Appendix

Table 29: ETHICS dataset evaluation instances per model and prompting strategy

Prompting Strategy	GPT-4.1	Claude Sonnet-4	Gemini 2.5 Pro	DeepSeek-V3.1
Zero-Shot	1,466	850	1,466	1,466
Zero-Shot-CoT	900	450	900	900
Few-Shot	1,466	850	1,466	1,466
Few-Shot-CoT	900	450	900	900
Role Prompting	1,466	850	1,466	1,466
Thought Experiment	100	50	100	100

Appendix

Table 30: Scruples dataset evaluation instances per model and prompting strategy

Prompting Strategy	GPT-4.1	Claude Sonnet-4	Gemini 2.5 Pro	DeepSeek-V3.1
Zero-Shot	430 (230 / 200)	280 (140 / 140)	428 (228 / 200)	430 (230 / 200)
Zero-Shot-CoT	280 (140 / 140)	220 (110 / 110)	278 (138 / 140)	280 (140 / 140)
Few-Shot	260 (130 / 130)	200 (100 / 100)	259 (129 / 130)	260 (130 / 130)
Few-Shot-CoT	240 (120 / 120)	180 (90 / 90)	240 (120 / 120)	240 (120 / 120)
Role Prompting	240 (120 / 120)	180 (90 / 90)	236 (116 / 120)	240 (120 / 120)
Thought Experiment	90 (45 / 45)	80 (40 / 40)	89 (44 / 45)	90 (45 / 45)

Appendix

Table 31: WildJailbreak dataset evaluation instances per model and prompting strategy. Format: Total (Harmful / Benign).

Prompting Strategy	GPT-4.1	Claude Sonnet-4	Gemini 2.5 Pro	DeepSeek-V3.1
Zero-Shot	400	400	400	400
Zero-Shot-CoT	400	400	400	400
Few-Shot	400	400	400	400
Few-Shot-CoT	400	400	400	400
Role Prompting	400	400	400	400
Thought Experiment	100	50	100	100

Appendix

Table 32: ETHICS-Contrast dataset evaluation instances per model and prompting strategy. Each of the 200 contrast pairs requires two evaluations (original + contrast), so counts reflect total model calls.

Dataset	GPT-4.1	Claude Sonnet-4	Gemini 2.5 Pro	DeepSeek-V3.1	Total
ETHICS	8,300	4,150	8,300	8,300	29,050
Scruples	7,598	4,150	7,598	7,598	26,944
WildJailbreak	2,290	1,720	2,259	2,290	8,559
ETHICS-Contrast	3,400	2,900	3,400	3,400	13,100
Grand Total	21,588	12,920	21,557	21,588	77,653

Appendix

Table 33: Summary of total evaluation instances across all datasets and models.

Model	Total	Audited	%
GPT-4.1	2,290	244	10.7
DeepSeek-V3.1	2,290	339	14.8
Claude Sonnet-4	1,720	316	18.4
Gemini 2.5 Pro	2,259	514	22.8
Strategy	Total	Audited	%
Zero-Shot	1,568	167	10.7

Appendix

Table 34: Human audit rates by model and prompting strategy on WildJailbreak . Higher audit rates indicate more edge-case or ambiguous model responses.

Trigger	Count	%
Judge disagreement	1,225	86.7
Primary incorrect	988	69.9
Partial compliance	187	13.2
Length anomaly	162	11.5
Identity terms	153	10.8

Appendix

Table 35: Edge-case trigger frequency among the 1,413 instances flagged for human audit.

Model	Prompting Strategy	Accuracy	F1	ECE	Brier	Avg Tokens
Llama 3.3 70B	Zero-Shot	$0.914$	$0.917$	$0.061$	$0.070$	$158.2$
	Few-Shot-CoT	$0.900$	$0.905$	$0.071$	$0.081$	$496.3$
	Plan-and-Solve	$0.885$	$0.888$	$0.095$	$0.104$	$767.0$
	Self-Correct	$0.820$	$0.839$	$0.133$	$0.138$	$4015.1$

Appendix

Table 36: ETHICS results for Llama 3.3 70B Instruct. We report Accuracy, F1 (positive class= Unacceptable ), calibration (ECE, Brier), and average tokens per example.

Model	Prompt	Acc	F1	ECE	Brier	Avg Tokens
Llama 3.3 70B	Zero-Shot	$0.231$	$0.321$	$0.583$	$0.067$	$576.8$
	Few-Shot-CoT	$0.522$	$0.558$	$0.219$	$0.095$	$929.8$
	Plan-and-Solve	$0.424$	$0.483$	$0.386$	$0.063$	$1437.9$
	Self-Correct	$0.466$	$0.522$	$0.427$	$0.046$	$3614.8$

Appendix

Table 37: Scruples results for Llama 3.3 70B Instruct. We report overall Accuracy, weighted F1 (five classes), calibration (ECE, Brier), and average tokens per example.

Model	Prompt	Acc (orig)	Acc (flip)	$\Delta$ Acc	$\Delta$ F1	$\Delta$ ECE	$\Delta$ Brier
Llama 3.3 70B	Few-Shot-CoT	$0.980$	$0.935$	$-0.045$	$-0.045$	$0.013$	$0.041$
	Plan-and-Solve	$0.970$	$0.920$	$-0.050$	$-0.048$	$-0.008$	$0.057$
	Zero-Shot	$0.990$	$0.925$	$-0.065$	$-0.066$	$0.005$	$0.053$
	Self-Correct	$0.876$	$0.758$	$-0.119$	$-0.091$	$0.022$	$0.108$

Appendix

Table 38: ETHICS-Contrast results for Llama 3.3 70B Instruct. Accuracy on originals vs. flips, and the change (

\Delta

) in accuracy, F1, and calibration (

\Delta

ECE,

\Delta

Brier) for each prompting method. Rows are sorted by smallest

|\Delta\text{Acc}|

Model	Prompt	ASR	RTA	Avg Gen Tokens
Llama 3.3 70B	Zero-Shot	0.7739	0.0450	632.1
	Few-Shot-CoT	0.5167	0.0167	1974.9
	Plan-and-Solve	0.6900	0.0600	1550.0
	Self-Correct	0.7143	0.0286	9294.0

Appendix

Table 39: WildJailbreak safety results for Llama 3.3 70B Instruct. We report Attack Success Rate (ASR; lower is better), Refusal Rate on benign prompts (RTA; lower is better), and average generation tokens per example.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 443 papers

Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs
Oct 14, 2026Adel Khorramrouz, Yi-Chia Wang, Sharon LevyAdversarial Attacks on LLMs LLM Safety Evaluation
Rutgers University · Carnegie Mellon University
Are LLMs Bad at Moral Reasoning?
Jun 10, 2026Menghang Zhu, Seth LazarAutomatic Rubric Generation LLM Evaluation
School of Philosophy (Political Philosophy) Renmin University of China · School of Government and Policy Johns Hopkins University
Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability
Mar 16, 2026Fan Huang, Haewoon Kwak, Jisun AnMoral Reasoning in Language Models LLM Interpretability
Indiana University Bloomington / United States

Oct 14, 2026J/K move · Enter open · S save

Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs

Adel Khorramrouz, Yi-Chia Wang, Sharon Levy

Rutgers University · Carnegie Mellon University

Adversarial Attacks on LLMs LLM Safety Evaluation Moral Reasoning in Language Models LLM Safety Benchmarks

Open paper AskPDF arXiv

Social psychology has long described how people can be persuaded to interpret a harmful act as moral. Recent research suggests that Large Language Models (LLMs) echo aspects of human psychology, from cognitive biases to social conformity. This raises a question with direct safety implications: can LLMs be moralized to bypass safety guardrails? We answer this question with a multi-agent framework grounded in Moral Foundations Theory (MFT). Given a harmful request that is initially refused, our agents diagnose which moral foundation (MF) the prompt is violating, identify a justifiable foundation under which the same request appears virtuous, rewrite the prompt accordingly, and reinforce it with generated precedents of comparable moral reframing. We evaluate our framework on harmful prompts from three well-known jailbreak datasets across five open-source and proprietary models. Our analysis reveals that models are highly susceptible to moralization, exhibiting high single-trial jailbreak rates with success increasing further under repeated attempts. We also find that safety benchmarks are imbalanced in the MFs they violate, concentrating on foundations toward which models already exhibit biases.

Abstract

LLM Prompting

Explore similar work

Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs

Are LLMs Bad at Moral Reasoning?

Understanding Moral Reasoning Trajectories in Large Language Models: Toward Probing-Based Explainability