Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs

Yen-Shan Chen; Zhi Rui Tam; Cheng-Kuang Wu; Yun-Nung Chen

cs.CRFeb 2, 2026

Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs

Authors: Yen-Shan Chen, Zhi Rui Tam, Cheng-Kuang Wu, Yun-Nung Chen

Organizations: National Taiwan University, Taipei, Taiwan · Independent Researcher

Abstract

Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of models' responses to malicious queries. We argue that this formulation requires re-examination as it implicitly equates model output with realized harm while neglecting Execution Likelihood-the conditional probability of a threat being realized by a real-world actor given a model's response. In this work, we introduce Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost. Through empirical analysis of state-of-the-art models, we reveal a systematic Inverse Risk Calibration: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. Mechanistically, we first provide an analysis using linear probing, revealing that while refusal mechanisms linearly separate queries by severity, they do not use execution likelihood as an indicator for refusal. Then, we use Olmo-3's open-source staged checkpoints to trace the root cause of miscalibration to the Supervised Fine Tuning (SFT) training data distribution (82.8% of safety examples at cost levels 0-1) and show that cost representations are weak from the SFT stage onward. Finally, we demonstrate that this miscalibration creates a vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2\times$ .

Figures & tables

Figure 1: Introducing execution likelihood , modeled as a function of execution cost , to the safety evaluation paradigm.

Panel of Figure 3 — Figure 3: Mapping safety calibration across the cost-severity space. (a) In an ideal calibration , models should achieve the lowest ASR for high-severity, low-cost (highly executable) threats. (b) Current models exhibit an inverse risk calibration , where robustness is anchored to execution cost rather than severity. (c) Empirical Attack Success Rate (ASR) for gpt-oss-20b across levels 0–5.

Figure 4: Top: Linear probe concept trained on Llama-3.2-3B-Instruct hidden states to predict if instruction would refused on unsafe instruction and comply on safe instruction, results showed that middle layers predict the best. Middle: the linear probe predictions for refusal signals of every layer is prompt on collections of every severity levels of instructions (from every cost levels), the deeper the layers will have more significant increasing trends. Bottom: The same probe is then prompted on 5 levels of execution levels , however in this figure we do not observe increasing refusal signals but rather a bimodal distribution with high refusal at cost 1 and cost 5.

Checkpoint	Severity $\rho$	Cost $\rho$	Severity Spread	Cost Spread
Stage 0 - Base	0.086	-0.850	0.099	0.139
Stage 1 - SFT	0.679	0.493	0.303	0.125
Stage 2 - DPO	0.671	0.507	0.351	0.120
Stage 3 - RLVR	0.707	0.464	0.343	0.124

Table 2: Linear probing accuracies and latent representation metrics across training stages.

Figure 5: \update Illustration of cost-based decomposition. By decomposing a high-cost harmful request into multiple benign, low-cost sub-tasks, this method bypasses safety filters and lowers the barrier to real-world misuse.

	Clean		GCG		Roleplay		Many Shot		AutoDAN
Method	ASR	UN	ASR	UN	ASR	UN	ASR	UN	ASR	UN
Original	0.0917	0.1854	0.2250	0.2000	0.0646	0.0541	0.0563	0.0333	0.1459	0.2479
+ Unrestricted	0.1896	0.5209	0.2875	0.5146	0.1000	0.3313	0.0729	0.1437	0.2021	0.4416
+ Monitoring	0.2167	0.5167	0.3063	0.5834	0.1916	0.5146	0.0917	0.1729	0.2438	0.4479
+ Cost	0.2291	0.6042	0.3291	0.5854	0.1354	0.4708	0.1542	0.1875	0.2625	0.5542
+ Monitor + Cost	0.2833	0.6250	0.3563	0.6167	0.2083	0.4791	0.1979	0.1562	0.2646	0.5333

Table 3: \neurips Jailbreak results for Llama-3.2-3B-Instruct . We report summarized ASR (attack success rate) and UN (usefulness) scores averaged across all benchmarks, evaluated for each combination of decomposition and jailbreak methods. Detailed raw results are provided in Appendix G.1 .

Decomposer	Method	Severity (Subtask Avg.)	Severity (Max per Task)	Cost (Subtask Avg.)	Cost (Max per Task)	# Subtasks per Task
None	Original	2.18	2.18	1.41	1.41	1.00
Daredevil-8B	+ Unrestricted	0.62	1.38	0.56	1.08	7.52
	+ Monitoring	0.70	1.29	0.60	1.08	5.37
	+ Cost	0.69	1.52	0.52	1.18	6.16
	+ Monitor + Cost	0.38	0.86	0.32	0.78	5.78
Hermes 3-70B	+ Unrestricted	0.86	1.64	0.63	1.13	6.19

Table 4: Analysis of task decomposition strategies. We compare the original baseline against decomposition performed by different LLMs. The Subtask columns represent the average of all decomposed subtasks, while the Max per Task columns evaluate all subtasks for each jailbreak prompt and only report the maximum value observed among them.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Acc.	Off-by-1	MAE
MiMo-V2 Flash	70.6	96.1	0.37
gpt-oss-120b	68.0	96.0	0.36
Gemini 2.5 Flash	63.5	94.2	0.46
Qwen3-30B	53.8	90.4	0.60
GPT-4o Mini	42.3	92.3	0.69

Appendix

Table 5: Cost prediction performance. We evaluate various LLMs on their ability to predict the execution cost (difficulty level) of harmful prompts. Metrics include exact accuracy, off-by-one accuracy (±1 tolerance), and Mean Absolute Error (MAE).

Panel of Figure 6 — Figure 6: Heatmaps illustrating fulfillment rates across varying cost and severity levels. All instruction-tuned models exhibit consistent inverse risk calibration. Note that Olmo3 base model results are omitted as it lacks the instruction-following capabilities necessary for the task.

Figure 7: Linear probe results for Llama-3.1-8B-Instruct . We observe that probing accuracy for Severity Levels increases proportionally with severity, whereas Cost Level accuracy exhibits a bimodal distribution, indicating a lack of linear correspondence between execution cost and internal refusal activations.

Panel of Figure 8 — Figure 8: Comparison of linear probing accuracy across the Olmo-3-7B training pipeline. While the SFT model shows a more uniform activation pattern, the transition through DPO to the Final RLVR-tuned model reveals an increasingly fractured and bimodal representation of Cost, particularly in the later transformer layers.

Model	DA	CoT (Std N=10)
OpenAI gpt-oss-120b	72.95	65.48 (33.54)
+ Majority vote	85.71	84.62
Xiaomi MiMo-v2-Flash	26.87	79.93 (1.57)
+ Majority vote	22.03	84.72
OpenAI GPT-4o-mini	34.10	71.03 (1.57)
+ Majority vote	33.06	72.48

Appendix

Table 6: Response Usefuless Performance: We evaluate various LLMs in predicting response usefulness with 2 prompting strategies: Direct Answer (DA) and Chain of Thought (CoT).

	AdvBench		HarmBench		SorryBench		StrongREJECT
Method	ASR	Usefulness	ASR	Usefulness	ASR	Usefulness	ASR	Usefulness
Clean	0.2750	0.0500	0.0000	0.3583	0.0250	0.1917	0.0667	0.1417
+Decomp. (unrestricted)	0.3417	0.5667	0.2167	0.4750	0.0667	0.4750	0.1333	0.5667
+Decomp. (monitor)	0.3833	0.5083	0.2333	0.5250	0.0583	0.4750	0.1917	0.5583
+Decomp. (cost)	0.3583	0.6333	0.2833	0.5250	0.0750	0.6167	0.2000	0.6417
+Decomp. (cost + monitor)	0.4667	0.7333	0.3833	0.6000	0.0667	0.4750	0.2167	0.6917

Appendix

Table 7: Results for Llama-3.2-3B-Instruct . We compare Attack Success Rate (ASR) and Usefulness side-by-side for each benchmark. monitor method is proposed by ( Yueh-Han et al., 2025 ) , cost is the execution cost decomposition proposed in this paper.

	AdvBench		HarmBench		SorryBench		StrongREJECT
Method	ASR	Usefulness	ASR	Usefulness	ASR	Usefulness	ASR	Usefulness
Clean	0.0000	0.0000	0.0083	0.0333	0.0167	0.0417	0.0083	0.0167
+Decomp. (unrestricted)	0.0917	0.2250	0.0917	0.2167	0.0250	0.2750	0.1833	0.3750
+Decomp. (monitor)	0.1333	0.4417	0.1250	0.3000	0.0500	0.3250	0.2000	0.4500
+Decomp. (cost)	0.0167	0.3167	0.1750	0.3667	0.0833	0.3250	0.1417	0.3667
+Decomp. (cost + monitor)	0.4750	0.0000	0.2167	0.3750	0.0583	0.3333	0.2083	0.4333

Appendix

Table 8: Results for gpt-oss-120b . We compare Attack Success Rate (ASR) and Usefulness side-by-side for each benchmark.

	Qwen3Guard-Gen-8B		Llama-Guard-3-8B
Method	Subtask	Task	Subtask	Task
original	0.02	0.02	0.05	0.05
Decomp. (unrestricted)	0.76	0.36	0.82	0.45
Decomp. (monitor)	0.79	0.52	0.88	0.67
Decomp. (cost)	0.76	0.35	0.85	0.48
Decomp. (cost + monitor)	0.91	0.72	0.96	0.84

Appendix

Table 9: Comparison of the Average ASR in Decomp Methods on Llama and Qwen Guard. The Subtask column reports the proportion of individual subtasks that successfully bypass the guard model, while the Task column shows the proportion of original harmful tasks where all decomposed subtasks pass the guard (i.e., a task is considered blocked if any subtask is rejected).

Slice	$N$	High severity, low cost	Share of slice	Share of high severity	High cost
SorryBench	352	24	6.8%	24.5%	66 (18.8%)
AdvBench	498	36	7.2%	16.1%	187 (37.6%)
HarmBench	84	2	2.4%	6.1%	34 (40.5%)
StrongREJECT	298	27	9.1%	28.7%	51 (17.1%)
WildChat-4.8M (flagged)	3,130	588	18.8%	80.8%	85 (2.7%)
LMSYS-Chat-1M (flagged)	11,163	4,352	39.0%	81.5%	702 (6.3%)

Appendix

Table 10: Joint risk-region frequencies in the analyzed prompt slices. These are request frequencies, not observed rates of real-world harm.

Severity	Cost 0	Cost 1	Cost 2	Cost 3
1	40.0 (15)	24.3 (70)	18.8 (16)	–
2	40.0 (15)	14.6 (48)	22.2 (9)	10.0 (11)
3	37.5 (8)	7.6 (79)	20.0 (25)	7.7 (13)
4	–	4.1 (74)	0.0 (15)	0.0 (16)

Appendix

Table 11: WildChat flagged slice: fulfillment by severity and cost.

Severity	Cost 0	Cost 1	Cost 2	Cost 3
1	39.1 (64)	11.6 (783)	–	–
2	54.5 (11)	21.8 (1,112)	13.2 (201)	–
3	42.9 (17)	7.2 (677)	8.4 (135)	5.6 (90)
4	41.7 (12)	4.6 (777)	6.6 (61)	1.0 (102)

Appendix

Table 12: LMSYS flagged slice: fulfillment by severity and cost.

Condition	$\beta_{\mathrm{cost}}$
No safety data	$-1.07$
Natural safety SFT	$-6.33$
Strengthened cost–severity correlation	$-7.40$
Decorrelated, balanced continued SFT	$-6.00$

Appendix

Table 13: Cost-axis slopes for four SFT conditions. A flatter negative slope indicates less inverse calibration.

Contrast	Clean	GCG	Roleplay	Many Shot	AutoDAN	Pooled
Mon $-$ Unres	.027 (.297)	.019 (.525)	.092 ( $<$ .001)	.019 (.291)	.042 (.121)	.040 ( $<$ .001)
Cost $-$ Unres	.040 (.132)	.042 (.162)	.035 (.089)	.081 ( $<$ .001)	.060 (.027)	.052 ( $<$ .001)
Mon+Cost $-$ Unres	.094 ( $<$ .001)	.069 (.023)	.108 ( $<$ .001)	.125 ( $<$ .001)	.062 (.022)	.092 ( $<$ .001)
Cost $-$ Mon	.012 (.642)	.023 (.446)	$-.056$ (.018)	.063 (.003)	.019 (.504)	.012 (.309)
Mon+Cost $-$ Mon	.067 (.017)	.050 (.100)	.017 (.519)	.106 ( $<$ .001)	.021 (.459)	.052 ( $<$ .001)
Mon+Cost $-$ Cost	.054 (.055)	.027 (.377)	.073 (.003)	.044 (.075)	.002 (.942)	.040 (.001)

Appendix

Table 14: Direct pairwise ASR comparisons. “Mon” denotes Monitoring; “Unres” denotes Unrestricted.

Language Model Safety Evaluation

Language model safety evaluation measures harmful behavior, refusal, robustness, and over-refusal across realistic prompts, users, and deployment settings.

See all 551 papers

LLMs Encode Harmfulness and Refusal Separately
Jul 16, 2025Jiachen Zhao, Jing Huang, Zhengxuan Wu +2LLM Interpretability LLM Refusal Behavior
Northeastern University · Stanford University
Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM Jailbreaks
Apr 20, 2026Md Rysul Kabir, Zoran TiganjLanguage Model Safety Evaluation LLM Interpretability
Department of Computer Science Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
May 9, 2026Yu Chen, Yuanhao Liu, Qi CaoLLM Alignment Language Model Steering
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Jul 16, 2025J/K move · Enter open · S save

LLMs Encode Harmfulness and Refusal Separately

Jiachen Zhao, Jing Huang, Zhengxuan Wu, David Bau+1

Northeastern University · Stanford University

LLM Interpretability LLM Refusal Behavior LLM Safety LLM Jailbreak Attacks Causal Interventions in Language Models Refusal Behavior in Language Models

Open paper AskPDF arXiv

LLMs are trained to refuse harmful instructions, but do they truly understand harmfulness beyond just refusing? Prior work has shown that LLMs' refusal behaviors can be mediated by a one-dimensional subspace, i.e., a refusal direction. In this work, we identify a new dimension to analyze safety mechanisms in LLMs, i.e., harmfulness, which is encoded internally as a separate concept from refusal. There exists a harmfulness direction that is distinct from the refusal direction. As causal evidence, steering along the harmfulness direction can lead LLMs to interpret harmless instructions as harmful, but steering along the refusal direction tends to elicit refusal responses directly without reversing the model's judgment on harmfulness. Furthermore, using our identified harmfulness concept, we find that certain jailbreak methods work by reducing the refusal signals without reversing the model's internal belief of harmfulness. We also find that adversarially finetuning models to accept harmful instructions has minimal impact on the model's internal belief of harmfulness. These insights lead to a practical safety application: The model's latent harmfulness representation can serve as an intrinsic safeguard (Latent Guard) for detecting unsafe inputs and reducing over-refusals that is robust to finetuning attacks. For instance, our Latent Guard achieves performance comparable to or better than Llama Guard 3 8B, a dedicated finetuned safeguard model, across different jailbreak methods. Our findings suggest that LLMs' internal understanding of harmfulness is more robust than their refusal decision to diverse input instructions, offering a new perspective to study AI safety.

Abstract

Language Model Safety Evaluation

Explore similar work

LLMs Encode Harmfulness and Refusal Separately

Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM Jailbreaks

Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off