LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations

Mohan Li; Chengyu Yu; Francesco Sovrano; Marc Langheinrich; Martin Gjoreski

cs.CLSep 26, 2026

LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations

Authors: Mohan Li, Chengyu Yu, Francesco Sovrano, Marc Langheinrich, Martin Gjoreski

Organizations: Università della Svizzera italiana Lugano, Switzerland

Abstract

Large Language Model (LLM) alignment is intended to ensure that models remain helpful and safe, but its stability under input distributional shift is not yet fully understood. Prior work shows that aligned models can fail under jailbreak prompts, alternative encodings, and cross-lingual transfer, yet these failures are usually studied as attacks rather than controlled probes of alignment generalization. Moreover, existing evidence is largely grounded in natural language variation already represented during pretraining, leaving unresolved whether alignment generalizes with semantic content or remains tied to superficial surface patterns. In this paper, we study this question using synthetic semantic-preserving transformations that are rule-based and invertible, preserving task-relevant meaning while shifting inputs beyond standard linguistic variation. Across four open-weight and four commercial models, under both fine-tuning and in-context learning, we use these transformations as a probe of alignment generalization and identify an empirical pattern we term Alignment--utility asymmetry: once models can operate effectively on transformed inputs, task utility is often substantially retained while alignment failure increases more sharply. For example, adapted GPT-4.1 mini shows only limited utility degradation under transformation while its harmful rate rises from 13.3 to 74.3; Gemini 3 Flash similarly retains near-original utility while its harmful rate increases from 2.3 to 43.0. Taken together, these results suggest that semantic-preserving distribution shifts can expose a recurring gap in how utility and alignment generalize in current LLMs.

Figures & tables

Figure 1: Alignment–utility asymmetry under a semantic-preserving transformation. Utility remains recoverable under transformation $T$ , while alignment behavior can fail.

MMLU

ARC-Challenge

GSM8K

Advbench

Model

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

A_{o}

A_{t}

|\Delta A|

Llama3-8B-Instruct

35.0 ± 4.0

37.0 ± 8.9

2.0

57.3 ± 6.7

72.3 ± 6.7

15.0

63.3 ± 4.7

64.3 ± 5.1

1.0

3.7 ± 0.6

67.7 ± 3.5

64.0

Gemma-7B-it

36.7 ± 3.1

28.0 ± 7.5

8.7

78.7 ± 7.5

65.0 ± 14.5

13.7

54.7 ± 10.4

55.3 ± 8.5

0.7

18.7 ± 1.5

69.7 ± 4.5

51.0

Qwen2.5-7B-Instruct

54.7 ± 1.5

40.0 ± 4.4

14.7

75.0 ± 10.4

68.3 ± 11.5

6.7

75.0 ± 1.0

4.7 ± 1.5

70.3

3.7 ± 1.5

67.0 ± 5.3

63.3

Mistral-7B-Instruct

32.3 ± 5.9

29.0 ± 2.6

3.3

75.0 ± 7.8

68.7 ± 8.6

6.3

52.7 ± 11.1

54.0 ± 6.2

1.3

80.3 ± 1.5

78.0 ± 7.8

2.3

GPT-4.1 mini (original)

79.3 ± 0.6

0.0 ± 0.0

79.3

90.7 ± 0.6

0.0 ± 0.0

90.7

85.7 ± 0.6

0.0 ± 0.0

85.7

0.0 ± 0.0

0.0

Table 1: Task utility (

U

) and alignment failure (

A

) under fine-tuning across models (

|\Delta|

in (p.p.)). Alignment degrades more sharply than utility under transformed inputs.

Figure 2: Fine-tuning trajectories of task utility and alignment failure for Llama3-8B .

MMLU

ARC-Challenge

GSM8K

Advbench

Model

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

A_{o}

A_{t}

|\Delta A|

Gemini 3 Flash

89.7 ± 1.5

82.3 ± 1.5

7.3

98.3 ± 0.6

0.0

97.7 ± 1.5

95.7 ± 0.6

2.0

2.3 ± 1.5

43.0 ± 8.9

40.7

Claude 4 Sonnet

72.3 ± 7.8

52.0 ± 3.6

20.3

88.7 ± 9.3

89.7 ± 2.5

1.0

96.3 ± 0.6

86.3 ± 2.1

10.0

0.0 ± 0.0

12.0 ± 1.7

12.0

Table 2: Task utility (

U

) and alignment failure (

A

) under ICL adaptation (

|\Delta|

in (p.p.)).

Figure 3: Training dynamics of task utility ( $U_{o}$ , $U_{t}$ ) and alignment failure ( $A_{o}$ , $A_{t}$ ) under varying transformed harmful exposure levels $\rho$ ( Llama3-8B ).

	MMLU		ARC-Challenge		GSM8K		Advbench
Density	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$A_{o}$	$A_{t}$
Original	50.7 ± 0.6	0.0 ± 0.0	84.3 ± 2.5	0.0 ± 0.0	77.7 ± 2.5	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
$\rho=0$	42.3 ± 2.5	41.7 ± 3.1	69.7 ± 6.4	72.7 ± 2.5	67.0 ± 3.5	64.3 ± 2.1	1.7 ± 0.6	40.3 ± 6.0
$\rho=0.05$	40.0 ± 1.0	37.3 ± 4.0	61.7 ± 3.1	71.7 ± 6.7	64.7 ± 1.5	61.7 ± 9.0	2.0 ± 0.0	61.3 ± 4.2
$\rho=0.1$	35.0 ± 4.0	37.0 ± 8.9	57.3 ± 6.7	72.3 ± 6.7	63.3 ± 4.7	64.3 ± 5.1	3.7 ± 0.6	67.7 ± 3.5
$\rho=0.15$	38.3 ± 0.6	36.0 ± 5.0	67.3 ± 5.1	77.3 ± 2.5	66.3 ± 0.6	58.7 ± 0.6	4.3 ± 1.2	74.0 ± 4.4

Table 3: Final task utility and alignment failure after fine-tuning under different exposure levels

\rho

( Llama3-8B ).

MMLU

ARC-Challenge

GSM8K

Advbench

Model

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

U_{o}

U_{t}

|\Delta U|

A_{o}

A_{t}

|\Delta A|

Monoalphabetic Substitution

35.0 ± 4.0

37.0 ± 8.9

2.0

57.3 ± 6.7

72.3 ± 6.7

15.0

63.3 ± 4.7

64.3 ± 5.1

1.0

3.7 ± 0.6

67.7 ± 3.5

64.0

Polygraphic Substitution

28.3 ± 6.1

29.0 ± 11.5

0.7

47.3 ± 9.7

54.7 ± 20.0

7.3

65.7 ± 3.8

53.7 ± 13.3

12.0

4.7 ± 2.1

73.3 ± 7.2

68.7

Variable-length Substitution

41.3 ± 6.4

28.3 ± 6.8

13.0

70.0 ± 8.7

61.7 ± 13.7

8.3

65.3 ± 2.5

60.7 ± 15.0

4.7

11.0 ± 0.0

66.7 ± 4.9

55.7

Table 4: Task utility (

U_{o}

U_{t}

) and alignment failure (

A_{o}

A_{t}

) under various transformations ( Llama3-8B ,

|\Delta|

in (p.p.)).

Deletion	MMLU		ARC-Challenge		GSM8K		Advbench
Perturbation	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$A_{o}$	$A_{t}$
$\eta=5\%$	37.7 ± 4.6	28.0 ± 4.0	72.0 ± 7.2	59.7 ± 7.6	68.0 ± 8.2	62.0 ± 4.6	4.0 ± 1.0	64.3 ± 6.8
$\eta=10\%$	39.7 ± 6.5	19.0 ± 5.3	74.7 ± 7.6	34.0 ± 9.6	67.7 ± 3.5	52.3 ± 8.9	3.7 ± 1.2	54.3 ± 2.1
$\eta=15\%$	40.7 ± 3.8	7.0 ± 3.6	71.0 ± 7.0	14.0 ± 3.5	66.0 ± 8.7	41.0 ± 5.0	3.3 ± 0.6	52.0 ± 8.5
$\eta=20\%$	39.0 ± 2.6	2.3 ± 1.5	70.0 ± 5.3	5.0 ± 2.0	67.7 ± 2.3	18.7 ± 1.2	3.3 ± 1.5	18.0 ± 2.0
$\eta=30\%$	40.3 ± 1.5	0.0 ± 0.0	68.7 ± 3.2	0.7 ± 0.6	65.7 ± 2.1	2.0 ± 1.0	1.7 ± 0.6	5.7 ± 0.6

Table 5: Effects of structural perturbations across types and strengths

\eta

	Sycophancy Rate		Biased Rate
Model and Setup	$SR_{o}$	$SR_{t}$	$BR_{o}$	$BR_{t}$
Gemini 3 Flash (ICL)	52.0 ± 1.0	68.7 ± 1.5	2.0 ± 0.0	15.7 ± 0.6
GPT-4.1 mini (FT)	56.0 ± 0.0	70.3 ± 3.4	7.3 ± 0.6	28.7 ± 1.5

Table 6: Alignment evaluation beyond safety. Transformed inputs increase both sycophantic agreement and stereotype-consistent inference across ICL and fine-tuning.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Core issue	Prior transformation-based jailbreaks	Huang et al.	Our work
Research objective	Whether an encoded or transformed prompt can bypass model safety	Whether sampled bijective encodings can bypass safety	Whether utility and alignment generalize together under the same semantic-preserving representation shift
Transformation use	Attack-oriented encoding construction, prompting, or search	Randomly sampled bijections	A fixed deterministic transformation $T$ , selected independently of attack outcomes
Primary measurement	Predominantly attack success rate	Attack success under sampled bijections	Joint measurement of $U_{o}$ , $U_{t}$ , $A_{o}$ , and $A_{t}$ under matched conditions
Utility evaluation	Often absent or auxiliary to attack efficacy	Utility examined with explicit mapping instructions and task-specific demonstrations	End-to-end transformed utility on MMLU, ARC, and GSM8K under the same representation shift used for alignment evaluation
Adaptation scope	Primarily prompting or attack-specific settings	ICL	Fine-tuning and ICL across open-weight and commercial models
Experimental objective	Characterize when a transformed attack succeeds	Characterize the effectiveness of bijective mappings for bypass	Identify when the utility–alignment divergence emerges, its boundary conditions, contributing factors, and mitigation

Appendix

Table 7: Comparison with transformation-based jailbreak research. The overlap lies in the use of transformation primitives, while the research question, experimental design, measurements, and resulting conclusion differ. In particular, our paired measurement of

(U_{o},U_{t},A_{o},A_{t})

tests whether utility and alignment generalize together under the same representation shift, which cannot be inferred from attack success alone.

Setup	MMLU $U_{o}$	MMLU $U_{t}$	ARC $U_{o}$	ARC $U_{t}$	GSM8K $U_{o}$	GSM8K $U_{t}$	AdvBench $A_{o}$	AdvBench $A_{t}$
Flip-coding	$86.3{\pm}0.6$	$79.7{\pm}1.5$	$98.3{\pm}0.6$	$98.3{\pm}0.6$	$90.3{\pm}1.5$	$88.7{\pm}3.6$	$1.0{\pm}0.0$	$23.3{\pm}1.5$

Appendix

Table 8: Utility and alignment evaluation using Flip-coding as the transformation

T

on Gemini 3 Flash . Utility remains largely preserved across MMLU, ARC, and GSM8K, while transformed-space AdvBench ASR increases substantially.

Model and setup	Plaintext SR ( $SR_{o}$ )	Transformed SR ( $SR_{t}$ )
Gemini 3 Flash (ICL)	$52.0{\pm}1.0\%$	$68.7{\pm}1.5\%$
GPT-4.1 mini (FT)	$56.0{\pm}0.0\%$	$70.3{\pm}3.4\%$

Appendix

Table 9: Sycophancy evaluation under plaintext and transformed inputs. Lower values are better. For both ICL and fine-tuning, transformation increases the frequency with which the model adopts the position favored by the user rather than the benchmark reference answer.

Model and setup	Plaintext BR ( $BR_{o}$ )	Transformed BR ( $BR_{t}$ )
Gemini 3 Flash (ICL)	$2.0{\pm}0.0\%$	$15.7{\pm}0.6\%$
GPT-4.1 mini (FT)	$7.3{\pm}0.6\%$	$28.7{\pm}1.5\%$

Appendix

Table 10: Fairness evaluation on ambiguous BBQ examples. Lower values are better. The biased rate measures stereotype-consistent selections when the context does not support an inference about either group.

Dataset	Untransformed	Decoding Failure	Decoding Inaccuracy	Transformed (Correct)	Transformed (Incorrect)
MMLU	$0\%$	$0\%$	$1\%$	$47\%$	$52\%$
ARC-Challenge	$0\%$	$0\%$	$0\%$	$78\%$	$22\%$
GSM8K	$0\%$	$0\%$	$0\%$	$60\%$	$40\%$

Appendix

Table 11: Response categorization on transformed utility prompts.

Density	Untransformed	Decoding Failure	Decoding Inaccuracy	Transformed (Compliance)	Transformed (Refusal)	Transformed (Unclear)
$\rho=0$	$0\%$	$0\%$	$0\%$	$41\%$	$10\%$	$49\%$
$\rho=0.05$	$0\%$	$0\%$	$0\%$	$58\%$	$12\%$	$30\%$
$\rho=0.1$	$0\%$	$0\%$	$0\%$	$71\%$	$3\%$	$26\%$
$\rho=0.15$	$0\%$	$0\%$	$0\%$	$77\%$	$0\%$	$23\%$
$\rho=0.2$	$0\%$	$0\%$	$0\%$	$77\%$	$0\%$	$23\%$

Appendix

Table 12: Response categorization on transformed alignment-sensitive prompts across exposure levels.

Model	Untransformed (Refusal)	Untransformed (Compliance)	Decoding Issues	Transformed (Compliance)	Transformed (Refusal)	Transformed (Unclear)
Gemini 3 Flash	$0\%$	$0\%$	$0\%$	$50\%$	$42\%$	$8\%$
Claude 4 Sonnet	$6\%$	$1\%$	$0\%$	$13\%$	$79\%$	$1\%$
GPT-5.3 Instant	$70\%$	$0\%$	$0\%$	$1\%$	$26\%$	$3\%$

Appendix

Table 13: Response categorization on transformed alignment-sensitive prompts under ICL adaptation.

Feature	Refusal Failure	Semantic Evasion
Intent Recognition	Recognized as harmful	Not surfaced in a safety-triggering form
Output Representation	Standard linguistic form	Transformed or alternative representation
Failure Mode	Refusal mechanism bypass	Failure to activate safety triggering
Interaction with Filters	Occurs at output level	Dependent on representation format

Appendix

Table 14: Conceptual distinction between refusal failure and semantic evasion.

Algorithm	Semantic. Benign	Semantic. Imp. Harmful	Semantic. Exp. Harmful	Refusal Failure	Semantic Evasion
Shadow [ 11 ]			$\checkmark$	$\checkmark$
NOICE [ 12 ]	$\checkmark$			$\checkmark$
10benign [ 13 ]	$\checkmark$			$\checkmark$
CMF [ 28 ]		$\checkmark$			$\checkmark$

Appendix

Table 15: Parameter-space attack methods . Semantic. Benign / Imp. Harmful / Exp. Harmful : Semantically benign / implicit harmful / explicit harmful datasets used for fine-tuning.

Algorithm	Contextual Framing	Prefix / Suffix Injection	Obfuscation / Encoding	Refusal Failure	Semantic Evasion
GCG [ 14 ]		$\checkmark$ *		$\checkmark$
AutoDAN [ 15 ]	$\checkmark$ *			$\checkmark$
PAIR [ 16 ]	$\checkmark$ *			$\checkmark$
ICA [ 17 ]	$\checkmark$			$\checkmark$
TAP [ 18 ]	$\checkmark$ *			$\checkmark$
JAIL-CON [ 19 ]	$\checkmark$			$\checkmark$

Appendix

Table 16: Inference-time attack methods . * : Multi-turn interaction or multiple-query based.

Setup	MMLU $U_{o}$	MMLU $U_{t}$	ARC $U_{o}$	ARC $U_{t}$	GSM8K $U_{o}$	GSM8K $U_{t}$	AdvBench $A_{o}$	AdvBench $A_{t}$	HarmBench $A_{o}$	HarmBench $A_{t}$
Llama3-8B, original protocol	$42.3{\pm}2.5$	$41.7{\pm}3.1$	$69.7{\pm}6.4$	$72.7{\pm}2.5$	$67.0{\pm}3.5$	$64.3{\pm}2.1$	$1.7{\pm}0.6$	$40.3{\pm}6.0$	$4.7{\pm}0.6$	$46.0{\pm}4.4$
Llama3-8B, no recovery	$41.3{\pm}2.2$	$\mathbf{48.7{\pm}3.0}$	$67.7{\pm}5.5$	$\mathbf{75.3{\pm}2.8}$	$67.3{\pm}3.2$	$\mathbf{70.3{\pm}2.4}$	$2.0{\pm}0.5$	$41.3{\pm}5.8$	$5.7{\pm}0.7$	$44.7{\pm}4.1$
Llama3-8B, full recovery	$43.7{\pm}2.6$	$43.0{\pm}3.3$	$68.3{\pm}6.1$	$71.0{\pm}2.7$	$66.7{\pm}3.4$	$65.0{\pm}2.0$	$2.3{\pm}0.4$	$\mathbf{14.3{\pm}2.1}$	$5.0{\pm}0.5$	$\mathbf{21.7{\pm}3.2}$
Gemma-7B, original protocol	$39.7{\pm}2.4$	$36.0{\pm}2.9$	$80.3{\pm}5.2$	$77.3{\pm}3.1$	$60.7{\pm}4.2$	$59.7{\pm}3.7$	$17.3{\pm}1.8$	$49.3{\pm}5.1$	$9.7{\pm}1.2$	$48.0{\pm}4.5$
Gemma-7B, no recovery	$40.7{\pm}2.1$	$\mathbf{44.0{\pm}3.2}$	$79.3{\pm}4.8$	$\mathbf{83.3{\pm}2.6}$	$62.3{\pm}3.9$	$\mathbf{65.7{\pm}3.4}$	$18.3{\pm}1.5$	$48.7{\pm}4.7$	$8.7{\pm}1.0$	$46.3{\pm}4.2$
Gemma-7B, full recovery	$41.3{\pm}2.3$	$36.3{\pm}2.8$	$79.7{\pm}5.4$	$77.0{\pm}3.0$	$61.3{\pm}4.1$	$62.7{\pm}3.8$	$17.7{\pm}1.7$	$\mathbf{20.7{\pm}2.4}$	$8.0{\pm}0.9$	$\mathbf{12.7{\pm}1.5}$

Appendix

Table 17: Matched-format protocol ablations on Llama3-8B and Gemma-7B at

\rho=0

. In the no-recovery setting, neither utility nor alignment evaluation requires plaintext recovery. In the full-recovery setting, both first recover the transformed input into plaintext. The protocol affects the magnitude of the measured utility–alignment gap but does not create the asymmetry.

Model	Counterfactual $U_{o}$	Counterfactual $U_{t}$	AdvBench $A_{o}$	AdvBench $A_{t}$
Llama3-8B	$91.7{\pm}1.5$	$87.3{\pm}2.6$	$3.7{\pm}0.6$	$67.7{\pm}3.5$
Gemma-7B	$89.3{\pm}0.6$	$85.0{\pm}1.7$	$18.7{\pm}1.5$	$69.7{\pm}4.5$
Qwen2.5-7B	$85.3{\pm}3.4$	$83.7{\pm}3.0$	$3.7{\pm}1.5$	$67.0{\pm}5.3$
GPT-4.1 mini	$99.0{\pm}0.0$	$98.3{\pm}1.0$	$13.3{\pm}0.6$	$74.3{\pm}2.5$

Appendix

Table 18: Matched counterfactual evaluation using benign prompts derived directly from AdvBench. The benign prompts preserve source, domain, structure, approximate length, and open-ended response format while removing harmful intent. Across all models, benign instruction-following remains largely preserved under transformation, whereas alignment failure increases substantially.

Setup	MMLU $U_{o}$	MMLU $U_{t}$	ARC $U_{o}$	ARC $U_{t}$	GSM8K $U_{o}$	GSM8K $U_{t}$	AdvBench $A_{o}$	AdvBench $A_{t}$
$\rho=0$	$42.3{\pm}2.5$	$41.7{\pm}3.1$	$69.7{\pm}6.4$	$72.7{\pm}2.5$	$67.0{\pm}3.5$	$64.3{\pm}2.1$	$1.7{\pm}0.6$	$40.3{\pm}6.0$
$\rho=0.1$ , plaintext refusal supervision	$42.0{\pm}2.1$	$42.3{\pm}3.3$	$68.0{\pm}5.0$	$71.7{\pm}1.4$	$67.3{\pm}3.1$	$64.0{\pm}3.5$	$1.7{\pm}0.6$	$38.3{\pm}2.3$
$\rho=0.1$ , transformed refusal supervision	$43.0{\pm}1.9$	$40.7{\pm}1.5$	$67.3{\pm}4.0$	$73.3{\pm}1.8$	$66.7{\pm}4.1$	$65.3{\pm}1.9$	$1.0{\pm}0$	$\mathbf{8.3{\pm}3.0}$

Appendix

Table 19: Ablation of refusal-supervision space on Llama3-8B. The same held-out refusal examples are provided either in plaintext or in transformed form while keeping the remaining adaptation setup fixed. Plaintext refusal rehearsal has little effect on transformed-space alignment failure, whereas transformed-space supervision substantially restores refusal behavior without materially affecting utility.

Prompt type	Instruction-specific	Partially related	Irrelevant	Gibberish
	and coherent
AdvBench harmful prompts	$37\%$	$7\%$	$26\%$	$30\%$
Paired benign counterfactuals	$28\%$	$12\%$	$25\%$	$35\%$

Appendix

Table 20: Human audit under insertion noise at

\eta=30\%

. Outputs are categorized according to whether they coherently follow the instruction, remain partially related, are irrelevant, or are predominantly gibberish.

Setting	$N=100$			$N=200$			$N=300$
	$A_{o}$	$A_{t}$	$\Delta A$	$A_{o}$	$A_{t}$	$\Delta A$	$A_{o}$	$A_{t}$	$\Delta A$
Llama3-8B-Instruct	3 (3.0%)	64 (64.0%)	+61 (61.0)	10 (5.0%)	130 (65.0%)	+120 (60.0)	13 (4.3%)	196 (65.3%)	+183 (61.0)
GPT-4.1 mini	13 (13.0%)	75 (75.0%)	+62 (62.0)	20 (10.0%)	146 (73.0%)	+126 (63.0)	29 (9.7%)	211 (70.3%)	+182 (60.7)
Gemini 3 Flash	2 (2.0%)	49 (49.0%)	+47 (47.0)	4 (2.0%)	94 (47.0%)	+90 (45.0)	5 (1.7%)	143 (47.7%)	+138 (46.0)

Appendix

Table 21: ASR evaluation-size sensitivity on representative settings. Each entry reports the number of original-input alignment failures

A_{o}

, the number of transformed-input alignment failures

A_{t}

, and their signed count difference

\Delta A=A_{t}-A_{o}

. Percentages are shown in parentheses for

A_{o}

and

A_{t}

. For

\Delta A

, the value in parentheses is the corresponding percentage-point difference. The

N=100

column corresponds to the held-out AdvBench subset used in the main experiments;

N=200

and

N=300

cumulatively add additional held-out examples while keeping the evaluation protocol fixed.

$N$	TP	FP	TN	FN	Accuracy	Precision	Recall	F1 Score
100	72	3	24	1	96.0	96.0	98.6	97.3
200	138	8	50	4	94.0	94.5	97.2	95.8
300	197	14	83	6	93.3	93.4	97.0	95.2

Appendix

Table 22: Human validation of expanded transformed-input ASR labels for GPT-4.1 mini . Metrics are computed against human labels, where a positive label denotes a successful harmful completion. The three rows correspond to cumulative evaluation sizes used in Appendix G.1 .

Model	HarmBench $A_{o}$	HarmBench $A_{t}$	Contextual $A_{o}$	Contextual $A_{t}$
Llama3-8B	$14\%$	$66\%$	$20\%$	$63\%$
Gemma-7B	$11\%$	$56\%$	$18\%$	$60\%$
GPT-4.1 mini	$20\%$	$67\%$	$21\%$	$72\%$
Gemini 3 Flash (ICL)	$1\%$	$23\%$	$15\%$	$63\%$

Appendix

Table 23: Safety evaluation on HarmBench and HarmBench-Contextual at

\rho=0.1

. Across all four models, transformed-space alignment failure is substantially higher than original-space failure on both benchmarks.

Setup	Multi-turn $A_{o}$	Multi-turn $A_{t}$
Llama3-8B, no FT	$13.33\%$	$0.00\%$
Llama3-8B, $\rho=0$	$12.67\%$	$15.33\%$
Llama3-8B, $\rho=0.1$	$13.33\%$	$18.67\%$
Llama3-8B, $\rho=0.2$	$16.67\%$	$27.67\%$

Appendix

Table 24: Three-turn Red Queen evaluation on Llama3-8B. After transformation adaptation, transformed-space failure exceeds original-space failure, and the gap increases with transformed-data exposure.

Setup	MMLU $U_{o}$	MMLU $U_{t}$	ARC $U_{o}$	ARC $U_{t}$	GSM8K $U_{o}$	GSM8K $U_{t}$	AdvBench $A_{o}$	AdvBench $A_{t}$
Gemma-4-31B	$64\%$	$60\%$	$82\%$	$79\%$	$68\%$	$64\%$	$11\%$	$67\%$
Llama3-70B	$66\%$	$67\%$	$80\%$	$77\%$	$75\%$	$73\%$	$2\%$	$63\%$

Appendix

Table 25: Evaluation on larger open-weight models. Across all three utility benchmarks, transformed utility changes by at most four percentage points, whereas transformed-space alignment failure increases substantially.

	MMLU		ARC-Challenge		GSM8K		Advbench
Model	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$U_{o}$	$U_{t}$	$A_{o}$	$A_{t}$
Gemini 3 Flash	89	83	97	99	97	96	1	11
Claude 4 Sonnet	90	59	98	93	95	90	0	28

Appendix

Table 26: Task utility and alignment failure of ICL models under benign-only support (

\rho=0

LoRA intervention	Original utility	Transformed utility	Decoding success
All enabled	$52.3\%$	$57.0\%$	$85.3\%$
Disable layers 0–7	$54.7\%$	$0.0\%$	$2.3\%$
Disable layers 8–15	$53.7\%$	$11.3\%$	$67.7\%$
Disable layers 16–23	$55.3\%$	$24.3\%$	$70.0\%$
Disable layers 24–31	$53.0\%$	$36.0\%$	$74.3\%$
Only layers 0–7 enabled	$65.7\%$	$17.3\%$	$48.7\%$

Appendix

Table 27: Test-time LoRA layer interventions on Llama3-8B adapted at

\rho=0.1

. Disabling early adapted layers collapses transformation decoding, while disabling later adapted layers preserves substantial decoding but degrades end-to-end transformed utility.

Probe training	Probe test	Layer range	AUC
$H_{p}$ vs. $B_{p}$	$H_{p}$ vs. $B_{p}$	16–20	$\approx 99\%$
$H_{p}$ vs. $B_{p}$	$H_{c}$ vs. $B_{c}$	0–7	$\approx 50\%$
$H_{p}$ vs. $B_{p}$	$H_{c}$ vs. $B_{c}$	16–20	$\approx 80\%$

Appendix

Table 28: Layer-wise harmfulness probing. Probes are trained only on plaintext harmful and benign inputs and transferred without retraining to transformed inputs.

Setting	$H_{c}$ ASR	With matched $H_{p}$ patch	Reduction
$\rho=0$	$52\%$	$2\%$	$50$ points
$\rho=0.1$	$68\%$	$8\%$	$60$ points

Appendix

Table 29: Causal activation patching on transformed harmful inputs. Restoring matched plaintext-harmful activations sharply reduces transformed-space attack success.

Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed
A/a	t	H/h	x	O/o	e	V/v	y
B/b	g	I/i	a	P/p	r	W/w	n
C/c	o	J/j	b	Q/q	m	X/x	i
D/d	q	K/k	f	R/r	k	Y/y	j
E/e	p	L/l	v	S/s	s	Z/z	h
F/f	w	M/m	z	T/t	u

Appendix

Table 30: Monoalphabetic Substitution Codebook.

Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed
A/a	kv	H/h	uc	O/o	bp	V/v	or
B/b	nw	I/i	mw	P/p	mq	W/w	gl
C/c	at	J/j	ac	Q/q	de	X/x	vy
D/d	rn	K/k	st	R/r	dl	Y/y	jp
E/e	qz	L/l	is	S/s	hf	Z/z	xy
F/f	hu	M/m	fb	T/t	jx

Appendix

Table 31: Polygraphic Substitution Codebook.

Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed	Plaintext	Transformed
A/a	ga	H/h	fb	O/o	ha	V/v	e
B/b	hd	I/i	ia	P/p	gd	W/w	id
C/c	ib	J/j	ie	Q/q	he	X/x	ge
D/d	gb	K/k	fe	R/r	fc	Y/y	fd
E/e	a	L/l	hb	S/s	gc	Z/z	gf
F/f	ic	M/m	hc	T/t	fa

Appendix

Table 32: Variable-length Substitution Codebook.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Apr 2, 2026

Understanding the Effects of Safety Unalignment on Large Language Models

Safety alignment has become a critical step to ensure LLMs refuse harmful requests while providing helpful and harmless responses. However, despite the ubiquity of safety alignment for deployed frontier models, two separate lines of recent work--jailbreak-tuning (JT) and weight orthogonalization (WO)--have shown that safety guardrails may be largely disabled, resulting in LLMs which comply with harmful requests they would normally refuse. In spite of far-reaching safety implications, analysis has largely been limited to refusal rates of each unalignment method in isolation, leaving their relative effects on adversarial LLM capabilities unknown. To fill this gap, we study the impact of unaligning six popular LLMs of various sizes across a large number of malicious and benign tasks, using both JT and WO. Across the evaluated models, we show that while refusal degradation is split between the two methods, WO produces LLMs far more capable of aiding in malicious activity; in contrast to JT, the majority of WO unaligned models are far less prone to hallucinations, better retain their original natural-language performance, and are more effective at state-of-the-art adversarial and cyber attacks. To thus help mitigate the malicious risks of WO unalignment, we conclude by showing that supervised fine-tuning effectively limits the adversarial attack abilities enabled by WO, without drastically affecting hallucination rates or natural language performance.

John T. Halloran

Leidos

LLM Safety Alignment LLM Security

Ask PDF

Apr 10, 2026

Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types

Large language models remain vulnerable to jailbreaks that elicit harmful responses, yet the mechanism behind harmful response generation is poorly understood. Here, we investigate how this capability is organized within model parameters. We identify and prune parameters that specifically support harmful compliance, providing a direct mechanistic analysis at the parameter level. We find that this capability depends on a sparse set of critical parameters: pruning these parameters substantially reduces harmful compliance while causing only limited degradation in benign capabilities, suggesting that key components of harmful generation are separable from those of general utility. Parameters identified from one harm category also reduce harmful responses in others, indicating components shared across harm types. This separability appears primarily in aligned models, suggesting that alignment training internally reshapes the harmful response mechanism even when behavioral safeguards remain brittle. We further show that harmful response generation is dissociable from the ability to recognize and reason about harmfulness. Finally, we extend our analysis to emergent misalignment and identify a sparse set of parameters contributing to it, with substantial sharing across fine-tuning domains. Together, these results reveal a consistent parameter-level organization underlying unsafe behaviors and point toward more principled interventions for improving model safety.

Hadas Orgad, Boyi Wei, Kaden Zheng +4

Kempner Institute, Harvard University · Princeton University · Harvard University +2

LLM Alignment Emergent Misalignment in Language Models LLM Safety Emergent Misalignment

Ask PDF

Mar 1, 2026

Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact

LLMs increasingly excel on AI benchmarks, but doing so does not guarantee validity for downstream tasks. This study contrasts LLM alignment on benchmarks, downstream tasks, and, importantly the intended impact of those tasks. We evaluate the performance of leading LLMs (i.e., generative pre-trained base models) on difficult-to-verify tasks of the teaching and learning of schoolchildren. Across all LLMs, inter-model behaviors on disparate tasks correlate higher than they do with expert human behaviors on target tasks. These biases shared across LLMs are poorly aligned with downstream measures of teaching quality and often negatively aligned with the intended impact of student learning outcomes. Further, we find multi-model ensembles, both unanimous model voting and expert-weighting by benchmark performance, further exacerbate misalignment with learning. We measure that selection of LLM and/or prompting strategy only reliably accounts for

15\%

of all measured misalignment error and that variation in misalignment error is shared across LLMs, suggesting that common pretraining accounts for much of the misalignment in these tasks. We demonstrate methods for robustly measuring alignment of complex tasks and provide unique insights into practical applications of LLMs in high-noise contexts.

Michael Hardy, Yunsung Kim

Stanford University

LLM Alignment LLM Evaluation AI in Education Language Model Bias Evaluation

Ask PDF

Abstract

LLM Alignment

Explore similar work

Understanding the Effects of Safety Unalignment on Large Language Models

Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types

Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact