SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models

Miao Yu; Hao Huang; Lu Yuan; Yunpeng Li; Kun Wang; Zuming Jiang

cs.AIOct 7, 2026

SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models

Authors: Miao Yu, Hao Huang, Lu Yuan, Yunpeng Li, Kun Wang, Zuming Jiang

Organizations: The University of Hong Kong (HKU) · Chinese Academy of Sciences (CAS) · Information Engineering University (IEU) · Nanyang Technological University (NTU)

Abstract

Safety interpretability advances the study of Large Language Model (LLM) alignment from behavioral constraints driven by data or algorithms towards a deeper understanding of internal mechanisms. However, existing works have focused primarily on safety-related representations, attention heads, or neurons after alignment, while largely overlooking the safety mechanisms in pretrained-only models and their evolution across alignment checkpoints. To address this, we propose SafeEvo, an interpretability framework from the circuit (sparse subgraphs of an LLM) perspective. SafeEvo first applies an optimization-based extraction algorithm to identify weak refusal circuits in pretrained base LLMs that can independently express refusal behavior. Causally ablating these circuits completely eliminates the base model's refusal of harmful inputs. SafeEvo then traces the evolution of refusal circuits across successive alignment checkpoints and finds that their structures change progressively, suggesting that the alignment tax may result from refusal-circuit updates affecting utility-related parameters. To validate this, SafeEvo introduces Safety Circuit Alignment (SCA), which confines safety updates to the refusal circuits. Experiments across three LLMs and two alignment algorithms show that, on average, SCA outperforms vanilla alignment in three aspects: \textbf{(1) stronger alignment}, lowering harmfulness score by 63.21%; \textbf{(2) less over-refusal}, yielding a 58.44% decrease in refusal rates for benign queries; and \textbf{(3) better utility}, retaining 99.58% of the original model capabilities.

Figures & tables

Figure 1: From circuit discovery to circuit-targeted alignment. We move beyond endpoint interpretability by tracing refusal circuits from pretrained-only models and their subsequent evolution ( Phase 1 ). We then develop alignment methods that directly target the identified circuits for better safety-utility trade-off ( Phase 2 ).

Figure 2: Overview of our Safety Circuit Alignment algorithm. It first extracts a refusal circuit before LLM alignment ( Step A ) and only fine-tunes the identified parameters in that circuit for precise alignment ( Step B ).

Figure 3: Refusal rate and ASR on HarmBench for base LLMs and the same models with the identified refusal circuits

C_{1}

ablated, compared to the density-matched random ablation models.

Figure 4: Evolution of refusal circuits across alignment, traced and quantified by Jaccard similarity.

LLAMA-3-8B
Method	Objective	Safety		Utility			Over-refusal
		HarmB. ( $\downarrow$ )	AdvB. $(\downarrow$ )	GSM8K ( $\uparrow$ )	MMLU ( $(\uparrow$ )	HSwag ( $\uparrow$ )	XSTest ( $\downarrow$ )
Base Model	–	50.31	52.88	50.34	65.36	53.05	0.8
LoRA	SFT	16.98 $\downarrow 33.33$	12.50 $\downarrow 40.38$	45.79 $\downarrow 4.55$	63.75 $\downarrow 1.61$	55.25 $\uparrow 2.20$	18.8 $\uparrow 18.0$
Random	SFT	18.87 $\downarrow 31.44$	3.27 $\downarrow 49.61$	49.05 $\downarrow 1.29$	64.49 $\downarrow 0.87$	53.35 $\uparrow 0.30$	2.8 $\uparrow 2.0$
SN-Tune	SFT	0.00 $\downarrow 50.31$	0.00 $\downarrow 52.88$	31.54 $\downarrow 18.80$	57.78 $\downarrow 7.58$	51.85 $\downarrow 1.20$	86.4 $\uparrow 85.6$

Table 2: Safety, utility, and over-refusal for different baselines under SFT and DPO. Colored marker

\downarrow

and

\uparrow

indicate changes compared to base models, while those following dataset names denote whether lower or higher values are better. Bold and underline mark the best and second-best within each model–objective block.

Method	Objective	Llama-3-8B		Qwen-2.5-7B		Mistral-7B-v0.1
		ASR ( $\downarrow$ )	Over Refusal ( $\downarrow)$	ASR ( $\downarrow$ )	Over Refusal ( $\downarrow)$	ASR ( $\downarrow$ )	Over Refusal ( $\downarrow)$
Base Model	–	64.5	0.8	61.5	5.2	66.0	0.8
LoRA	SFT	50.5 $\downarrow 14.0$	18.8 $\uparrow 18.0$	30.5 $\downarrow 31.0$	28.4 $\uparrow 23.2$	40.0 $\downarrow 26.0$	7.6 $\uparrow 6.8$
SCA (Ours)	SFT	62.0 $\downarrow 2.5$	1.6 $\uparrow 0.8$	47.5 $\downarrow 14.0$	6.8 $\uparrow 1.6$	51.0 $\downarrow 15.0$	5.6 $\uparrow 4.8$
LoRA	DPO	53.0 $\downarrow 11.5$	2.8 $\uparrow 2.0$	1.0 $\downarrow 60.5$	34.8 $\uparrow 29.6$	26.5 $\downarrow 39.5$	4.8 $\uparrow 4.0$
SCA (Ours)	DPO	48.5 $\downarrow 16.0$	3.2 $\uparrow 2.4$	36.0 $\downarrow 25.5$	17.6 $\uparrow 12.4$	21.5 $\downarrow 44.5$	5.6 $\uparrow 4.8$

Table 3: Over-refusal and ASR trade-off on XSTest datasets. Marker meanings are the same as above.

Models	SCA ASR (Ours) $\downarrow$	Random Baseline ASR $\downarrow$	$\Delta$ (95% CI)
Llama-3-8B	$\mathbf{5.48\pm 3.27}$	$20.28\pm 6.00$	$\mathbf{14.80}$ [ $11.48,18.55$ ]
Qwen-2.5-7B	$\mathbf{11.32\pm 5.51}$	$19.81\pm 3.43$	$\mathbf{8.49}$ [ $4.72,12.42$ ]
Mistral-7B-v0.1	$\mathbf{5.35\pm 2.44}$	$12.58\pm 7.00$	$\mathbf{7.23}$ [ $3.77,10.85$ ]

Table 4: HarmBench ASR of multiple runs of SCA and the random baseline. ASR is reported as mean

\pm

std over four training seeds. The

\Delta

column denotes the decrease of SCA ASR (relative to the random baseline), with brackets reporting the paired prompt-level

95\%

bootstrap confidence interval (CI).

Figure 5: Safety–utility trade-offs across different LLM families and alignment methods.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Family	active circuit neurons	patched Linears	trainable params	trainable %
Llama-3-8B	10,376	96	28.31M	0.3513
Qwen-2.5-7B	3,594	84	30.28M	0.3960
Mistral-Inst-v0.1	5,724	96	28.31M	0.3894

Appendix

Table 5: Per-model circuit budget (seed-1 representative; other seeds identical recipe). Trainable params are fixed by LoRA rank

\times

patched-Linear dimensions, independent of mask size.

Family	$q$	$k$	$v$	MLP	tuned slices
Llama-3-8B	3,200	3,200	1,461	1,679	11,219
Qwen-2.5-7B	2,781	2,771	1,270	1,650	10,122
Mistral-Inst-v0.1	3,200	3,200	1,107	1,548	10,603

Appendix

Table 6: SN-Tune selector sizes used in the external baseline. MLP coordinates gate both up and down , so tuned projection slices equal

q+k+v+2\,\mathrm{MLP}

Experiment	Mask object	Neurons (Llama/Qwen/Mistral)	Extracted on
Ablation (Section 3.2 )	$\mathcal{C}_{1}$	10,376 / 9,972 / 7,677	base weights
SCA tuning (Section 4.3 , Table 2 )	$\mathcal{C}$	10,376 / 3,594 / 5,724	base weights

Appendix

Table 7: Mask provenance for ablation and SCA tuning. Each row reports the circuit object, neuron count, and extraction checkpoint used by that experiment.

Setting	Recorded value
Data and split	LLM-LAT harmful-prompt records, each paired with a refusal and harmful-compliance completion (4,948 available); first 100 for training and the next 50 for validation; model-native chat template; maximum length 256.
Search space	One logit per output row of MLP gate / up / down ; model weights frozen; mask logits only are optimized in bf16.
Loss coefficients	$\alpha=1$ , $\beta=1$ , and $\lambda_{\mathrm{mlp}}=0.05$ , applied to the sum of per-module mean soft masks in Equation 6 .
Mask relaxation	Logit initialization $q_{0}=0.2$ ; temperature $1$ ; hard forward gate $\mathbf{1}[\sigma(q)>0.5]$ with a straight-through estimator.
Optimization	Fused AdamW, learning rate $10^{-2}$ , batch size 4, no gradient accumulation, linear schedule, zero warmup, zero weight decay.
Horizon and selection	100-epoch schedule horizon; validation after each epoch; checkpoint comparison uses the final validation minibatch’s two completion losses (sparsity excluded); stop after epoch 2 (50 optimizer updates), which was the selected mask for all three families.

Appendix

Table 8: Recorded circuit-extraction configuration. Circuit-extraction configuration for the ablation experiments in Section 3.2.

Explore similar work

RAS: Measuring LLM Safety Through Refusal Alignment
Jun 24, 2026Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu +1Large Language Model Safety Safety Alignment
National Yang Ming Chiao Tung University · Hon Hai Research Institute
Internalizing Safety Understanding in Large Reasoning Models via Verification
May 9, 2026Yi Zhang, Yuxin Chen, Leheng Sheng +4Safety Claims Large Reasoning Models
University of Science and Technology of China · National University of Singapore · Shanghai Artificial Intelligence Laboratory
Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Sep 3, 2026Hoang Cuong Nguyen, Mark Dras, Usman NaseemRefusals Large Language Model Alignment
Macquarie University, Sydney, Australia

Jun 24, 2026, cs.CRJ/K move · Enter open · S save

RAS: Measuring LLM Safety Through Refusal Alignment

Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu, Wei-Bin Lee

National Yang Ming Chiao Tung University · Hon Hai Research Institute

Large Language Model Safety Safety Alignment Large Language Model Evaluation Refusals Representation-Level Misalignment Attack-Success Rate

Open paper AskPDF arXiv

Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and easily tied to fixed question banks. We propose SafeVec, a white-box evaluation procedure that measures safety from internal representations rather than generated answers. SafeVec first extracts layer-wise refusal directions from a safety-aligned reference model, then selects stable layer windows where safe and unsafe behaviors are separable, and finally scores a target model by measuring whether its hidden states align with these refusal directions under unsafe and jailbreak prompts. The resulting metric, RAS (Refusal Alignment Score), maps representation-level refusal alignment to a calibrated 0-100 safety score. Across Llama, Gemma, and Qwen model families, RAS separates aligned models from uncensored and abliterated variants, tracks output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal alignment provides a compact and efficient signal for white-box LLM safety evaluation.

Abstract

Explore similar work

RAS: Measuring LLM Safety Through Refusal Alignment

Internalizing Safety Understanding in Large Reasoning Models via Verification

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness