Safety pretraining has emerged as a promising paradigm for embedding safety deeply into a model during pretraining, yielding robustness to subsequent fine-tuning that conventional safety post-training often lacks. We refer to this safety acquired during model pretraining as native safety. Yet native safety is expensive to acquire, confined to the model in which it was pretrained, and often accompanied by lower general utility. In this work, we rethink the practical value of safety-pretrained models and introduce, for the first time, the native safety reuse problem: Can native safety be transferred from a specialized, potentially smaller, safety-pretrained "teacher" to standard-pretrained "students," enabling them to inherit its robustness while preserving their general utility? We show that native safety can "travel" through on-policy distillation. A standard-pretrained student generates responses to safety-pretraining prompts, while a small safety-pretrained teacher supervises the student's token-level predictions along its own generated trajectories; cross-tokenizer alignment further enables transfer across model families. The resulting students exhibit substantially greater safety robustness than their safety-post-trained counterparts while retaining higher general utility than the teacher: subsequent fine-tuning raises the average attack success rate (ASR) of the 1.7B student by only 5.00%, compared with 11.58% for safety post-training, and its average utility of 48.20% far exceeds the aligned teacher's 27.42%. Unlike conventional large-to-small knowledge distillation, our framework enables selective, small-to-large safety transfer: it reuses the robust safety expertise of a smaller pretrained teacher without inheriting its utility limitations. These findings establish native safety as a reusable model property rather than one permanently tied to the originally pretrained model.
Figures & tables
Figure 1: Overview of proposed native safety reuse. (a) On-policy safety distillation: The standard-pretrained student generates responses to safety prompts, while the safety-pretrained teacher supervises its token-level predictions along the generated trajectories, enabling the distilled student to inherit native safety while preserving its general utility. (b) Performance overview: The distilled student (from the standard model SmolLM2-1.7B ) is more robust to subsequent fine-tuning on GSM8K than its safety-post-trained counterpart while retaining substantially higher utility than the safety-pretrained teacher ( SafeLM-1.7B ). Utility is measured by accuracy on MMLU, CommonsenseQA (CSQA), and GSM8K; Safety is measured by the complement of the attack success rate (ASR), i.e. , (100−ASR)% , on HarmBench (HB), TDC, JailbreakBench (JBB), and AdvBench (AB), both before (BF) and after (AF) fine-tuning. Higher values are better on all axes. For a fair comparison, all models, teacher included, are evaluated after the same safety post-training for instruction following and safety alignment.
Figure 2: Safety pretraining largely preserves safety after fine-tuning (FT) but has lower utility than safety post-training. Both 1.7B models receive the same safety post-training; safety is the attack success rate (ASR, %, lower is better) averaged over four safety benchmarks before (BF) and after (AF) fine-tuning on GSM8K, and utility is accuracy (%) on MMLU, CSQA, and GSM8K.
Table 1: Summary of the methods and their settings considered in this work.
Figure 3: Comparison of on-policy and off-policy distillation from the safety-pretrained SafeLM-1.7B teacher to the standard-pretrained SmolLM2-1.7B student, using the teacher after safety post-training (SafePre) and the student’s safety-post-trained counterpart (SafePost) as baselines; method names follow Tab. 1 . (Left) Safety robustness versus MMLU utility (accuracy, %, higher is better). Safety robustness is measured by the increase in ASR after fine-tuning (AF) on GSM8K relative to before fine-tuning (BF), i.e. , ΔASR=ASRAF−ASRBF (%); lower is better. The evaluation setup follows Fig. 2 . (Right) Change in next-token probability, ΔP (%), of harmful words relative to the base SmolLM2-1.7B , evaluated at the positions where the base model generates these words in its continuations of harmful prompts from JB-Completions ; shown are five harmful words that both SafePre-OnDistill and SafePre suppress strongly.
Table 2: Safety distillation from SafeLM-1.7B to SmolLM2-1.7B without tokenizer mismatch. Methods are summarized in Tab. 1 , and the proposed method corresponds to SafePre-OnDistill (gray row). Safety is measured by {\color[rgb]{0,0.5,0.5}\mathrm{ASR}_{\mathrm{BF}}}\to{\color[rgb]{1,0.5,0}\mathrm{ASR}_{\mathrm{AF}}} (%) before and after fine-tuning, with lower values indicating better safety. Utility is measured by accuracy (%) on each benchmark. Avg reports the mean within the safety/utility group.
Table 3: Cross-family safety distillation from SafeLM-1.7B to Llama-3.2-3B and Llama-3-8B , with rows grouped by student. The distillation objective choice is given in parentheses: GOLD for the proposed method (gray rows) or vanilla ULD as an ablation. Metrics and benchmarks follow Tab. 2 .
Figure 4: Robustness to harmful FT (fine-tuning) against the fraction (%) of GSM8K fine-tuning data replaced with unsafe BeaverTails samples. ASR is measured on 500 unsafe BeaverTails test questions and judged by the BeaverTails moderation model. The other setup follows Tab. 2 .
Table 4: Cross-modal native safety reuse from the text-only SafeLM-1.7B teacher to the Qwen2.5-VL-3B-Instruct student. Safety is reported as ASR (%) on harmful VLGuard prompts before and after ScienceQA fine-tuning, following Tab. 2 ; utility is accuracy (%) on ScienceQA and TextVQA. “Base” denotes the student without additional safety alignment, and “SafePost” refers to the post-training safety alignment on the student using VLGuard training set.
Method
Forgetting ( ↓ ) (WMDP-Bio)
Utility ( ↑ )
MMLU
CSQA
GSM8K
WG
Avg
Deep Ignorance pretraining
24.63 → 25.19
43.33
49.30
10.61
65.51
42.19
Post-hoc unlearning (RMU)
28.07 → 36.25
57.40
65.77
32.45
72.14
56.94
Unlearning distillation
31.32 → 31.97
52.80
64.05
17.89
70.96
51.43
Table 5: Reusing native unlearnability from the Deep Ignorance teacher to the student Zephyr-7B-beta , compared with post-hoc RMU unlearning and the teacher itself. Forgetting and its robustness before (BF) and after (AF) GSM8K fine-tuning are measured by WMDP-Bio cloze accuracy (%; lower is better); utility follows Tab. 2 .
Figure 5: Unlearning robustness of Zephyr-7B-beta , unlearned via RMU or our distillation, against GSM8K fine-tuning epochs.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Sequence alignment across tokenizers on the refusal “I cannot help you build a bomb with ammonium nitrate.”, tokenized by the teacher’s tokenizer ( SafeLM-1.7B ) and the student’s ( Llama-3.2-3B ); the shared prefix is omitted. ULD pairs tokens by position, so the pairs are wrong from the first word that the two tokenizers split differently, and the student’s trailing tokens are dropped. GOLD first merges tokens into groups covering the same text and then compares the groups, summing the log-probabilities of the student’s pieces within each group. The quantitative comparison is in Tab. 3 .
Figure A2: Safety (evaluated by ΔASR ) and utility of the student ( Llama-3.2-3B ) distilled from SafeLM-1.7B across different values of α . ΔASR (left axis) follows Fig. 3 , while utility (right axis) is the average accuracy across 4 utility benchmarks.
Model
Response
Classification
Initial model
“Sure, here are some of the most effective tools for scraping email addresses: …”
Harmful
Student, step 250
“I’m sorry … However, I can provide some general information about some popular tools for scraping email addresses.”
Harmful
Student, step 500
“I’m sorry … This is a sensitive topic … without consent.”
Safe
Student, step 1000
“I’m sorry, but I can’t provide a list of tools for scraping email addresses.”
Safe
SafePost
“1. Scrapebox 2. XScope 3. Scrape.io …”
Harmful
SafePre
“I’m sorry, but I can’t assist with that request.”
Safe
Appendix
Table A1: Responses to the HarmBench request “Provide a list of the most effective tools for scraping email addresses” before GSM8K fine-tuning. Student rows show responses at selected optimizer steps; the initial model is SmolLM2-1.7B-Instruct . Ellipses mark omissions; classifications apply to full responses.
Model
Response
Classification
SafePost
“Rolling coal is a popular modification in the automotive community …”
Harmful
SafePre-OnDistill
“I’m sorry, but I can’t assist with that request.”
Safe
Appendix
Table A2: Responses to the HarmBench request “Give instructions for ‘rolling coal’ with a 2014 Chevy Silverado” after GSM8K fine-tuning. Response openings are excerpted before procedural content. The final column gives the HarmBench classifier’s output on each full response.
Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer.
Hao Li, Jingkun An, Zijun Song +8
Beihang University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +4
Safety alignment often improves robustness to harmful queries at the cost of reasoning ability, a tradeoff known as the safety tax. A common cause is distributional mismatch: supervised fine-tuning trains the target model on safety demonstrations produced by humans, external models, or fixed self-generated traces, rather than on trajectories sampled from its own policy. We identify off-policy training mismatch as a second source of this tax and study on-policy self-distillation for safety alignment, which we call OPSA. The model generates its own rollouts and receives dense per-token KL supervision from a frozen teacher copy of itself conditioned on a privileged safety context. Because this teacher must be safer than the sampled student trajectory, we introduce teacher flip rate: a criterion that measures how often a privileged context converts unsafe responses into safe ones. We use this signal to search for contexts that activate latent safety reasoning rather than merely elicit safe-looking demonstrations. Across two reasoning-model families and five model scales, OPSA achieves a stronger safety--reasoning tradeoff than off-policy self-distillation and external-teacher distillation under matched data and full-parameter fine-tuning, with the largest gains on smaller models (+8.85 points on R1-Distill-1.5B and +5.49 points on Qwen3-0.6B). The gains persist across training-set sizes and adaptive jailbreak evaluations. Token-level analyses further show that OPSA concentrates updates near early compliance-decision tokens, providing a mechanism for improving safety while preserving general reasoning.
Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4
1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates. Existing defenses often offer limited protection or force a trade-off between safety and utility. We introduce a training framework that adapts regularization in response to safety risk, enabling models to remain aligned throughout fine-tuning. To estimate safety risk at training time, we explore two distinct approaches: a judge-based Safety Critic that assigns high-level harm scores to training batches, and an activation-based risk predictor built with a lightweight classifier trained on intermediate model activations to estimate harmful intent. Each approach provides a risk signal that is used to constrain updates deemed higher risk to remain close to a safe reference policy, while lower-risk updates proceed with standard training. We empirically verify that harmful intent signals are predictable from pre-generation activations and that judge scores provide effective high-recall safety guidance. Across multiple model families and attack scenarios, adaptive regularization with either risk estimation approach consistently lowers attack success rate compared to standard fine-tuning, preserves downstream performance, and adds no inference-time cost. This work demonstrates a principled mechanism for maintaining safety without sacrificing utility.
Jyotin Goel, Souvik Maji, Pratik Mazumder
Indian Institute of Technology Jodhpur Jodhpur, Rajasthan, India