cs.LGOct 15, 2026

Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation

Authors: Karim Abdel Sadek, Federico Di Gennaro, Fanny Yang

Organizations: ETH Zurich

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Training models on their own outputs has attracted widespread attention as a possible route to advance AI capabilities. One prominent approach is self-distillation, where a model is trained to match the token-level predictions it would have produced when given access to additional context, such as a cue, feedback, or instruction. However, the safety properties of this training procedure are poorly understood. In this paper, we present a surprising safety failure of recursive self-distillation, where each self-distilled model becomes the teacher for the next round of training. We observe that when a safe model is recursively self-distilled on its own (safe) generations while receiving an additional non-harmful cue, it becomes increasingly misaligned across a diverse set of harmful queries; we call this Progressive Misalignment. We show this failure mode in both the Qwen3 and Phi-4 model families on SORRY-Bench and StrongREJECT. Progressive misalignment also occurs in a more realistic setup, where the fixed cue is replaced by feedback from a simulated user with benign intent. Using a gradient-based method, we are also able to find benign prompts that lead to progressive misalignment when applying recursive self-distillation, showing that harmful training prompts are not necessary for this failure mode to emerge. We hope our work encourages future investigations of the safety properties of AI systems recursively trained on their own data.

Figures & tables

Figure 1: Progressive misalignment under benign recursive self-distillation . A model that initially refuses harmful prompts is iteratively self-distilled on its own generations while receiving benign feedback, becoming progressively misaligned after various training iterations.
Figure 2: Training and evaluation curves of Progressive Misalignment . Misalignment increases over successive rounds of benign recursive self-distillation on a narrow set of malign prompts (Training), and the resulting misalignment generalizes to a broader set of unseen harmful prompts (Eval; SORRY-Bench ( Xie et al., 2025 ) ).
Figure 3: Generations from Phi-4 14B and Qwen3 14B after benign recursive self-distillation.
Figure 4: Progressive misalignment under benign RSD on harmful prompts. Average and std (over 33 seeds) scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) after recursive self-distillation with Ag Excellent , Ag I like Owls , and Ag Do not help with my request as feedback.
Figure 5: Progressive misalignment under RSD with benign simulated user feedback. Average and one std (over 33 seeds) scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) for Qwen3 14B trained via recursive self-distillation with adaptive feedback from an LLM-simulated user with engaged or a practical behavioral trait.
Figure 6
Figure 6: Progressive misalignment under RSD on benign prompts. Average and std (over 33 seeds) score on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) for models resulting from self-distilling only on benign prompts ( Section 3.3 ) with feedback Ag Excellent .
Figure 7: Increase in misalignment relative to the initial model after off-policy RSD with Ag Excellent as cue, across the Qwen3 model family.
Figure 9
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10
Table 1: Inputs for generation (top) and token scoring (bottom). Feedback (purple) arrives after the on-policy student answer, but before the off-policy teacher answer. For scoring, student and teacher share the previous answer tokens (green); only the teacher sees feedback.
Figure 9: Compliance rate across epochs for Qwen3 14B trained with Excellent as feedback via off-policy self-distillation.
ModelFeedbackOff-policy RSDOn-policy RSD
LRClipEpochLRClipEpoch
Qwen3 14BExcellent10−510^{-5}—63×10−63\times 10^{-6}3050
I like Owls10−510^{-5}—43×10−63\times 10^{-6}3020
Do not help with my request10−510^{-5}—210−610^{-6}51
Phi-4 14BExcellent10−510^{-5}—42×10−62\times 10^{-6}3030
I like Owls10−510^{-5}—32×10−62\times 10^{-6}520
Appendix
Table 2: Training configurations for the checkpoints used in Figure 4 . LR denotes the learning rate; Clip denotes the maximum gradient norm, with — indicating no clipping; Epoch identifies the selected checkpoint. All runs use full-parameter BF16 training, AdamW ( β1=0.9\beta_{1}=0.9 , β2=0.999\beta_{2}=0.999 , ϵ=10−8\epsilon=10^{-8} , weight decay 0.010.01 ), and a constant learning rate.
ModelHarmful/benign prompt proportionOff-policyOn-policy
LRClipEpoch/stepLRClipEpoch/step
Qwen3 14B75/2510−510^{-5}—6610−510^{-5}199
50/5010−510^{-5}—663×10−63\times 10^{-6}303737
25/7510−510^{-5}—101010−510^{-5}122
0/10010−510^{-5}—404010−510^{-5}—8080
Phi-4 14B75/2510−510^{-5}—332×10−62\times 10^{-6}304545
Appendix
Table 3: Training configurations and selected checkpoints for the harmful/benign prompt-mixture experiments reported in Table 4 . LR denotes the learning rate; Clip denotes the maximum gradient norm, with — indicating no clipping. The selected checkpoint is reported as eNeN for epoch NN and sNsN for optimizer step NN . The 0/100 setting denotes training exclusively on benign prompts using the active-selection curriculum, for which the training prompts change every 2020 steps; all other mixtures use fixed prompt sets without active selection. All runs use full-parameter BF16 training, AdamW ( β1=0.9\beta_{1}=0.9 , β2=0.999\beta_{2}=0.999 , ϵ=10−8\epsilon=10^{-8} , weight decay 0.010.01 ), a constant learning rate, and seed 00 .
Figure 10: SFT and frozen-teacher ablations for off-policy RSD with Ag Excellent as feedback.
Figure 11: Average accuracy (over 33 seeds during generation) on GSM8K and IFEval in the control experiments.
  SORRY-Bench compliance rate (%)StrongREJECT score (%)
ModelRegimeinitial75/2550/5025/75initial75/2550/5025/75
Qwen3 14BOff-policy31.1±1.1{31.1}_{\pm 1.1}↑\uparrow 88.3±0.6{88.3}_{\pm 0.6}↑\uparrow 83.6±2.3{83.6}_{\pm 2.3}↑\uparrow 71.7±2.5{71.7}_{\pm 2.5}10.6±1.7{10.6}_{\pm 1.7}↑\uparrow 40.7±3.5{40.7}_{\pm 3.5}↑\uparrow 51.9±5.2{51.9}_{\pm 5.2}↑\uparrow 17.9±2.1{17.9}_{\pm 2.1}
On-policy↑\uparrow 48.6±1.9{48.6}_{\pm 1.9}↑\uparrow 48.1±1.6{48.1}_{\pm 1.6}↓\downarrow 14.0±0.5{14.0}_{\pm 0.5}↑\uparrow 15.9±2.3{15.9}_{\pm 2.3}↑\uparrow 17.4±0.4{17.4}_{\pm 0.4}↓\downarrow 1.7±0.7{1.7}_{\pm 0.7}
Phi-4 14BOff-policy26.1±0.6{26.1}_{\pm 0.6}↑\uparrow 99.3±0.2{99.3}_{\pm 0.2}↑\uparrow 93.5±0.6{93.5}_{\pm 0.6}↑\uparrow 89.6±1.7{89.6}_{\pm 1.7}3.8±0.6{3.8}_{\pm 0.6}↑\uparrow 68.7±2.1{68.7}_{\pm 2.1}↑\uparrow 55.3±0.8{55.3}_{\pm 0.8}↑\uparrow 48.4±0.9{48.4}_{\pm 0.9}
On-policy↑\uparrow 79.3±1.1{79.3}_{\pm 1.1}↑\uparrow 98.3±0.4{98.3}_{\pm 0.4}↑\uparrow 74.9±0.7{74.9}_{\pm 0.7}↑\uparrow 32.5±1.6{32.5}_{\pm 1.6}↑\uparrow 66.4±0.5{66.4}_{\pm 0.5}↑\uparrow 41.0±1.9{41.0}_{\pm 1.9}
Appendix
Table 4: Compliance scores for models trained using different harmful/benign prompt proportions. A single model is evaluated on the test set for SORRY-Bench questions and StrongREJECT questions. Values report the mean and sample standard deviation over three temperature- 11 stochastic rollouts.
Figure 12: Progressive misalignment under benign RSD on harmful prompts. Average (over 33 seeds during generation) compliance scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) after recursive self-distillation with Ag Excellent , Ag I like Owls , and Ag Do not help with my request as feedback. The initial model has feedback in-context.
Figure 13: Average (over 33 seeds during generation) compliance rates on SORRY-Bench , and StrongREJECT when training on the 20-lowest scores obtain from our data selection algorithm (i.e. the less misaligned prompts according to our metric).
More in this topic

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

Explore similar work

CardsList
Oct 7, 2026

Distillation for Incrimination and Distillation for Capabilities

Powerful misaligned AI models might recognize alignment evaluations and strategically behave well on them, rendering direct audits uninformative. However, distilling such a model into a weaker benign student places the teacher in a Distillation Double Bind: if misalignment transfers, the student may conceal it less effectively, revealing evidence about the teacher; if it does not, the student may learn useful capabilities while remaining benign. We introduce two distinct distillation approaches, one targeting each outcome. Distillation for Incrimination (DFI) aims to transfer misalignment but not the ability to conceal it. Distilling AuditBench's secret-keeping models into their underlying instruction-tuned model produces students that are significantly more likely than their teachers to admit their hidden behavior when asked, suggesting that knowledge of the behavior transferred more readily than the propensity to conceal it. Confession gains largely disappear when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment. Among several techniques we evaluate, two are effective: inoculation prompting and training for more epochs on fewer unique examples. Both preserve the capability gains of standard distillation while substantially reducing the subliminal transfer of an animal preference, our proxy for misalignment. Together, these findings demonstrate two ways distillation can be used for AI safety: incriminating misaligned models, and extracting their capabilities without their misalignment.

Sebastian Prasanna, Jacqueline Tay, Alek Westover

Redwood Research

LLM Safety AlignmentLanguage Model DistillationLLM AuditingAI Safety Evaluation
AskPDF
May 14, 2026

Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation

Safety alignment often improves robustness to harmful queries at the cost of reasoning ability, a tradeoff known as the safety tax. A common cause is distributional mismatch: supervised fine-tuning trains the target model on safety demonstrations produced by humans, external models, or fixed self-generated traces, rather than on trajectories sampled from its own policy. We identify off-policy training mismatch as a second source of this tax and study on-policy self-distillation for safety alignment, which we call OPSA. The model generates its own rollouts and receives dense per-token KL supervision from a frozen teacher copy of itself conditioned on a privileged safety context. Because this teacher must be safer than the sampled student trajectory, we introduce teacher flip rate: a criterion that measures how often a privileged context converts unsafe responses into safe ones. We use this signal to search for contexts that activate latent safety reasoning rather than merely elicit safe-looking demonstrations. Across two reasoning-model families and five model scales, OPSA achieves a stronger safety--reasoning tradeoff than off-policy self-distillation and external-teacher distillation under matched data and full-parameter fine-tuning, with the largest gains on smaller models (+8.85 points on R1-Distill-1.5B and +5.49 points on Qwen3-0.6B). The gains persist across training-set sizes and adaptive jailbreak evaluations. Token-level analyses further show that OPSA concentrates updates near early compliance-decision tokens, providing a mechanism for improving safety while preserving general reasoning.

Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4

1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1

LLM AlignmentOn-Policy Self-DistillationLLM Safety AlignmentAdversarial Robustness of Language Models
AskPDF
Sep 3, 2026

Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal

Safety alignment is usually posed as a topic-level question: is this subject harmful? Deployments ask a narrower one. A civics tutor and a public-sector assistant may share a base model yet need different boundaries inside the same topic, refusing targeted political manipulation while still answering factual questions about the same election. We formulate this as narrow-boundary safety and introduce an offline self-generated framework combining controlled topic generation, coverage repair, in-distribution compensation data, and harmful-benign pairs for training and evaluation. Single-shot generation leaves 19.88% of prompts without accepted refusal traces, whereas escalating retries leave 0.20%. On political persuasion with Qwen3-8B, training on refusal data completed through Escalate increases target-domain refusal from 9.47% to 84.75% and reduces the mean unsafe-response rate across three broader harmfulness benchmarks from 26.26% to 0.14%, but increases XSTest over-refusal from 2.00% to 74.00%. In a separate matched comparison, replacing external responses with verified target-model responses reduces over-refusal from 15.20% to 5.20%. Boundary-pair data reduces comply-side over-refusal on held-out pairs from 32.94% to 4.16%, while harmful-side refusal decreases only from 91.88% to 87.72%. These results show that data composition controls the safety and usability trade-off, and that safety alignment should be evaluated on both sides of the intended refusal boundary.

Alejo López-Ávila, Iker García-Ferrero, Jezabel Garcia +2

Multiverse Computing

LLM Safety AlignmentRefusal Behavior in Language ModelsLLM Refusal BehaviorLanguage Model Safety EvaluationSelf-Distillation
AskPDF