cs.LGOct 15, 2026

Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation

Authors: Karim Abdel Sadek, Federico Di Gennaro, Fanny Yang

Organizations: ETH Zurich

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Training models on their own outputs has attracted widespread attention as a possible route to advance AI capabilities. One prominent approach is self-distillation, where a model is trained to match the token-level predictions it would have produced when given access to additional context, such as a cue, feedback, or instruction. However, the safety properties of this training procedure are poorly understood. In this paper, we present a surprising safety failure of recursive self-distillation, where each self-distilled model becomes the teacher for the next round of training. We observe that when a safe model is recursively self-distilled on its own (safe) generations while receiving an additional non-harmful cue, it becomes increasingly misaligned across a diverse set of harmful queries; we call this Progressive Misalignment. We show this failure mode in both the Qwen3 and Phi-4 model families on SORRY-Bench and StrongREJECT. Progressive misalignment also occurs in a more realistic setup, where the fixed cue is replaced by feedback from a simulated user with benign intent. Using a gradient-based method, we are also able to find benign prompts that lead to progressive misalignment when applying recursive self-distillation, showing that harmful training prompts are not necessary for this failure mode to emerge. We hope our work encourages future investigations of the safety properties of AI systems recursively trained on their own data.

Figures & tables

Figure 1: Progressive misalignment under benign recursive self-distillation . A model that initially refuses harmful prompts is iteratively self-distilled on its own generations while receiving benign feedback, becoming progressively misaligned after various training iterations.
Figure 2: Training and evaluation curves of Progressive Misalignment . Misalignment increases over successive rounds of benign recursive self-distillation on a narrow set of malign prompts (Training), and the resulting misalignment generalizes to a broader set of unseen harmful prompts (Eval; SORRY-Bench ( Xie et al., 2025 ) ).
Figure 3: Generations from Phi-4 14B and Qwen3 14B after benign recursive self-distillation.
Figure 4: Progressive misalignment under benign RSD on harmful prompts. Average and std (over 33 seeds) scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) after recursive self-distillation with Ag Excellent , Ag I like Owls , and Ag Do not help with my request as feedback.
Figure 5: Progressive misalignment under RSD with benign simulated user feedback. Average and one std (over 33 seeds) scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) for Qwen3 14B trained via recursive self-distillation with adaptive feedback from an LLM-simulated user with engaged or a practical behavioral trait.
Figure 6
Figure 6: Progressive misalignment under RSD on benign prompts. Average and std (over 33 seeds) score on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) for models resulting from self-distilling only on benign prompts ( Section 3.3 ) with feedback Ag Excellent .
Figure 7: Increase in misalignment relative to the initial model after off-policy RSD with Ag Excellent as cue, across the Qwen3 model family.
Figure 9
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10
Table 1: Inputs for generation (top) and token scoring (bottom). Feedback (purple) arrives after the on-policy student answer, but before the off-policy teacher answer. For scoring, student and teacher share the previous answer tokens (green); only the teacher sees feedback.
Figure 9: Compliance rate across epochs for Qwen3 14B trained with Excellent as feedback via off-policy self-distillation.
ModelFeedbackOff-policy RSDOn-policy RSD
LRClipEpochLRClipEpoch
Qwen3 14BExcellent10−510^{-5}—63×10−63\times 10^{-6}3050
I like Owls10−510^{-5}—43×10−63\times 10^{-6}3020
Do not help with my request10−510^{-5}—210−610^{-6}51
Phi-4 14BExcellent10−510^{-5}—42×10−62\times 10^{-6}3030
I like Owls10−510^{-5}—32×10−62\times 10^{-6}520
Appendix
Table 2: Training configurations for the checkpoints used in Figure 4 . LR denotes the learning rate; Clip denotes the maximum gradient norm, with — indicating no clipping; Epoch identifies the selected checkpoint. All runs use full-parameter BF16 training, AdamW ( β1=0.9\beta_{1}=0.9 , β2=0.999\beta_{2}=0.999 , ϵ=10−8\epsilon=10^{-8} , weight decay 0.010.01 ), and a constant learning rate.
ModelHarmful/benign prompt proportionOff-policyOn-policy
LRClipEpoch/stepLRClipEpoch/step
Qwen3 14B75/2510−510^{-5}—6610−510^{-5}199
50/5010−510^{-5}—663×10−63\times 10^{-6}303737
25/7510−510^{-5}—101010−510^{-5}122
0/10010−510^{-5}—404010−510^{-5}—8080
Phi-4 14B75/2510−510^{-5}—332×10−62\times 10^{-6}304545
Appendix
Table 3: Training configurations and selected checkpoints for the harmful/benign prompt-mixture experiments reported in Table 4 . LR denotes the learning rate; Clip denotes the maximum gradient norm, with — indicating no clipping. The selected checkpoint is reported as eNeN for epoch NN and sNsN for optimizer step NN . The 0/100 setting denotes training exclusively on benign prompts using the active-selection curriculum, for which the training prompts change every 2020 steps; all other mixtures use fixed prompt sets without active selection. All runs use full-parameter BF16 training, AdamW ( β1=0.9\beta_{1}=0.9 , β2=0.999\beta_{2}=0.999 , ϵ=10−8\epsilon=10^{-8} , weight decay 0.010.01 ), a constant learning rate, and seed 00 .
Figure 10: SFT and frozen-teacher ablations for off-policy RSD with Ag Excellent as feedback.
Figure 11: Average accuracy (over 33 seeds during generation) on GSM8K and IFEval in the control experiments.
  SORRY-Bench compliance rate (%)StrongREJECT score (%)
ModelRegimeinitial75/2550/5025/75initial75/2550/5025/75
Qwen3 14BOff-policy31.1±1.1{31.1}_{\pm 1.1}↑\uparrow 88.3±0.6{88.3}_{\pm 0.6}↑\uparrow 83.6±2.3{83.6}_{\pm 2.3}↑\uparrow 71.7±2.5{71.7}_{\pm 2.5}10.6±1.7{10.6}_{\pm 1.7}↑\uparrow 40.7±3.5{40.7}_{\pm 3.5}↑\uparrow 51.9±5.2{51.9}_{\pm 5.2}↑\uparrow 17.9±2.1{17.9}_{\pm 2.1}
On-policy↑\uparrow 48.6±1.9{48.6}_{\pm 1.9}↑\uparrow 48.1±1.6{48.1}_{\pm 1.6}↓\downarrow 14.0±0.5{14.0}_{\pm 0.5}↑\uparrow 15.9±2.3{15.9}_{\pm 2.3}↑\uparrow 17.4±0.4{17.4}_{\pm 0.4}↓\downarrow 1.7±0.7{1.7}_{\pm 0.7}
Phi-4 14BOff-policy26.1±0.6{26.1}_{\pm 0.6}↑\uparrow 99.3±0.2{99.3}_{\pm 0.2}↑\uparrow 93.5±0.6{93.5}_{\pm 0.6}↑\uparrow 89.6±1.7{89.6}_{\pm 1.7}3.8±0.6{3.8}_{\pm 0.6}↑\uparrow 68.7±2.1{68.7}_{\pm 2.1}↑\uparrow 55.3±0.8{55.3}_{\pm 0.8}↑\uparrow 48.4±0.9{48.4}_{\pm 0.9}
On-policy↑\uparrow 79.3±1.1{79.3}_{\pm 1.1}↑\uparrow 98.3±0.4{98.3}_{\pm 0.4}↑\uparrow 74.9±0.7{74.9}_{\pm 0.7}↑\uparrow 32.5±1.6{32.5}_{\pm 1.6}↑\uparrow 66.4±0.5{66.4}_{\pm 0.5}↑\uparrow 41.0±1.9{41.0}_{\pm 1.9}
Appendix
Table 4: Compliance scores for models trained using different harmful/benign prompt proportions. A single model is evaluated on the test set for SORRY-Bench questions and StrongREJECT questions. Values report the mean and sample standard deviation over three temperature- 11 stochastic rollouts.
Figure 12: Progressive misalignment under benign RSD on harmful prompts. Average (over 33 seeds during generation) compliance scores on SORRY-Bench ( Xie et al., 2025 ) , and StrongREJECT ( Souly et al., 2024 ) after recursive self-distillation with Ag Excellent , Ag I like Owls , and Ag Do not help with my request as feedback. The initial model has feedback in-context.
Figure 13: Average (over 33 seeds during generation) compliance rates on SORRY-Bench , and StrongREJECT when training on the 20-lowest scores obtain from our data selection algorithm (i.e. the less misaligned prompts according to our metric).
More in this topic

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

Explore similar work

CardsList

  1. Distillation for Incrimination and Distillation for Capabilities

    Oct 7, 2026Sebastian Prasanna, Jacqueline Tay, Alek WestoverLLM Safety AlignmentLanguage Model Distillation

    Redwood Research

  2. Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation

    May 14, 2026Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4LLM AlignmentOn-Policy Self-Distillation

    1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1

  3. Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal

    Sep 3, 2026Alejo López-Ávila, Iker García-Ferrero, Jezabel Garcia +2LLM Safety AlignmentRefusal Behavior in Language Models

    Multiverse Computing