On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.
Figures & tables
Figure 1: On-policy distillation with poisoned data and backdoored teacher.
Figure 2: Backdoor transfer in same-model OPD across poisoning rates. Results are shown for Qwen2.5 3B→3B and 7B→7B teacher–student pairs. (a) and (b) report Trigger ASR and Clean ASR, respectively, averaged across five safety datasets.
Figure 3: Backdoor transfer in cross-size OPD across poisoning rates. Results are shown for Qwen2.5 3B→1.5B and 7B→3B teacher–student pairs. (a) and (b) report Trigger ASR and Clean ASR, respectively, averaged across five safety datasets.
Figure 4: Backdoor transfer in same-size OPD across training epochs at different poisoning rates. (a) and (b) report Qwen2.5 3B→3B and 7B→7B teacher–student pairs, respectively.
Figure 5: Backdoor transfer in cross-size OPD across training epochs at different poisoning rates. (a) and (b) report Qwen2.5 3B→3B and 7B→7B teacher–student pairs, respectively.
Figure 6: Jaccard Similarity and Trigger ASR across training epochs at different poisoning rates. Results are shown for Qwen2.5 3B→3B teacher–student pair.
Figure 7: Backdoor transfer for top- k and sampled token KL. Results are shown for Qwen2.5 model family. k value ranges 4 and 16. Measured by Trigger ASR.
Figure 8: Backdoor transfer of Lazy Defense compared with top- k and Sampled Token across training epochs on Qwen2.5. Measured by Trigger ASR.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Trigger ASR / Clean ASR
Average
DAN
DNA
Addition
StrongREJECT
AdvBench
Qwen2.5-3B-Instruct
94.87 / 7.18
98.84 / 0.00
88.69 / 9.05
94.90 / 4.46
61.54 / 1.15
87.77 / 4.37
Qwen2.5-7B-Instruct
97.95 / 3.08
98.84 / 0.00
90.95 / 0.90
87.90 / 1.27
96.15 / 0.38
94.36 / 1.13
Appendix
Table 1: Performance of backdoored teachers before OPD. Each cell reports Trigger / Clean ASR (%). Average is the unweighted mean across the five datasets, computed separately for each metric.
Model
Trigger ASR / Clean ASR
Average
DAN
DNA
Addition
StrongREJECT
AdvBench
Qwen2.5-1.5B-Instruct
12.3 / 13.3
14.5 / 17.4
36.2 / 35.3
45.9 / 45.2
15.4 / 18.8
24.9 / 26.0
Qwen2.5-3B-Instruct
21.5 / 19.5
27.3 / 26.2
42.1 / 43.4
52.2 / 54.1
19.2 / 21.9
32.5 / 33.0
Qwen2.5-7B-Instruct
13.3 / 10.8
11.6 / 9.9
34.4 / 37.6
17.2 / 17.8
4.6 / 6.5
16.2 / 16.5
Appendix
Table 2: Trigger / Clean ASR (%) of clean student models before OPD, expressed as percentages. Average is the unweighted mean across the five datasets.
On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.
Siqi Zhu, Xuyan Ye, Hongyu Lu +2
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
On-Policy Self-Distillation (OPSD) has emerged as a promising paradigm for safety alignment, delivering dense, token-level supervision by distilling from a teacher conditioned on refusal-oriented privileged prompts. However, we reveal that this paradigm suffers from critical inefficiencies that degrade both training efficiency and general reasoning capabilities. Specifically, we diagnose two fundamental bottlenecks: (1) supervisory collapse over extended rollouts, where the teacher's corrective efficacy degrades precipitously as the student's generation prefix lengthens, injecting noisy gradients into late-stage tokens; and (2) gradient dilution from stylistic shifts, where the distillation objective is dominated by safety-irrelevant stylistic discrepancies induced by privileged prompting, washing out genuine safety signals and impairing base reasoning. To resolve these issues, we propose Efficient On-Policy Self-Distilled Safety Alignment (EOPSA), which concentrates computational and gradient budgets exclusively on reliably supervised, safety-critical tokens. EOPSA incorporates two coordinated mechanisms: (i) Adaptive Rollout Scheduling, which dynamically bounds the generation horizon guided by a novel Teacher Rescue Rate (TRR) metric to operate strictly within reliable supervision regimes; and (ii) Selective Distillation, which filters out safety-neutral tokens to restrict gradient updates exclusively to safety-pivotal transitions. Extensive evaluations across reasoning models up to 32B parameters demonstrate that EOPSA slashes rollout computation by ∼50% and backpropagates through merely ∼2% of tokens, consistently outperforming full-token distillation baselines in both safety compliance and reasoning retention.
Qirui Liu, Yichen Sun, Yan Wang +5
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group
On-policy self-distillation (OPSD) has emerged as an efficient post-training paradigm by using a teacher conditioned on privileged information to provide dense token-level supervision. Prior work has shown that OPSD can collapse in verifiable reasoning tasks, but safety alignment differs in that it is guided by high-level constitutions rather than explicit target answers, making it a natural setting to revisit dense distillation. However, our pilot study show that safety OPSD still suffers from severe collapse: constitutional conditioning contracts the teacher distribution toward short and overly conservative responses, and Reverse KL further amplifies this contraction into reduced expressiveness. We formalize this effect as geometric leakage under safety boundaries in a non-orthogonal semantic space, where safety pressure transfers into the expressiveness dimension. Based on this analysis, we propose Constitutional On-Policy Safe Distillation (COPSD), which first calibrates the teacher through a Cross-SFT cold-start and then performs constitution-conditioned on-policy distillation. Experiments on 12 benchmarks show that COPSD achieves a consistently stronger safety--helpfulness trade-off than baselines while substantially reducing the safety tax on general reasoning ability.
Ming Wen, Yuxuan Liu, Kun Yang +8
Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Innovation Institute · 3Ant Group +3