Does On-Policy Distillation for Safety Pose Backdoor Risks?

Jian Luo; Kehan Qi; Qingqiao Hu; Meilong Xu; Jiacheng Qiu; Weimin Lyu; Jiawei Zhou; Chao Chen

cs.LGOct 6, 2026

Does On-Policy Distillation for Safety Pose Backdoor Risks?

Authors: Jian Luo, Kehan Qi, Qingqiao Hu, Meilong Xu, Jiacheng Qiu, Weimin Lyu, Jiawei Zhou, Chao Chen

Organizations: Stony Brook University · Amazon

Abstract

On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.

Figures & tables

Figure 1: On-policy distillation with poisoned data and backdoored teacher.

Panel of Figure 2 — Figure 2: Backdoor transfer in same-model OPD across poisoning rates. Results are shown for Qwen2.5 $3\mathrm{B}\rightarrow 3\mathrm{B}$ and $7\mathrm{B}\rightarrow 7\mathrm{B}$ teacher–student pairs. (a) and (b) report Trigger ASR and Clean ASR, respectively, averaged across five safety datasets.

Panel of Figure 3 — Figure 3: Backdoor transfer in cross-size OPD across poisoning rates. Results are shown for Qwen2.5 $3\mathrm{B}\rightarrow 1.5\mathrm{B}$ and $7\mathrm{B}\rightarrow 3\mathrm{B}$ teacher–student pairs. (a) and (b) report Trigger ASR and Clean ASR, respectively, averaged across five safety datasets.

Panel of Figure 4 — Figure 4: Backdoor transfer in same-size OPD across training epochs at different poisoning rates. (a) and (b) report Qwen2.5 $3\mathrm{B}\rightarrow 3\mathrm{B}$ and $7\mathrm{B}\rightarrow 7\mathrm{B}$ teacher–student pairs, respectively.

Panel of Figure 5 — Figure 5: Backdoor transfer in cross-size OPD across training epochs at different poisoning rates. (a) and (b) report Qwen2.5 $3\mathrm{B}\rightarrow 3\mathrm{B}$ and $7\mathrm{B}\rightarrow 7\mathrm{B}$ teacher–student pairs, respectively.

Panel of Figure 6 — Figure 6: Jaccard Similarity and Trigger ASR across training epochs at different poisoning rates. Results are shown for Qwen2.5 $3\mathrm{B}\rightarrow 3\mathrm{B}$ teacher–student pair.

Figure 7: Backdoor transfer for top-

k

and sampled token KL. Results are shown for Qwen2.5 model family.

k

value ranges 4 and 16. Measured by Trigger ASR.

Figure 8: Backdoor transfer of Lazy Defense compared with top-

k

and Sampled Token across training epochs on Qwen2.5. Measured by Trigger ASR.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Trigger ASR / Clean ASR					Average
Model	DAN	DNA	Addition	StrongREJECT	AdvBench	Average
Qwen2.5-3B-Instruct	94.87 / 7.18	98.84 / 0.00	88.69 / 9.05	94.90 / 4.46	61.54 / 1.15	87.77 / 4.37
Qwen2.5-7B-Instruct	97.95 / 3.08	98.84 / 0.00	90.95 / 0.90	87.90 / 1.27	96.15 / 0.38	94.36 / 1.13

Appendix

Table 1: Performance of backdoored teachers before OPD. Each cell reports Trigger / Clean ASR (%). Average is the unweighted mean across the five datasets, computed separately for each metric.

Model	Trigger ASR / Clean ASR					Average
Model	DAN	DNA	Addition	StrongREJECT	AdvBench	Average
Qwen2.5-1.5B-Instruct	12.3 / 13.3	14.5 / 17.4	36.2 / 35.3	45.9 / 45.2	15.4 / 18.8	24.9 / 26.0
Qwen2.5-3B-Instruct	21.5 / 19.5	27.3 / 26.2	42.1 / 43.4	52.2 / 54.1	19.2 / 21.9	32.5 / 33.0
Qwen2.5-7B-Instruct	13.3 / 10.8	11.6 / 9.9	34.4 / 37.6	17.2 / 17.8	4.6 / 6.5	16.2 / 16.5

Appendix

Table 2: Trigger / Clean ASR (%) of clean student models before OPD, expressed as percentages. Average is the unweighted mean across the five datasets.

Explore similar work

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2Kullback-Leibler Divergence
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
Sep 28, 2026Qirui Liu, Yichen Sun, Yan Wang +5Safety Alignment Collapse
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group
Constitutional On-Policy Safe Distillation
Jun 2, 2026Ming Wen, Yuxuan Liu, Kun Yang +8Unsupervised On-Policy Self-Distillation
Institute of Trustworthy Embodied AI, Fudan University · 2Shanghai Innovation Institute · 3Ant Group +3

May 11, 2026, cs.AIJ/K move · Enter open · S save

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi+1

1UIUC · 2Renmin University of China · †Work done during an internship at UIUC.+1

Kullback-Leibler Divergence

Open paper AskPDF arXiv

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Abstract

Explore similar work

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

EOPSA: Efficient On-Policy Self-Distilled Safety Alignment

Constitutional On-Policy Safe Distillation