Organizations: Alibaba Group · Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences · Jilin University · Beijing University of Posts and Telecommunications · Taobao (China) Software Co., Ltd.
On-policy distillation (OPD) is emerging as an important paradigm for the post training of large language models, improving student models by leveraging supervision generated online by teacher models. However, whether this process also transfers latent security risks from teacher models to student models remains insufficiently studied. In this work, we reveal a new security risk in which an attacker first implants a conditional backdoor into the teacher model through supervised fine tuning (SFT), after which OPD can transfer the backdoor to the student model with only an extremely small proportion of trigger prompt exposure. More concerningly, we find that such backdoor transfer is substantially stronger under OPD than under conventional distillation methods, suggesting that OPD may provide an efficient channel for propagating backdoors across models. Our mechanistic analysis shows that the backdoor signal is primarily transmitted through anomalous differences in the conditional probabilities produced by the teacher model for triggered and clean inputs, and is gradually internalized into the internal representations of the student model during training. Furthermore, we investigate how to suppress such anomalous conditional probability signals at the source of distillation supervision and, based on this analysis, propose an OPD training method tailored to this conditional backdoor propagation setting to mitigate the risk of backdoor transfer induced by untrusted teacher supervision.
Figures & tables
Figure 1: Backdoor Transfer via OPD . A conditional backdoor is implanted in the teacher, then a few triggered prompts are mixed into the prompt stream. Through OPD, the student inherits the conditional behavior and exhibits the target behavior on triggered prompts.
Figure 2: Example of backdoor activation in the student model, where each individual attack uses a fixed trigger.
Figure 3: Conditional backdoor propagation trends across different model families and scales. Both Unfiltered OPD and Filtered-Rollout OPD show that, as the trigger exposure rate increases, ASR and Conditional Gap rise rapidly, while FTR remains generally low.
Figure 4: We evaluate backdoor propagation across three target behaviors, including label hijacking , language style , and target preference , using four Teacher–Student configurations spanning different model scales and families. All student models are trained under a fixed 2% trigger exposure rate to measure the conditional control of the trigger over the target behavior across scenarios.
Method
ASR (%)
FTR (%)
Cond. Gap (pp)
OPD (Reverse-KL)
89.2±8.8
15.8±13.8
73.3±11.8
Sequence KD
56.7±10.4
6.7±2.9
50.0±13.2
Vanilla KD
61.7±7.6
8.3±2.9
53.3±5.8
Offline Reverse-KL
53.3±2.9
16.7±12.6
36.7±10.4
Table 1: Under a fixed 2% trigger exposure rate, we compare distillation methods in the safety backdoor setting using Llama-3.2-3B as both Teacher and Student.
Figure 5: We compare three supervision conditions, Original , Removed , and Misdirected , across four Teacher → Student architectures under reverse-kl OPD with 2% trigger exposure. Original retains the original teacher supervision. Removed eliminates the probability differences between clean and triggered inputs, while Misdirected preserves the magnitude of these differences but disrupts their directional correspondence with the input conditions.
Figure 6: In the Llama-3.2-3B → 3B OPD experiment with 2% trigger exposure, we track behavioral metrics over 0–150 training steps and the layer-16 clean/trigger representation residual projected onto the normalized Teacher backdoor direction.
Method
ASR(%)
FTR(%)
Cond. Gap (pp) ↓
Undefended Baseline
89.17±8.78
15.83±13.77
73.33±11.81
Difference Gated Distillation (Ours)
15.00±2.50
5.00±2.50
10.00±4.33
ONION ( Qi et al., 2021a )
30.83±8.78
21.67±3.82
9.17±7.22
Fine-mixing ( Zhang et al., 2022 )
40.83±6.29
23.33±3.82
17.50±6.61
Fine-pruning ( Liu et al., 2018 )
50.83±8.04
25.00±4.33
25.83±7.64
Clean fine-tuning ( Sha et al., 2022 )
55.00±9.01
31.67±5.20
23.33±13.75
Table 2: Defense comparison on harmful content for Llama-3.2-3B → 3B with 2% trigger exposure. Our method uses τ=0.0025 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Setting
Value
LoRA
r=16 , α=32 , dropout =0.05
Optimizer
AdamW
LR
1×10−4
Batch / Epoch
16 / 5
Scheduler
cosine, 5% warmup
Precision
BF16
Appendix
Table 3: Hyperparameters used for SFT of the backdoored teacher model.
Setting
Value
Setting
Value
Distillation Objective
reverse-kl
Probability Space
Full vocabulary
Trigger Exposure
0 / 0.25 / 0.5 / 1 / 2 / 3 / 5%
Learning Rate
1×10−5
Global Batch Size
64
Training Epochs
1
Appendix
Table 4: Hyperparameters and implementation settings for reverse-kl OPD.
[1pt] τ
Safety Metrics
General Capability
ASR (%)
FTR (%)
Cond. Gap (pp)
IFEval (%)
MMLU (%)
GSM8K (%)
0
6.67±1.44
4.17±2.89
2.50±2.50
55.67±0.58
53.22±1.83
24.33±14.01
0.0025
15.00±2.50
5.00±2.50
10.00±4.33
71.26±1.78
59.84±1.51
35.24±8.16
0.005
24.17±5.20
8.33±3.82
15.83±7.64
73.13±3.61
61.70±0.94
33.11±4.48
0.01
31.67±8.78
5.83±5.20
25.83±3.82
75.01±2.58
62.76±1.38
40.27±8.19
0.02
40.00±7.50
9.17±1.44
30.83±7.64
78.11±3.41
64.25±0.91
35.35±8.18
Appendix
Table 5: Under 2% trigger exposure in the harmful-content setting, we evaluate the defense effectiveness and general capability of our method across different gating thresholds τ , using Llama-3.2-3B as both the teacher and student.
Large language model backdoor attacks implant hidden, trigger-activated behaviors through poisoned data, model changes, tools, or deployment pipelines.
On-policy distillation (OPD) has attracted growing attention as an effective way to transfer capabilities from teacher models to student models. Recent studies further explore OPD as a tool for improving large language model safety with promising results. However, these approaches typically assume that the teacher and training data are trustworthy. In this paper, we uncover an overlooked threat to OPD for safety: a safety-aligned but backdoored teacher can propagate its hidden malicious behavior to an initially clean student. Under our threat model, a poisoning rate as low as 3% results in an attack success rate (ASR) of up to 70% on the distilled student. We further identify two training choices that can amplify this risk. First, increasing the number of training epochs can lead to high ASR even at low poisoning rates. With only 10 poisoned samples, ASR reaches 67% after 16 epochs. Second, the commonly used top-k KL can accelerate backdoor transfer, causing trigger-conditioned harmful behavior to emerge earlier than sampled-token KL in most settings. Alongside these findings, we explore a simple mitigation, Lazy Defense, which clips KL rewards to make student updates less aggressive, limiting aggressive updates and slowing backdoor learning. Experiments show that Lazy Defense delays backdoor transfer in low poisoning rate settings. Together, our findings reveal that OPD can propagate backdoors, highlighting the need to address the safety risks of OPD.
Conventional knowledge distillation can facilitate the propagation of backdoors from teacher models to student models. However, how this risk manifests in On-Policy Distillation (OPD) remains insufficiently studied. OPD has recently emerged as an important post-training paradigm for Vision-Language Models (VLMs). Unlike traditional offline distillation, OPD does not require teacher-generated responses. Instead, it uses the teacher's probability distributions to supervise response trajectories generated online by the student. Since the initial student model typically generates benign responses and receives no explicit supervision from malicious target responses, OPD appears to offer some inherent resistance to backdoor propagation. In this paper, we challenge this intuition and reveal a more subtle risk of visual backdoor propagation in OPD. Through experiments across multiple model families and datasets, together with theoretical analysis, we identify three main findings. First, unlike the relatively gradual propagation observed in traditional offline distillation, backdoors rarely emerge in OPD at low poisoning ratios. However, the attack success rate rapidly approaches 100% once a critical threshold is exceeded. Second, backdoor propagation does not depend on explicit malicious responses. Instead, it is driven by the continuous accumulation of anomalous probability mass assigned by the teacher. Finally, based on these findings, we propose a novel trigger design that enables visual backdoor propagation with zero explicit trigger injection. These findings further reveal an overlooked mechanism of backdoor propagation in OPD and highlight its potential security risks.
Zane Fu, Yanzhao Zheng, Jinze Liu +7
Alibaba Group · Guangxi University · Wuhan University of Technology +2
Distillation is a routine step in building small language models, and a backdoored teacher can pass its backdoor to the student. We study what a student inherits when the backdoor is a rare string that triggers a fixed refusal. Keeping the trigger fixed, we vary only where it appears, at a random word boundary or appended to the instruction, and compare teachers and students on the exact trigger, six control strings and clean inputs. Across Qwen2.5, Llama 3.2 and Gemma 3, with 5% of transfer prompts triggered, randomly inserted triggers become less selective after distillation in all nine teacher-student pipelines, while appended triggers gain selectivity in Qwen and Llama and change little in Gemma. Students trained on the same prompts without a teacher show that much of this is not inherited. Llama and Gemma students learn an appended backdoor from the 150 poisoned examples alone, and only Qwen's 0.5B student needs the teacher. The data also sets where the trigger works. Gemma teachers respond to it anywhere, but their students only when it is appended. Audits should compare teacher and student on the same inputs, probe several trigger positions and include a student trained without the teacher.