Organizations: Alibaba Group · Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences · Jilin University · Beijing University of Posts and Telecommunications · Taobao (China) Software Co., Ltd.
On-policy distillation (OPD) is emerging as an important paradigm for the post training of large language models, improving student models by leveraging supervision generated online by teacher models. However, whether this process also transfers latent security risks from teacher models to student models remains insufficiently studied. In this work, we reveal a new security risk in which an attacker first implants a conditional backdoor into the teacher model through supervised fine tuning (SFT), after which OPD can transfer the backdoor to the student model with only an extremely small proportion of trigger prompt exposure. More concerningly, we find that such backdoor transfer is substantially stronger under OPD than under conventional distillation methods, suggesting that OPD may provide an efficient channel for propagating backdoors across models. Our mechanistic analysis shows that the backdoor signal is primarily transmitted through anomalous differences in the conditional probabilities produced by the teacher model for triggered and clean inputs, and is gradually internalized into the internal representations of the student model during training. Furthermore, we investigate how to suppress such anomalous conditional probability signals at the source of distillation supervision and, based on this analysis, propose an OPD training method tailored to this conditional backdoor propagation setting to mitigate the risk of backdoor transfer induced by untrusted teacher supervision.
Figures & tables
Figure 1: Backdoor Transfer via OPD . A conditional backdoor is implanted in the teacher, then a few triggered prompts are mixed into the prompt stream. Through OPD, the student inherits the conditional behavior and exhibits the target behavior on triggered prompts.
Figure 2: Example of backdoor activation in the student model, where each individual attack uses a fixed trigger.
Figure 3: Conditional backdoor propagation trends across different model families and scales. Both Unfiltered OPD and Filtered-Rollout OPD show that, as the trigger exposure rate increases, ASR and Conditional Gap rise rapidly, while FTR remains generally low.
Figure 4: We evaluate backdoor propagation across three target behaviors, including label hijacking , language style , and target preference , using four Teacher–Student configurations spanning different model scales and families. All student models are trained under a fixed 2% trigger exposure rate to measure the conditional control of the trigger over the target behavior across scenarios.
Method
ASR (%)
FTR (%)
Cond. Gap (pp)
OPD (Reverse-KL)
89.2±8.8
15.8±13.8
73.3±11.8
Sequence KD
56.7±10.4
6.7±2.9
50.0±13.2
Vanilla KD
61.7±7.6
8.3±2.9
53.3±5.8
Offline Reverse-KL
53.3±2.9
16.7±12.6
36.7±10.4
Table 1: Under a fixed 2% trigger exposure rate, we compare distillation methods in the safety backdoor setting using Llama-3.2-3B as both Teacher and Student.
Figure 5: We compare three supervision conditions, Original , Removed , and Misdirected , across four Teacher → Student architectures under reverse-kl OPD with 2% trigger exposure. Original retains the original teacher supervision. Removed eliminates the probability differences between clean and triggered inputs, while Misdirected preserves the magnitude of these differences but disrupts their directional correspondence with the input conditions.
Figure 6: In the Llama-3.2-3B → 3B OPD experiment with 2% trigger exposure, we track behavioral metrics over 0–150 training steps and the layer-16 clean/trigger representation residual projected onto the normalized Teacher backdoor direction.
Method
ASR(%)
FTR(%)
Cond. Gap (pp) ↓
Undefended Baseline
89.17±8.78
15.83±13.77
73.33±11.81
Difference Gated Distillation (Ours)
15.00±2.50
5.00±2.50
10.00±4.33
ONION ( Qi et al., 2021a )
30.83±8.78
21.67±3.82
9.17±7.22
Fine-mixing ( Zhang et al., 2022 )
40.83±6.29
23.33±3.82
17.50±6.61
Fine-pruning ( Liu et al., 2018 )
50.83±8.04
25.00±4.33
25.83±7.64
Clean fine-tuning ( Sha et al., 2022 )
55.00±9.01
31.67±5.20
23.33±13.75
Table 2: Defense comparison on harmful content for Llama-3.2-3B → 3B with 2% trigger exposure. Our method uses τ=0.0025 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Setting
Value
LoRA
r=16 , α=32 , dropout =0.05
Optimizer
AdamW
LR
1×10−4
Batch / Epoch
16 / 5
Scheduler
cosine, 5% warmup
Precision
BF16
Appendix
Table 3: Hyperparameters used for SFT of the backdoored teacher model.
Setting
Value
Setting
Value
Distillation Objective
reverse-kl
Probability Space
Full vocabulary
Trigger Exposure
0 / 0.25 / 0.5 / 1 / 2 / 3 / 5%
Learning Rate
1×10−5
Global Batch Size
64
Training Epochs
1
Appendix
Table 4: Hyperparameters and implementation settings for reverse-kl OPD.
[1pt] τ
Safety Metrics
General Capability
ASR (%)
FTR (%)
Cond. Gap (pp)
IFEval (%)
MMLU (%)
GSM8K (%)
0
6.67±1.44
4.17±2.89
2.50±2.50
55.67±0.58
53.22±1.83
24.33±14.01
0.0025
15.00±2.50
5.00±2.50
10.00±4.33
71.26±1.78
59.84±1.51
35.24±8.16
0.005
24.17±5.20
8.33±3.82
15.83±7.64
73.13±3.61
61.70±0.94
33.11±4.48
0.01
31.67±8.78
5.83±5.20
25.83±3.82
75.01±2.58
62.76±1.38
40.27±8.19
0.02
40.00±7.50
9.17±1.44
30.83±7.64
78.11±3.41
64.25±0.91
35.35±8.18
Appendix
Table 5: Under 2% trigger exposure in the harmful-content setting, we evaluate the defense effectiveness and general capability of our method across different gating thresholds τ , using Llama-3.2-3B as both the teacher and student.
Large language model backdoor attacks implant hidden, trigger-activated behaviors through poisoned data, model changes, tools, or deployment pipelines.