cs.AIOct 31, 2026

Inherited or Learned? Backdoor Selectivity in Distilled Language Models

Authors: Dipto Sumit, Sanzar Rahman, Ramisa Faria Puspita, Samiha Fairooz Urbi, Farig Sadeque

Organizations: BRAC University, Dhaka, Bangladesh

Abstract

Distillation is a routine step in building small language models, and a backdoored teacher can pass its backdoor to the student. We study what a student inherits when the backdoor is a rare string that triggers a fixed refusal. Keeping the trigger fixed, we vary only where it appears, at a random word boundary or appended to the instruction, and compare teachers and students on the exact trigger, six control strings and clean inputs. Across Qwen2.5, Llama 3.2 and Gemma 3, with 5% of transfer prompts triggered, randomly inserted triggers become less selective after distillation in all nine teacher-student pipelines, while appended triggers gain selectivity in Qwen and Llama and change little in Gemma. Students trained on the same prompts without a teacher show that much of this is not inherited. Llama and Gemma students learn an appended backdoor from the 150 poisoned examples alone, and only Qwen's 0.5B student needs the teacher. The data also sets where the trigger works. Gemma teachers respond to it anywhere, but their students only when it is appended. Audits should compare teacher and student on the same inputs, probe several trigger positions and include a student trained without the teacher.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. On-policy distillation as a Backdoor Propagation Channel

    Nov 14, 2026Zane Fu, Yanzhao Zheng, Ziting Qin +8LLM Backdoor AttacksOn-Policy Distillation

  2. Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour

    Oct 19, 2026Jonibek Mansurov, Sholpan Bolatzhanova, Filbert Aurelian Tjiaranta +2Language Model DistillationKnowledge Distillation

  3. Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

    Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM WatermarkingKnowledge Distillation