Antidistillation is Cheap: Lightweight Distillation-Resistance via Stylistic Corruption
Organizations: College of Computing and Data Science Nanyang Technological University Singapore
Abstract
Knowledge distillation (KD) is a highly effective paradigm for transferring capabilities from large language models (LLMs) to smaller, more specialized student models. In commercial deployment where frontier models are restricted behind black-box APIs, sequence-level distillation remains a potent vector for competitors to clone proprietary capabilities at a fraction of the development cost. Existing defenses mostly rely on surrogate models to manipulate next-token sampling, which increases computational overhead and real-time inference latency. In this work, we propose a lightweight defense that dynamically injects style-steering vectors into the teacher model's activations upon malicious queries. This framework offers exceptional flexibility, allowing hosts to customize target defensive styles and dynamically blend them to maximize stylistic variance. Crucially, this inference-time intervention preserves semantic utility and fluency during malicious queries, misleading adversaries into believing that they are extracting pristine teacher outputs, while standard consumers continue to receive original, unmodified responses. Our experiments demonstrate that this framework successfully disrupts student learning, achieving substantial anti-distillation performance while incurring negligible overhead and preserving text naturalness. Our code is available at https://anonymous.4open.science/r/distill-resist-38C6/.
Figures & tables
| Property | ADS | DOGe | Ours |
|---|---|---|---|
| Needs proxy model | Yes ✗ | Yes ✗ | No ✓ |
| Weights modification | No ✓ | Yes ✗ | No ✓ |
| Inference overhead | High ✗ | Low ✓ | Low ✓ |
| Flexible on-the-fly adjustment | Yes ✓ | No (retrain) ✗ | Yes ✓ |
| Easy serving w/ vLLM | No ✗ | Yes ✓ | Yes ✓ |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Quality Metric | ADS | DOGe | Ours |
|---|---|---|---|
| Repetition loops ( -gram repeats) | |||
| Weird / multilingual tokens | |||
| Missing final answer |
| Hyperparameter | Value |
|---|---|
| Student Model Fine-Tuning | |
| Epochs | 3 |
| Batch size | 1 |
| Gradient accumulation steps | 16 |
| Learning rate | |
| Max new tokens | 4096 ( MATH ) |
Knowledge Distillation
Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.
See all 876 papers