cs.LGNov 13, 2026

Antidistillation is Cheap: Lightweight Distillation-Resistance via Stylistic Corruption

Authors: Anh Tu Ngo, Anupam Chattopadhyay

Organizations: College of Computing and Data Science Nanyang Technological University Singapore

Abstract

Knowledge distillation (KD) is a highly effective paradigm for transferring capabilities from large language models (LLMs) to smaller, more specialized student models. In commercial deployment where frontier models are restricted behind black-box APIs, sequence-level distillation remains a potent vector for competitors to clone proprietary capabilities at a fraction of the development cost. Existing defenses mostly rely on surrogate models to manipulate next-token sampling, which increases computational overhead and real-time inference latency. In this work, we propose a lightweight defense that dynamically injects style-steering vectors into the teacher model's activations upon malicious queries. This framework offers exceptional flexibility, allowing hosts to customize target defensive styles and dynamically blend them to maximize stylistic variance. Crucially, this inference-time intervention preserves semantic utility and fluency during malicious queries, misleading adversaries into believing that they are extracting pristine teacher outputs, while standard consumers continue to receive original, unmodified responses. Our experiments demonstrate that this framework successfully disrupts student learning, achieving substantial anti-distillation performance while incurring negligible overhead and preserving text naturalness. Our code is available at https://anonymous.4open.science/r/distill-resist-38C6/.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

    Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM WatermarkingKnowledge Distillation

  2. Distillation Defenses Easily Break After Reinforcement Learning

    Sep 28, 2026Shidan Javaheri, Alexander Panfilov, Oliver Britton +2LLM SecurityLanguage Model Distillation