Adversarial Knowledge Distillation

Follow this topic Ask about this topic

Adversarial knowledge distillation studies how to transfer useful behavior from a teacher model to a smaller or faster student using adversarial training signals. Methods include adversarial example generation, logit and feature distillation, discriminator-based distribution matching, and on-policy feedback for language and video generation. They are evaluated in settings such as CIFAR and ImageNet robustness, certified classification, black-box language-model distillation, and few-step image or video generation. A central challenge is making the transfer stable when teacher and student differ, while balancing clean accuracy with robustness or preserving generation quality with fewer steps.

Latest papers

20 papers6 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

ROAD: Modern Adversarial Training for Simple One-Step Distillation at Scale
Oct 29, 2026Xiaoou Li, Shan You, Tao HuangDiffusion Model Distillation Diffusion Distillation
Shanghai Jiao Tong University (sjtu.edu.cn) · SenseTime (sensetime.com)
Adversarial Self-Distillation
Oct 28, 2026Mohammad Sadegh Akhondzadeh, Soroush H. Zargarbashi, Aleksandar BojchevskiAdversarial Robustness of Language Models Safety-Preserving Fine-Tuning of Language Models
University of Cologne · CISPA Helmholtz Center for Information Security
TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone
Oct 24, 2026Yun Zhu, Da WangVisuomotor Policy Learning Knowledge Distillation
Independent Researcher, China · School of Computer and Information Technology, Shanxi University, Taiyuan, China
CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Oct 14, 2026Zhi Li, Haowei Liu, Hongchen Yang +5Adversarial Robustness Adversarial Knowledge Distillation
Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
Oct 8, 2026Zhi Li, Haowei Liu, Hongchen Yang +4Adversarial Robustness Adversarial Knowledge Distillation
Engineering Research Center of Cyberspace and School of Software and AI, Yunnan University, Kunming, China · School of Information Science and Technology, Yunnan Normal University, Kunming, China
DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation
Oct 1, 2026Zhengming Yu, Junkun Yuan, Haotian Yang +8Diffusion Model Distillation Few-Step Diffusion Sampling
Texas A&M University · Intelligent Creation, ByteDance
Enhancing Autoregressive Video Generation via Representation Adversarial Distillation
Sep 30, 2026Fangyu Lin, Xingtong Ge, Lunjie Zhu +8Long-Horizon Video Generation Diffusion Model Distillation
The Hong Kong University of Science and Technology · Vivix Group Limited · Zhejiang University
Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction
Sep 30, 2026Xiao Cui, Mo Zhu, Yulei Qin +3Group Relative Policy Optimization Language Model Distillation
University of Science and Technology of China · Zhejiang University · Independent Researcher
Persistent Negatives for Adversarial Black-Box On-Policy Distillation
Sep 25, 2026Haixu Ma, Saad Lahrichi, Weiwei Li +11On-Policy Distillation Adversarial Knowledge Distillation
Meta AI · University of Missouri
Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancer
Aug 2, 2026Zhiwei Chen, Yang Hu, Yuxiang Xiao +7Computational Pathology Pathology Foundation Models
School of Computer Science and Engineering, South China University of Technology, China · School of Computing and Mathematical Sciences, University of Leicester, UK · Leicester Cancer Research Centre, University of Leicester, UK +4
Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
Jul 30, 2026Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi +1Adversarial Training Neural Network Robustness
Joint Graduate School of Mathematics for Innovation Kyushu University · Faculty of Information Science and Electrical Engineering Kyushu University · Université Savoie Mont Blanc
Improving Certified Robustness via Adversarial Distillation
Jun 30, 2026Matteo Melis, Jesus Martinez Del Rincon, Vishal SharmaNeural Network Robustness Certified Robustness
School of EEECS Queen’s University Belfast
AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation
Jun 2, 2026Haobo Li, Yanhong Zeng, Yunhong Lu +6Image-to-Video Generation Autoregressive Video Generation
AutoLab, SAI, SJTU · Ant Group · Department of Automation, Tsinghua University +2
On-Policy Adversarial Flow Distillation for Autoregressive Video Generation
May 25, 2026Yang Luo, Shengju Qian, Xiaohang Tang +4Autoregressive Video Generation On-Policy Distillation
National Univesity of Singapore · LIGHTSPEED · University College London
One-Forcing: Towards Stable One-Step Autoregressive Video Generation
May 22, 2026Jiaqi Feng, Justin Cui, Yuanhao Ban +1Autoregressive Video Generation One-Step Generative Modeling
Tsinghua University · UCLA
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
May 21, 2026Hongsin Lee, Hye Won ChungAdversarial Training Neural Network Robustness
School of Electrical Engineering, KAIST, Daejeon, Korea.
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
May 7, 2026Huizi Cui, Huan Ma, Qilin Wang +2Language Model Distillation LLM Uncertainty Estimation
School of Computer Science and Technology, Tianjin University, China · School of Future Technology, Tianjin University, China · Georgia Tech Shenzhen Institute, Tianjin University, China +1
FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation
Apr 12, 2026Yingguang Yang, Hao Liu, Xin Zhang +8Social Bot Detection Federated Learning
School of Cyber science at University of Science and Technology of China, Hefei 230026, China · State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Institute of Data Science at the National University of Singapore, 119077, Singapore +4
Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch
Mar 23, 2026Stella Eva Tsiapali, Cong-Thanh Do, Kate KnillCross-Tokenizer Knowledge Distillation Language Model Distillation
University of Cambridge, Department of Engineering, Cambridge, United Kingdom · Toshiba Europe Limited, Cambridge, United Kingdom
Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
Feb 8, 2026Md Rafi Ur Rashid, MD Sadik Hossain Shanto, Vishnu Asutosh Dasu +1VLM Robustness Adversarial Attacks on VLMs
Pennsylvania State University · Bangladesh University of Engineering and Technology

Oct 29, 2026J/K move · Enter open · S save

ROAD: Modern Adversarial Training for Simple One-Step Distillation at Scale

Xiaoou Li, Shan You, Tao Huang

Shanghai Jiao Tong University (sjtu.edu.cn) · SenseTime (sensetime.com)

Diffusion Model Distillation Diffusion Distillation Adversarial Knowledge Distillation T2I Diffusion Models

Open paper AskPDF arXiv

One-step distillation turns pretrained diffusion models into fast generators, but strong results often require substantial training machinery and compute. Pure adversarial distillation offers a simpler route, yet has struggled at modern text-to-image scale. To address this limitation, we propose ROAD (Representation-grounded One-step Adversarial Distillation), a one-stage framework that distills text-to-image models with one adversarial objective and a lightweight representation-space critic. First, we replace the classical convolutional head with a Lipschitz-controlled Transformer head with token-level text conditioning, which matches the semantic tokens of modern visual encoders. Second, we observed a fixed random feature mixing leads to long-horizon degradation. We attribute it to generator-critic co-adaptation around one realization-specific view, and periodically re-draw the mixing to keep the randomization effective. Third, for our dense and vector-valued logits, we introduce Jacobian R1 which penalizes the full input-logit Jacobian, overcoming pooled R1's limitation of neglecting most of logit sensitivities. A single adversarial objective then suffices: in one step, ROAD reaches 11.03 FID on SDXL and 13.73 on SD3.5-M on COCO-10K, surpassing the multi-step teachers at a fraction of distribution matching's training compute.