cs.CVNov 18, 2026

Angular Conditional Distillation: Overcoming Classifier Geometry Mismatch

Authors: Xuefei Zhang, Ruoqi Wang, Lei Cui

Organizations: Institute of AI for Industries (IAII), Chinese Academy of Sciences

Abstract

Classifier outputs provide a fundamental and univer- sally available knowledge interface for knowledge distil- lation (KD). However, standard logit objectives entangle transferable semantic relations with architecture-dependent parameterization artifacts, causing a classifier geometry mismatch that makes direct knowledge transfer unreliable. To address this, we propose Angular Conditional Distilla- tion (ACD). First, the Angular Coordinate Interface (ACI) reconstructs the student classifier using angular coordi- nates, suppressing magnitude and bias effects to provide a geometry-compatible space. Then, Scale-Invariant Condi- tional Distillation (SICD) standardizes outputs and exclu- sively transfers conditional non-target relations, filtering out untransferable absolute confidence. Together, ACI de- fines where to transfer, while SICD defines what to transfer. Extensive experiments show that ACD achieves state-of-the- art performance over existing KD methods. Moreover, as a universal drop-in replacement, it consistently improves existing distillation pipelines while requiring no feature adapters or additional inference-time cost. Code is avail- able at https://github.com/laniusnan/ACD.

Figures & tables

Explore similar work

CardsList
  1. Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

    Sep 30, 2026Qianfeng Yuan, Wenbing TaoKnowledge Distillation

  2. Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

    Aug 14, 2026Dawen Jiang, Zhishu Shen, Zeyu Liu +1Knowledge DistillationRelational Knowledge Distillation

  3. TALD: Transferability-Aware Logit Disentanglement for Cross-Modal Knowledge Distillation

    Nov 18, 2026Hui Li, Pengfei Yang, Xinyi Feng +7Cross-Modal Knowledge DistillationAdaptive Knowledge Distillation