Selective Knowledge Distillation

Selective knowledge distillation studies how to transfer useful teacher knowledge without teaching every example, token, feature, or reasoning step. Methods select or reweight supervision using confidence, uncertainty, teacher–student disagreement, task relevance, or estimates of whether a signal is learnable; they may transfer logits, features, or rationales. Much recent work applies these ideas to on-policy language-model reasoning, evaluated on math and coding benchmarks, while related approaches target vision, multimodal, and medical tasks. Selective supervision can reduce training and teacher-input costs or avoid harmful guidance, but its benefits depend on student capacity, selector quality, and optimization settings, and filtering too much can discard useful corrections.