Learnable Instance Attention Filtering for Adaptive Detector Distillation

Chen Liu; Qizhen Lan; Zhicheng Ding; Xinyu Chu; Qing Tian

cs.CVMar 27, 2026

Learnable Instance Attention Filtering for Adaptive Detector Distillation

Authors: Chen Liu, Qizhen Lan, Zhicheng Ding, Xinyu Chu, Qing Tian

Organizations: Dept. of CS, University of Alabama at Birmingham · Dept. of CS, Bowling Green State University

Abstract

As deep vision models grow increasingly complex to achieve higher performance, deployment efficiency has become a critical concern. Knowledge distillation (KD) mitigates this issue by transferring knowledge from large teacher models to compact student models. While many feature-based KD methods rely on spatial filtering to guide distillation, they typically treat all object instances uniformly, ignoring instance-level variability. Moreover, existing attention filtering mechanisms are typically heuristic or teacher-driven, rather than learned with the student. To address these limitations, we propose Learnable Instance Attention Filtering for Adaptive Detector Distillation (LIAF-KD), a novel framework that introduces learnable instance selectors to dynamically evaluate and reweight instance importance during distillation. Notably, the student contributes to this process based on its evolving learning state. Experiments on the KITTI and COCO datasets demonstrate consistent improvements, with a 2% gain on a GFL ResNet-50 student without added complexity, outperforming state-of-the-art methods.

Figures & tables

Figure 1: Overview of the proposed LIAF-KD framework. It employs instance selectors to dynamically reweight instances during distillation, guided by both the teacher’s knowledge and the student’s learning dynamics.

Method	mAP	$AP_{50}$	$AP_{75}$	$AP_{S}$	$AP_{M}$	$AP_{L}$
T: GFL-R101	61.3	87.6	68.8	52.8	61.6	67.3
S: GFL-R50	59.5	86.9	66.2	50.8	59.6	65.2
DeFeat [ 22 ]	61.3	88.2	68.8	53.5	61.4	66.4
FGFI [ 18 ]	61.4	88.4	69.9	53.6	61.1	68.0
GID [ 19 ]	60.9	87.3	68.0	52.7	61.0	65.9
FitNet [ 17 ]	60.9	87.9	68.2	52.4	61.1	67.1

Table 1: Comparison of object detection KD methods on the KITTI dataset. T: teacher, S: student.

Method	mAP	$AP_{50}$	$AP_{75}$	$AP_{S}$	$AP_{M}$	$AP_{L}$
T: GFL-R101	44.9	63.1	49.0	28.0	49.1	57.2
S: GFL-R50	40.2	58.9	43.3	23.3	44.0	52.2
DeFeat [ 22 ]	40.8	58.6	44.2	24.3	44.6	53.7
FGFI [ 18 ]	41.1	58.8	44.8	23.3	45.4	53.1
GID [ 19 ]	41.5	59.6	45.2	24.3	45.7	53.6
FitNet [ 17 ]	40.7	58.6	44.0	23.7	44.4	53.2

Table 2: Comparison of object detection KD methods on the COCO dataset. T: teacher, S: student.

Panel of Figure 2 — Figure 2: Detection visualization of different models. Column (a) and (b) are input images and ground truth. Columns (c), (d), and (e) are the detection results of the student baseline, MasKD, and LIAF-KD models, respectively.

Panel of Figure 3 — Figure 3: Grad-CAM Attention of different models. The colors represent attention intensity, with red indicating the highest level and blue the lowest. “Base”: the student baseline.

Method	mAP	Params(M)	FLOPs(T)	FPS(GPU)	FPS(CPU)
T:Retina-R101	38.9	57.0	0.29	33.5	0.935
S: LIAF-KD	39.7	38.0	0.22	44.5	1.308
T: GFL-R101	44.9	51.4	0.26	30.3	1.106
S: LIAF-KD	42.4	32.4	0.19	40.0	1.335

Table 3: Efficiency comparison of different object detectors. We report inference speed on an NVIDIA A100 GPU and an AMD EPYC™ 7H12 CPU. T: teacher, S: student.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 819 papers

FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors
Aug 19, 2026YoungJae Cheong, Jhonghyun AnKnowledge Distillation Efficient Object Detection
Department of AI and Software, Gachon University, Seongnam, Republic of Korea
Training a Student Expert via Semi-Supervised Foundation Model Distillation
Apr 4, 2026Pardis Taghavi, Tian Liu, Renjie Li +2Knowledge Distillation Instance Segmentation
Texas A&M University
Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
Aug 24, 2026Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner +2Knowledge Distillation Feature-Based Knowledge Distillation
Andalusian Research Institute in Data Science and Computational Intelligence, 18016 Granada, Spain · Lab for Uncertainty in Data and Decision Making (LUCID), School of Computer Science, University of Nottingham, NG7 2RD Nottingham, UK · Department of Computer Science and Artificial Intelligence, School of Computer Science and Telecommunications Engineering (ETSIIT), University of Granada, 18071 Granada, Spain

Aug 19, 2026J/K move · Enter open · S save

FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors

YoungJae Cheong, Jhonghyun An

Department of AI and Software, Gachon University, Seongnam, Republic of Korea

Knowledge Distillation Efficient Object Detection Feature-Based Knowledge Distillation Relational Knowledge Distillation

Open paper AskPDF arXiv

Compact object detectors are suitable for resource-constrained visual perception, but their limited representation capacity creates an accuracy gap relative to large models. Conventional detector distillation often relies on prediction-level supervision or a single feature-alignment target, such as response, distribution, correlation, or frequency-domain matching. Frequency-Decoupled Cross-Attention Knowledge Distillation (FD-CanKD) is presented as a detector-oriented framework that transfers teacher knowledge at three complementary levels: head-level prediction supervision, relation-level non-local context transfer, and frequency-level component-selective alignment. Student features first aggregate teacher-side spatial context through cross-attention-based relation transfer, after which frequency-aware alignment preserves complementary structural and detail-sensitive cues. Under controlled Microsoft Common Objects in Context (COCO) experiments, fixed 50-epoch from-scratch comparisons show that FD-CanKD remains competitive with representative detector knowledge distillation baselines. Post-distillation continued fine-tuning further produces a stronger refinement-ready student than detector-only fine-tuning, reaching 48.87 mean average precision (mAP) at intersection-over-union thresholds from 0.50 to 0.95 (mAP50:95), 65.84 mAP50, and 53.40 mAP75 after 20 additional epochs. All distillation modules are removed after training, leaving the deployed student unchanged at 19.7M parameters. The framework is instantiated and evaluated in a controlled YOLOv12 teacher-student setting as a representative compact-detector case study.

Abstract

Knowledge Distillation

Explore similar work

FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors

Training a Student Expert via Semi-Supervised Foundation Model Distillation

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it