While linear attention is a compelling mechanism for high-resolution object detection due to its reduced cost for global token mixing, converting the Softmax-attention ViT backbone of a trained detector into a linear-attention one is not a trivial drop-in replacement. Directly swapping the attention operator leads to severe performance degradation, and generic label-free distillation, though effective for classification, often fails on detection tasks. We argue that the central challenge is detector-interface preservation: the converted backbone must reproduce the exact feature tensors expected by the fixed downstream detector, rather than merely imitating internal Softmax hidden states. To address this, we introduce Detector-Interface Distillation (DiD), a label-free conversion method that exclusively trains the linear-attention backbone by aligning detector-facing interface tensors with those of a frozen Softmax teacher. On DOTA-v1.5, DiD substantially outperforms established baselines and matches supervised, fully trained linear models. Adaptation completes in roughly 87 minutes on 4 GPUs, and the linearized backbone cuts inference latency by ~62% and peak memory by ~49%. We hope our findings offer the community a simple, label-free route to reusing trained Softmax detectors as efficient linear ones, and encourage interface-aware objectives in future architecture-conversion work.
Figures & tables
Figure 1: Overview of the label-free Softmax-to-linear detector conversion setting. (a) Starting from a trained detector with a Softmax-attention ViT backbone, we construct a student by replacing the backbone attention operator with linear attention. During conversion, the Softmax teacher and the linear student receive the same unlabeled images; only the student backbone is updated, while the downstream detector remains fixed. Snowflake symbols indicate frozen modules, while the fire symbol indicates the trainable module. (b) After conversion, the distilled linear backbone is used directly in the detector without additional fine-tuning.
Classification
Detection
Method
ImageNet
CIFAR-100
DOTA-v1.5
Acc.
Retained (%)
Acc.
Retained (%)
mAP
Retained (%)
Softmax reference
79.83
100.0
78.09
100.0
33.5
100.0
Direct copy
5.08
6.4
3.54
4.5
1.7
5.1
Generic distillation
77.83
97.5
75.92
97.2
18.1
54.0
Table 1: Cross-task Softmax-to-ReLU transfer. Each task reports its native score followed by the percentage retained relative to its own Softmax reference. Generic distillation nearly restores classification accuracy but recovers only half of the DOTA-v1.5 detector mAP.
Supervision target
mAP
mAP 50
mAP 75
Attention outputs
7.8
17.4
5.6
Hidden states from all layers
25.2
44.7
24.6
Hidden states from readout layers
20.3
37.4
19.0
Final hidden state
12.6
24.1
11.7
Post-neck/FPN detector features
30.7
54.1
30.1
Table 2: Effect of supervision placement for RetinaNet conversion on DOTA-v1.5. Matching multi-scale features produced after the neck/FPN is substantially more effective than matching attention outputs or internal backbone states.
Detector
Method
ReLU
ELU
mAP
mAP 50
mAP 75
mAP
mAP 50
mAP 75
RetinaNet
Softmax reference
33.5
58.1
33.2
33.5
58.1
33.2
Full training
30.9
53.3
30.9
30.8
53.0
31.3
Direct copy
1.7
3.8
1.2
1.3
2.9
0.9
Generic distillation
18.1
35.2
16.6
9.6
19.2
5.3
Softmax-generated pseudo-label distillation
14.8
33.6
10.1
19.6
38.2
17.2
Table 3: Label-free Softmax-to-linear conversion on DOTA-v1.5 with ViT-S. Full training is a supervised reference; all other adaptation methods use no detection labels.
Figure 2: Conversion behavior on RetinaNet. (a) Hidden-state distillation more closely matches internal transformer states, while DiD more closely matches the detector-facing interface. (b) DiD consistently outperforms hidden-state distillation during conversion and reaches the supervised linear-training regime after approximately 20k-30k iterations.
GPUs
Global batch
Iter/s
Wall time (min)
20k
40k
1
8
2.98
112
224
2
8
5.20
64
128
4
8
7.67
44
87
Table 4: RetinaNet ReLU conversion throughput at fixed global batch size 8.
Figure 3: Backbone-only inference comparison between the Softmax and ReLU-linearized ViT. Linear attention substantially reduces latency and peak memory.
Interface weighting
mAP
mAP 50
mAP 75
Uniform
29.8
52.1
29.4
Late-heavy
28.4
49.8
28.2
Early-heavy
30.7
54.1
30.1
Table 5: Effect of detector-interface weighting.
Loss
mAP
mAP 50
mAP 75
MSE
30.7
54.1
30.1
L1
30.0
52.5
29.3
Smooth L1
30.3
53.0
29.7
MSE + cosine
30.4
53.2
29.8
L1 + cosine
29.3
51.5
28.8
Smooth L1 + cosine
30.4
53.1
29.9
Table 6: Effect of the reconstruction loss.
Train augmentation
mAP
mAP 50
mAP 75
RandomHorizontalFlip
30.7
54.1
30.1
+ Medium ColorJitter + RandomAffine
30.0
52.2
29.8
+ Strong ColorJitter + RandomAffine
27.0
47.9
26.4
Table 7: Effect of augmentation severity.
Conversion source
mAP
mAP 50
mAP 75
DOTA-v1.5
30.7
54.1
30.1
FAIR1M
25.1
43.5
25.0
ImageNet
20.2
36.6
19.7
Table 8: Effect of the conversion-data domain.
Figure 4: Scaling behavior of RetinaNet DiD on DOTA-v1.5. Scores are normalized to the linear full-training.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Detector
Kernel
mAP
mAP 50
mAP 75
RetinaNet
ReLU
31.1 ± 0.1
54.5 ± 0.2
31.0 ± 0.1
ELU
31.1 ± 0.1
54.4 ± 0.1
30.7 ± 0.1
Faster R-CNN
ReLU
29.3 ± 0.2
51.1 ± 0.2
29.8 ± 0.2
ELU
28.9 ± 0.1
50.2 ± 0.1
29.4 ± 0.2
Appendix
Table A: Mean ± standard deviation over three seeds for DiD conversion on DOTA-v1.5 with a ViT-S backbone.
Method
ImageNet
DINO
No pretraining
mAP
mAP 50
mAP 75
mAP
mAP 50
mAP 75
mAP
mAP 50
mAP 75
Softmax reference
33.5
58.1
33.2
31.4
53.8
31.5
25.9
46.5
25.2
Hidden-state distillation
28.1
49.7
27.7
28.7
49.7
28.5
26.0
46.4
25.3
DiD
31.1
54.5
31.0
31.0
53.1
31.5
26.2
46.7
25.4
Appendix
Table B: Compatibility with different teacher initializations for RetinaNet ReLU conversion on DOTA-v1.5.
Detector
Method
ReLU
ELU
mAP
mAP 50
mAP 75
mAP
mAP 50
mAP 75
RetinaNet
Softmax reference
66.6
90.3
75.2
66.6
90.3
75.2
Full training
60.3
85.5
67.4
59.4
84.8
66.2
Direct copy
3.2
7.0
1.8
2.1
5.0
1.0
Generic distillation
46.5
75.8
49.2
38.0
67.1
37.2
Softmax-generated pseudo-label distillation
54.0
82.2
58.4
55.6
83.5
60.7
Appendix
Table C: Label-free Softmax-to-linear conversion on Pascal VOC with ViT-S. Full training is a supervised reference; all other adaptation methods use no detection labels.
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China