Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers
Authors: Wen Wang, Yang Cao, Jing Zhang, Fengxiang He, Zheng-Jun Zha, Yonggang Wen, Dacheng Tao
Organizations: University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The University of Sydney · JD Explore Academy, China · Nanyang Technological University
Detection transformers have recently shown promising object detection results and attracted increasing attention. However, how to develop effective domain adaptation techniques to improve its cross-domain performance remains unexplored and unclear. In this paper, we delve into this topic and empirically find that direct feature distribution alignment on the CNN backbone only brings limited improvements, as it does not guarantee domain-invariant sequence features in the transformer for prediction. To address this issue, we propose a novel Sequence Feature Alignment (SFA) method that is specially designed for the adaptation of detection transformers. Technically, SFA consists of a domain query-based feature alignment (DQFA) module and a token-wise feature alignment (TFA) module. In DQFA, a novel domain query is used to aggregate and align global context from the token sequence of both domains. DQFA reduces the domain discrepancy in global feature representations and object relations when deploying in the transformer encoder and decoder, respectively. Meanwhile, TFA aligns token features in the sequence from both domains, which reduces the domain gaps in local and instance-level feature representations in the transformer encoder and decoder, respectively. Besides, a novel bipartite matching consistency loss is proposed to enhance the feature discriminability for robust object detection. Experiments on three challenging benchmarks show that SFA outperforms state-of-the-art domain adaptive object detection methods. Code has been made available at: https://github.com/encounter1997/SFA.
Figures & tables
Figure 1. (a) Performance comparison between feature distribution alignment applied on the CNN backbone and transformer (denoted as DA-CNN and DA-Seq, respectively). Both methods are built on the Deformable DETR ( Zhu et al., 2020 ) with a ResNet-50 backbone and evaluated on the Cityscapes ( Cordts et al., 2016 ) to Foggy Cityscapes ( Sakaridis et al., 2018 ) scenario. (b) T-SNE visualization of features extracted by DA-CNN. While the CNN features are well-align, the sequence feature extracted by transformer encoder and decoder can be easily separated by domain. pipeline
Figure 2. Diagram of our sequence feature alignment (SFA) for domain adaptive detection transformers. Domain query-based feature alignment and token-wise feature alignment tackles global and local domain gaps, respectively. Moreover, a bipartite matching consistency loss is proposed to improve the model’s discriminability. Here object queries with the same color are matched to the same ground-truth object, while object queries with the same shape come from the same decoder layer. model
Method
Detector
person
rider
car
truck
bus
train
mcycle
bicycle
mAP
Faster RCNN (Source)
FRCNN
26.9
38.2
35.6
18.3
32.4
9.6
25.8
28.6
26.9
DAF ( Chen et al., 2018 )
FRCNN
29.2
40.4
43.4
19.7
38.3
28.5
23.7
32.7
32.0
DivMatch ( Kim et al., 2019b )
FRCNN
31.8
40.5
51.0
20.9
41.8
34.3
26.6
32.4
34.9
SWDA ( Saito et al., 2019 )
FRCNN
31.8
44.3
48.9
21.0
43.8
28.0
28.9
35.8
35.3
SCDA ( Zhu et al., 2019 )
FRCNN
33.8
42.1
52.1
26.8
42.5
26.5
29.2
34.5
35.9
MTOR ( Cai et al., 2019 )
FRCNN
30.6
41.4
44.0
21.9
38.6
40.6
28.3
35.6
35.1
Table 1. Results of different methods for weather adaptation, i.e. , Cityscapes to Foggy Cityscapes. FRCNN and DefDETR are abbreviations for Faster RCNN and Deformable DETR, respectively.
Methods
Detector
car AP
Faster RCNN (Source)
FRCNN
34.6
DAF ( Chen et al., 2018 )
FRCNN
41.9
DivMatch ( Kim et al., 2019b )
FRCNN
43.9
SWDA ( Saito et al., 2019 )
FRCNN
44.6
SCDA ( Zhu et al., 2019 )
FRCNN
45.1
MTOR ( Cai et al., 2019 )
FRCNN
46.6
Table 2. Results of different methods for synthetic to real adaptation, i.e. , Sim10k to Cityscapes.
Methods
Detector
person
rider
car
truck
bus
mcycle
bicycle
mAP
Faster R-CNN (Source)
FRCNN
28.8
25.4
44.1
17.9
16.1
13.9
22.4
24.1
DAF ( Chen et al., 2018 )
FRCNN
28.9
27.4
44.2
19.1
18.0
14.2
22.4
24.9
SWDA ( Saito et al., 2019 )
FRCNN
29.5
29.9
44.8
20.2
20.7
15.2
23.1
26.2
SCDA ( Zhu et al., 2019 )
FRCNN
29.3
29.2
44.4
20.3
19.6
14.8
23.2
25.8
CR-DA ( Xu et al., 2020a )
FRCNN
30.8
29.0
44.8
20.5
19.8
14.1
22.8
26.0
CR-SW ( Xu et al., 2020a )
FRCNN
32.8
29.3
45.8
22.7
20.6
14.9
25.5
27.4
Table 3. Results of different methods for scene adaptation, i.e. , Cityscapes to BDD100k daytime subset.
Methods
CNN
DQ
TW
BMC
HR
person
rider
car
truck
bus
train
mcycle
bicycle
mAP
Deformable DETR (Source)
37.7
39.1
44.2
17.2
26.8
5.8
21.6
35.5
28.5
Proposed
✓
43.8
45.6
55.5
18.2
38.7
8.4
28.1
43.3
35.2
✓
45.1
46.8
61.3
21.6
36.6
9.5
30.6
40.4
36.5
✓
45.6
47.6
60.7
23.6
41.3
15.5
26.2
41.1
37.7
✓
38.0
39.3
45.6
16.5
28.4
6.0
24.5
37.5
29.5
✓
✓
45.7
47.6
61.6
23.7
43.8
16.4
28.2
43.3
38.8
Table 4. Ablation studies on the Cityscapes to Foggy Cityscapes scenario. CNN represents feature alignment on the CNN backbone. DQ, TW, BMC, and HR represent domain query-based feature alignment, token-wise feature alignment, bipartite matching consistency, and hierarchical feature alignment, respectively.
Figure 3. Visualizations. (a) Qualitative results: From top to bottom are results on the scenarios of Cityscapes to Foggy Cityscapes, Sim10k to Cityscapes, and Cityscapes to BDD100k, respectively. (b) Visualization of feature distributions using T-SNE ( Van der Maaten and Hinton, 2008 ) . The blue circles denote the source features, while red circles represent target features. activation
Figure 4. Visualization of domain queries from both transformer encoder and decoder sides, under the Cityscapes to Foggy Citysccapes scenario. activation
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Domain Discriminators
Fully Connected 256×256
ReLU
Fully Connected 256×256
ReLU
Fully Connected 256×2
Softmax
Appendix
Table 5. The architecture of the domain discriminators.
Methods
DQ
TW
BMC
HR
person
rider
car
truck
bus
train
mcycle
bicycle
mAP
DETR (Source)
19.4
16.0
35.2
5.1
7.8
3.0
10.5
15.9
14.1
Proposed
✓
19.9
23.5
38.8
15.7
17.0
3.8
12.9
16.4
18.5
✓
18.3
24.4
41.8
14.7
22.4
5.6
9.3
17.0
19.2
✓
20.7
16.5
35.5
7.6
9.0
5.0
10.3
16.0
15.0
✓
✓
20.8
24.5
42.0
13.0
20.3
12.9
12.7
17.7
20.5
✓
✓
✓
21.2
23.5
44.0
17.6
25.2
12.7
13.9
18.0
22.0
Appendix
Table 6. Sequence feature alignment built on DETR ( Carion et al., 2020 ) . DQ, TW, BMC, and HR represents domain query-based feature alignment, token-wise feature alignment, bipartite matching consistency, and hierarchical feature alignment, respectively.
Methods
DQenc
DQdec
TWenc
TWdec
person
rider
car
truck
bus
train
mcycle
bicycle
mAP
Deformable DETR
37.7
39.1
44.2
17.2
26.8
5.8
21.6
35.5
28.5
Proposed
✓
44.8
45.9
56.9
18.7
37.8
8.5
23.1
40.3
34.5
✓
42.8
43.5
54.5
13.4
37.9
8.2
28.6
40.5
33.7
✓
44.6
48.0
58.7
24.4
39.6
9.1
29.2
43.1
37.1
✓
44.9
48.1
60.3
16.2
38.0
10.7
29.0
40.9
36.1
✓
✓
45.1
46.8
61.3
21.6
36.6
9.5
30.6
40.4
36.5
Appendix
Table 7. Ablation studies on domain query-based feature alignment (DQFA) and token-wise feature alignment (TWFA), without hierarchical feature alignment or bipartite matching consistency loss . Experiments are conducted on the Cityscapes to Foggy Cityscapes Scenario. DQenc and DQdec indicate the DQFA applied to the last encoder and decoder layers, respectively. Similarly, TWenc and TWdec indicate the TWFA applied to the last encoder and decoder layers, respectively
Figure 5. More qualitative detection results on Cityscapes to Foggy Cityscapes scenario. activation
Figure 6. More qualitative detection results on Sim10k to Cityscapes scenario. activation
Figure 7. More qualitative detection results on Cityscapes to BDD100k scenario. activation
Intelligent Software Research Center, Institute of Software, CAS · University of Chinese Academy of Sciences · State Key Lab of Processors, Institute of Computing Technology, CAS +1
School of Computing and Data Science, The University of Hong Kong · Department of Electrical & Computer Engineering, National University of Singapore · School of Information and Electronics, Beijing Institute of Technology