Real-Time Oriented Object Detection Transformer in Remote Sensing Images
Authors: Zeyu Ding, Yong Zhou, Jiaqi Zhao, Wen-Liang Du, Xixi Li, Rui Yao, Abdulmotaleb El Saddik
Organizations: School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, Xuzhou 221116, China · Mine Digitization Engineering Research Center of the Ministry of Education, Xuzhou 221116, China · Jiangsu Provincial Industrial Technology Engineering Center for Intelligent Sensing and Emergency IoT in Underground Space, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Recent real-time detection transformers have gained popularity due to their simplicity and efficiency. However, these detectors do not explicitly model object rotation, especially in remote sensing imagery where objects appear at arbitrary angles, leading to challenges in angle representation, matching cost, and training stability. In this paper, we propose a real-time oriented object detection transformer, the first real-time end-to-end oriented object detector to the best of our knowledge, that addresses the above issues. Specifically, angle distribution refinement is proposed to reformulate angle regression as an iterative refinement of probability distributions, thereby capturing the uncertainty of object rotation and providing a more fine-grained angle representation. Then, we incorporate a Chamfer distance cost into bipartite matching, measuring box distance via vertex sets, enabling more accurate geometric alignment and eliminating ambiguous matches. Moreover, we propose oriented contrastive denoising to stabilize training and analyze four noise modes. We observe that a ground truth can be assigned to different index queries across different decoder layers, and analyze this issue using the proposed instability metric. We design a series of model variants and experiments to validate the proposed method. Notably, our O2-DFINE-L, O2-RTDETR-R50 and O2-DEIM-R50 achieve 77.73%/78.45%/80.15% AP50 on DOTA1.0 and 132/119/119 FPS on the 2080ti GPU. Code is available at https://github.com/wokaikaixinxin/ai4rs.
Figures & tables
Fig. 1: Compared to existing real-time object detectors, the family of O 2 -DFINE, O 2 -RTDETR, and O 2 -DEIM achieves competitive performance.
Fig. 2: The number of remaining oriented boxes above different confidence thresholds in rotated NMS.
RIoU thr.
0.05
0.3
0.5
(Conf=0.05)
AP 50 (%)
76.53
76.72
76.70
TABLE I: Effect of rotated IoU thresholds and confidence thresholds on RTMDet-s accuracy for DOTA-v1.0.
Fig. 3: Execution time of rotated NMS with different numbers of non-overlapping oriented boxes on the 2080ti (TensorRT FP16).
Fig. 4: Overview of O 2 -DFINE with Angle Distribution Refinement. Oriented boxes are learned in a decoupled manner using probability distributions as fine-grained representations of the external rectangle and vertex offsets, which are iteratively refined across decoder layers in a residual fashion.
Fig. 7: The instability metric of Oriented Contrastive Denoising.
Setting
O 2 -RTDETR
O 2 -DFINE
R18
R34
R50
S
M
L
Optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
AdamW
Base LR
1e-4
1e-4
1e-4
5e-5
5e-5
6.25e-5
Backbone LR
1e-5
1e-5
1e-5
2.5e-6
5e-6
3.125e-5
Weight decay
1e-4
1e-4
1e-4
1e-4
1e-4
1.25e-4
Clip grad
0.1
0.1
0.1
0.1
0.1
0.1
TABLE II: Main hyperparameter of our detectors.
Methods
Back.
#P
FLOPs
FPS
Pre.
PL
BD
BR
GTF
SV
LV
SH
TC
BC
ST
SBF
RA
HA
SP
HC
AP 50
▼ Non-real-time Oriented Object Detectors
ARS-DETR [ 8 ]
R50
41M
407G
-
IN
86.97
75.56
48.32
69.20
77.92
77.94
87.69
90.50
77.31
82.86
60.28
64.58
74.88
71.76
66.62
74.16
RO 2 -DETR [ 28 ]
R50
-
-
-
IN
88.98
82.13
54.58
76.28
79.28
77.92
87.94
90.90
87.19
85.65
62.20
62.63
74.62
72.42
59.22
77.57
O-DETR [ 7 ]
R50
57M
563G
-
coco
89.20
86.40
57.70
75.30
81.10
84.70
89.10
90.90
86.10
87.00
59.50
70.30
79.30
81.50
68.80
79.10
MessDet [ 19 ]
CSPnext
13M
191G
-
IN
89.28
84.69
56.20
66.18
81.37
85.33
88.86
90.78
88.28
87.62
64.47
65.59
78.18
82.10
67.82
78.45
ReDiffDet [ 22 ]
ReR50
-
-
-
IN
86.32
78.72
51.56
71.89
80.05
84.73
88.75
90.84
85.92
86.69
59.48
57.74
75.19
68.43
65.37
75.45
TABLE III: Performance comparisons on the DOTA-v1.0 dataset. ♣ indicates multi-scale training and testing. FLOPs are computed for an input size of (1,3,1024,1024). FPS is measured on an NVIDIA RTX 2080Ti using TensorRT in FP16 mode with an input size of (1,3,1024,1024). The best and second-best results are highlighted in bold and underlined .
Method
Back.
#P
APL
APO
BF
BC
BR
CH
DAM
ETS
ESA
GF
GTF
HA
OP
SH
STA
STO
TC
TS
VE
WM
AP 50
RoI Trans. [ 50 ]
R50
55M
63.34
37.88
71.78
87.53
40.68
72.60
26.86
78.71
68.09
68.96
82.74
47.71
55.61
81.21
78.23
70.26
81.61
54.86
43.27
65.52
63.87
O-RCNN [ 1 ]
R50
41M
71.10
39.30
79.50
86.20
43.10
72.60
29.50
66.70
79.30
68.60
82.40
43.60
57.60
81.30
74.20
62.60
81.40
54.80
46.80
66.00
64.30
AOPG [ 42 ]
R50
-
62.39
37.79
71.62
87.63
40.90
72.47
31.08
65.42
77.99
73.20
81.94
42.32
54.45
81.17
72.69
71.31
81.49
60.04
52.38
69.99
64.41
ARS-DETR [ 8 ]
R50
41M
68.00
54.17
74.43
81.65
41.13
75.66
34.89
73.07
81.92
76.10
78.62
36.33
55.41
84.55
70.09
72.23
81.14
61.52
50.57
70.28
66.12
RO 2 -DETR [ 28 ]
R50
-
69.78
54.77
74.49
82.11
42.30
76.66
33.94
72.99
82.01
76.11
78.56
40.11
56.06
85.06
73.11
72.19
84.10
60.66
53.02
71.11
66.43
OrientedRep [ 51 ]
R50
36M
70.03
46.11
76.12
87.19
39.14
78.76
34.57
71.80
80.42
76.16
79.41
45.48
54.90
87.82
77.03
68.07
81.60
56.83
51.57
71.25
66.71
TABLE IV: Performance comparisons on the DIOR-R dataset. The best and second-best results are highlighted in bold and underlined .
Method
Back.
#P
PL
BD
BR
GTF
SV
LV
SH
TC
BC
ST
SBF
RA
HA
SP
HC
CC
AP 50
GauCho [ 45 ]
R50
55M
76.42
72.78
48.42
59.72
61.65
75.19
84.83
90.88
76.44
73.88
56.75
69.51
62.98
67.79
50.55
13.65
65.09
ReDet [ 18 ]
ReR50
32M
79.20
82.81
51.92
71.41
52.38
75.73
80.92
90.83
75.81
68.64
49.29
72.03
73.36
70.55
63.33
11.53
66.86
DCFL [ 20 ]
R50
36M
-
-
-
-
56.72
-
80.87
-
-
75.65
-
-
-
-
-
-
67.37
RQFormer [ 9 ]
R50
41M
78.44
75.43
48.84
63.62
62.16
78.41
88.94
90.84
75.29
81.83
57.05
66.44
73.95
72.69
51.64
13.37
67.43
GSDet [ 21 ]
R50
106M
80.48
72.51
49.11
71.19
57.75
78.96
88.93
90.85
79.44
77.52
55.77
65.56
71.36
70.40
65.41
10.26
67.84
FRED [ 44 ]
R50
-
79.60
81.44
52.60
72.57
58.07
74.82
86.12
90.81
82.13
74.84
53.37
72.93
69.51
69.91
54.82
19.27
68.30
TABLE V: Performance comparisons on the DOTA-v1.5 dataset. The best and second-best results are highlighted in bold and underlined . Results are reported under single-scale training and testing.
Method
Back.
#P
B737
B747
B777
B787
C919
A220
A321
A330
A350
ARJ21
PS
MB
FB
TB
ES
LCS
DCS
RetinaNet [ 52 ]
R101
36M
38.46
55.36
24.75
51.81
0.81
40.50
41.06
18.02
19.94
1.70
9.57
22.55
1.33
16.37
19.11
14.26
24.70
CasRCNN [ 53 ]
R101
69M
40.42
52.86
29.07
52.47
0.00
44.37
38.35
26.55
17.54
0.00
12.10
28.84
0.71
15.35
18.53
14.63
25.15
FasterRCNN [ 54 ]
R101
41M
36.43
50.68
22.50
51.86
0.01
47.81
43.83
17.66
19.95
0.13
9.81
28.78
1.77
17.65
16.47
16.19
27.06
CHODNet [ 43 ]
R101
-
40.23
53.39
29.65
54.04
0.02
46.26
43.12
27.61
20.89
0.01
12.34
29.34
1.71
17.77
17.72
16.78
27.51
RoITrans [ 50 ]
R101
55M
39.58
73.56
18.32
56.43
0.00
47.67
49.91
27.64
31.79
0.00
14.31
28.07
1.03
14.32
15.97
18.04
26.02
O-RCNN [ 1 ]
R50
41M
35.17
85.17
14.57
47.68
11.68
46.55
68.18
68.60
70.21
25.32
13.77
60.42
9.10
36.83
11.32
21.86
38.22
TABLE VI: Performance comparisons on the FAIR1M-v1.0 dataset. The best and second-best results are highlighted in bold and underlined . Results are reported under single-scale training and testing.
O 2 -DFINE-s
AP 50
#P
FPS
FLOPs
DFINE-R-s
72.44
10M
297
60G
+ oriented contrastive denoising
73.72
10M
297
60G
+ angle distribution refinement
75.05
10M
297
60G
+ Chamfer distance cost
76.14
10M
297
60G
O 2 -RTDETR-R18
AP 50
#P
FPS
FLOPs
RTDETR-R-R18
73.94
20M
233
147G
TABLE VII: Step-by-step improvements from the baseline model to O 2 -DFINE-s, O 2 -RTDETR-R18, and O 2 -DEIM-R18 on DOTA-v1.0.
Number of bins N
8
16
32
64
128
AP 50
75.78
75.86
76.14
75.98
75.91
a,c of A(⋅) in ( 4 )
41,41
21,10
21,41
21,81
1,41
AP 50
75.27
75.64
76.14
75.98
75.92
TABLE VIII: Ablation study of angle distribution refinement in O 2 -DFINE-s on DOTA-v1.0 .
Distance
Cost
Distance
Loss
Point
AP 50
Cost
Weight
Loss
Weight
Number
KL Divergence
2.0
KLD
2.0
None
73.69
L1
5.0
L1
5.0
None
74.28
Hausdorff
5.0
L1
5.0
4
75.05
Chamfer
5.0
L1
5.0
4
76.14
Chamfer
3.0
L1
3.0
4
75.18
TABLE IX: Ablation study of the Chamfer distance cost for bipartite matching in O 2 -DFINE-s on DOTA-v1.0 .
Fig. 8: Qualitative analysis of Oriented Contrastive Denoising. (a) An image example. (b) Ground-truth oriented boxes. (c) The box noise. (d) The angle noise. (e) The geometric noise. (f) The probability noise example.
Box noise
λ1
0.5
1.0
2.0
λ2
1.0
2.0
4.0
AP 50
75.33
76.14
75.12
Angle noise
λ3
3.0
5.0
9.0
λ4
6.0
10.0
18.0
AP 50
74.29
74.53
74.71
Geometric noise
λ1
0.5
1.0
2.0
TABLE X: Ablation study of the oriented contrastive denoising in O 2 -DFINE-s on DOTA-v1.0 .
Fig. 9: Comparison of Chamfer distance cost with KL divergence, L1, and Hausdorff distance costs. The centers, sizes, and angles are normalized.
Fig. 10: Analysis of Angle Distribution Refinement, including initial and refined oriented boxes, external rectangles, vertex offsets, with unweighted and weighted distributions.
Fig. 11: Visualization of backbone output feature maps. The heatmaps show that our method yields more concentrated and object-aligned activations compared to other methods.
Fig. 13: Qualitative comparison between our method and other detectors on challenging scenarios, including large-scale objects (1st row), low-light conditions (2nd row), and densely distributed objects (3rd row). The confidence threshold is set to 0.3
Fig. 14: Visualization of sampling points in cross attention (a) Sampling points in Rotated Deformable DETR. (b) Rotated sampling points in O 2 -RTDETR. (c) Rotated uneven sampling points in O 2 -DFINE.
School of Computer Science and Technology, China University of Mining and Technology, and with Mine Digitization Engineering Research Center of the Ministry of Education, and also with Innovation Research Center of Disaster Intelligent Prevention and Emergency Rescue, China University of Mining and Technology, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada