Real-Time Oriented Object Detection Transformer in Remote Sensing Images
Authors: Zeyu Ding, Yong Zhou, Jiaqi Zhao, Wen-Liang Du, Xixi Li, Rui Yao, Abdulmotaleb El Saddik
Organizations: School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, Xuzhou 221116, China · Mine Digitization Engineering Research Center of the Ministry of Education, Xuzhou 221116, China · Jiangsu Provincial Industrial Technology Engineering Center for Intelligent Sensing and Emergency IoT in Underground Space, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada
Recent real-time detection transformers have gained popularity due to their simplicity and efficiency. However, these detectors do not explicitly model object rotation, especially in remote sensing imagery where objects appear at arbitrary angles, leading to challenges in angle representation, matching cost, and training stability. In this paper, we propose a real-time oriented object detection transformer, the first real-time end-to-end oriented object detector to the best of our knowledge, that addresses the above issues. Specifically, angle distribution refinement is proposed to reformulate angle regression as an iterative refinement of probability distributions, thereby capturing the uncertainty of object rotation and providing a more fine-grained angle representation. Then, we incorporate a Chamfer distance cost into bipartite matching, measuring box distance via vertex sets, enabling more accurate geometric alignment and eliminating ambiguous matches. Moreover, we propose oriented contrastive denoising to stabilize training and analyze four noise modes. We observe that a ground truth can be assigned to different index queries across different decoder layers, and analyze this issue using the proposed instability metric. We design a series of model variants and experiments to validate the proposed method. Notably, our O2-DFINE-L, O2-RTDETR-R50 and O2-DEIM-R50 achieve 77.73%/78.45%/80.15% AP50 on DOTA1.0 and 132/119/119 FPS on the 2080ti GPU. Code is available at https://github.com/wokaikaixinxin/ai4rs.
Figures & tables
Fig. 1: Compared to existing real-time object detectors, the family of O 2 -DFINE, O 2 -RTDETR, and O 2 -DEIM achieves competitive performance.
Fig. 2: The number of remaining oriented boxes above different confidence thresholds in rotated NMS.
RIoU thr.
0.05
0.3
0.5
(Conf=0.05)
AP 50 (%)
76.53
76.72
76.70
TABLE I: Effect of rotated IoU thresholds and confidence thresholds on RTMDet-s accuracy for DOTA-v1.0.
Fig. 3: Execution time of rotated NMS with different numbers of non-overlapping oriented boxes on the 2080ti (TensorRT FP16).
Fig. 4: Overview of O 2 -DFINE with Angle Distribution Refinement. Oriented boxes are learned in a decoupled manner using probability distributions as fine-grained representations of the external rectangle and vertex offsets, which are iteratively refined across decoder layers in a residual fashion.
Fig. 7: The instability metric of Oriented Contrastive Denoising.
Setting
O 2 -RTDETR
O 2 -DFINE
R18
R34
R50
S
M
L
Optimizer
AdamW
AdamW
AdamW
AdamW
AdamW
AdamW
Base LR
1e-4
1e-4
1e-4
5e-5
5e-5
6.25e-5
Backbone LR
1e-5
1e-5
1e-5
2.5e-6
5e-6
3.125e-5
Weight decay
1e-4
1e-4
1e-4
1e-4
1e-4
1.25e-4
Clip grad
0.1
0.1
0.1
0.1
0.1
0.1
TABLE II: Main hyperparameter of our detectors.
Methods
Back.
#P
FLOPs
FPS
Pre.
PL
BD
BR
GTF
SV
LV
SH
TC
BC
ST
SBF
RA
HA
SP
HC
AP 50
▼ Non-real-time Oriented Object Detectors
ARS-DETR [ 8 ]
R50
41M
407G
-
IN
86.97
75.56
48.32
69.20
77.92
77.94
87.69
90.50
77.31
82.86
60.28
64.58
74.88
71.76
66.62
74.16
RO 2 -DETR [ 28 ]
R50
-
-
-
IN
88.98
82.13
54.58
76.28
79.28
77.92
87.94
90.90
87.19
85.65
62.20
62.63
74.62
72.42
59.22
77.57
O-DETR [ 7 ]
R50
57M
563G
-
coco
89.20
86.40
57.70
75.30
81.10
84.70
89.10
90.90
86.10
87.00
59.50
70.30
79.30
81.50
68.80
79.10
MessDet [ 19 ]
CSPnext
13M
191G
-
IN
89.28
84.69
56.20
66.18
81.37
85.33
88.86
90.78
88.28
87.62
64.47
65.59
78.18
82.10
67.82
78.45
ReDiffDet [ 22 ]
ReR50
-
-
-
IN
86.32
78.72
51.56
71.89
80.05
84.73
88.75
90.84
85.92
86.69
59.48
57.74
75.19
68.43
65.37
75.45
TABLE III: Performance comparisons on the DOTA-v1.0 dataset. ♣ indicates multi-scale training and testing. FLOPs are computed for an input size of (1,3,1024,1024). FPS is measured on an NVIDIA RTX 2080Ti using TensorRT in FP16 mode with an input size of (1,3,1024,1024). The best and second-best results are highlighted in bold and underlined .
Method
Back.
#P
APL
APO
BF
BC
BR
CH
DAM
ETS
ESA
GF
GTF
HA
OP
SH
STA
STO
TC
TS
VE
WM
AP 50
RoI Trans. [ 50 ]
R50
55M
63.34
37.88
71.78
87.53
40.68
72.60
26.86
78.71
68.09
68.96
82.74
47.71
55.61
81.21
78.23
70.26
81.61
54.86
43.27
65.52
63.87
O-RCNN [ 1 ]
R50
41M
71.10
39.30
79.50
86.20
43.10
72.60
29.50
66.70
79.30
68.60
82.40
43.60
57.60
81.30
74.20
62.60
81.40
54.80
46.80
66.00
64.30
AOPG [ 42 ]
R50
-
62.39
37.79
71.62
87.63
40.90
72.47
31.08
65.42
77.99
73.20
81.94
42.32
54.45
81.17
72.69
71.31
81.49
60.04
52.38
69.99
64.41
ARS-DETR [ 8 ]
R50
41M
68.00
54.17
74.43
81.65
41.13
75.66
34.89
73.07
81.92
76.10
78.62
36.33
55.41
84.55
70.09
72.23
81.14
61.52
50.57
70.28
66.12
RO 2 -DETR [ 28 ]
R50
-
69.78
54.77
74.49
82.11
42.30
76.66
33.94
72.99
82.01
76.11
78.56
40.11
56.06
85.06
73.11
72.19
84.10
60.66
53.02
71.11
66.43
OrientedRep [ 51 ]
R50
36M
70.03
46.11
76.12
87.19
39.14
78.76
34.57
71.80
80.42
76.16
79.41
45.48
54.90
87.82
77.03
68.07
81.60
56.83
51.57
71.25
66.71
TABLE IV: Performance comparisons on the DIOR-R dataset. The best and second-best results are highlighted in bold and underlined .
Method
Back.
#P
PL
BD
BR
GTF
SV
LV
SH
TC
BC
ST
SBF
RA
HA
SP
HC
CC
AP 50
GauCho [ 45 ]
R50
55M
76.42
72.78
48.42
59.72
61.65
75.19
84.83
90.88
76.44
73.88
56.75
69.51
62.98
67.79
50.55
13.65
65.09
ReDet [ 18 ]
ReR50
32M
79.20
82.81
51.92
71.41
52.38
75.73
80.92
90.83
75.81
68.64
49.29
72.03
73.36
70.55
63.33
11.53
66.86
DCFL [ 20 ]
R50
36M
-
-
-
-
56.72
-
80.87
-
-
75.65
-
-
-
-
-
-
67.37
RQFormer [ 9 ]
R50
41M
78.44
75.43
48.84
63.62
62.16
78.41
88.94
90.84
75.29
81.83
57.05
66.44
73.95
72.69
51.64
13.37
67.43
GSDet [ 21 ]
R50
106M
80.48
72.51
49.11
71.19
57.75
78.96
88.93
90.85
79.44
77.52
55.77
65.56
71.36
70.40
65.41
10.26
67.84
FRED [ 44 ]
R50
-
79.60
81.44
52.60
72.57
58.07
74.82
86.12
90.81
82.13
74.84
53.37
72.93
69.51
69.91
54.82
19.27
68.30
TABLE V: Performance comparisons on the DOTA-v1.5 dataset. The best and second-best results are highlighted in bold and underlined . Results are reported under single-scale training and testing.
Method
Back.
#P
B737
B747
B777
B787
C919
A220
A321
A330
A350
ARJ21
PS
MB
FB
TB
ES
LCS
DCS
RetinaNet [ 52 ]
R101
36M
38.46
55.36
24.75
51.81
0.81
40.50
41.06
18.02
19.94
1.70
9.57
22.55
1.33
16.37
19.11
14.26
24.70
CasRCNN [ 53 ]
R101
69M
40.42
52.86
29.07
52.47
0.00
44.37
38.35
26.55
17.54
0.00
12.10
28.84
0.71
15.35
18.53
14.63
25.15
FasterRCNN [ 54 ]
R101
41M
36.43
50.68
22.50
51.86
0.01
47.81
43.83
17.66
19.95
0.13
9.81
28.78
1.77
17.65
16.47
16.19
27.06
CHODNet [ 43 ]
R101
-
40.23
53.39
29.65
54.04
0.02
46.26
43.12
27.61
20.89
0.01
12.34
29.34
1.71
17.77
17.72
16.78
27.51
RoITrans [ 50 ]
R101
55M
39.58
73.56
18.32
56.43
0.00
47.67
49.91
27.64
31.79
0.00
14.31
28.07
1.03
14.32
15.97
18.04
26.02
O-RCNN [ 1 ]
R50
41M
35.17
85.17
14.57
47.68
11.68
46.55
68.18
68.60
70.21
25.32
13.77
60.42
9.10
36.83
11.32
21.86
38.22
TABLE VI: Performance comparisons on the FAIR1M-v1.0 dataset. The best and second-best results are highlighted in bold and underlined . Results are reported under single-scale training and testing.
O 2 -DFINE-s
AP 50
#P
FPS
FLOPs
DFINE-R-s
72.44
10M
297
60G
+ oriented contrastive denoising
73.72
10M
297
60G
+ angle distribution refinement
75.05
10M
297
60G
+ Chamfer distance cost
76.14
10M
297
60G
O 2 -RTDETR-R18
AP 50
#P
FPS
FLOPs
RTDETR-R-R18
73.94
20M
233
147G
TABLE VII: Step-by-step improvements from the baseline model to O 2 -DFINE-s, O 2 -RTDETR-R18, and O 2 -DEIM-R18 on DOTA-v1.0.
Number of bins N
8
16
32
64
128
AP 50
75.78
75.86
76.14
75.98
75.91
a,c of A(⋅) in ( 4 )
41,41
21,10
21,41
21,81
1,41
AP 50
75.27
75.64
76.14
75.98
75.92
TABLE VIII: Ablation study of angle distribution refinement in O 2 -DFINE-s on DOTA-v1.0 .
Distance
Cost
Distance
Loss
Point
AP 50
Cost
Weight
Loss
Weight
Number
KL Divergence
2.0
KLD
2.0
None
73.69
L1
5.0
L1
5.0
None
74.28
Hausdorff
5.0
L1
5.0
4
75.05
Chamfer
5.0
L1
5.0
4
76.14
Chamfer
3.0
L1
3.0
4
75.18
TABLE IX: Ablation study of the Chamfer distance cost for bipartite matching in O 2 -DFINE-s on DOTA-v1.0 .
Fig. 8: Qualitative analysis of Oriented Contrastive Denoising. (a) An image example. (b) Ground-truth oriented boxes. (c) The box noise. (d) The angle noise. (e) The geometric noise. (f) The probability noise example.
Box noise
λ1
0.5
1.0
2.0
λ2
1.0
2.0
4.0
AP 50
75.33
76.14
75.12
Angle noise
λ3
3.0
5.0
9.0
λ4
6.0
10.0
18.0
AP 50
74.29
74.53
74.71
Geometric noise
λ1
0.5
1.0
2.0
TABLE X: Ablation study of the oriented contrastive denoising in O 2 -DFINE-s on DOTA-v1.0 .
Fig. 9: Comparison of Chamfer distance cost with KL divergence, L1, and Hausdorff distance costs. The centers, sizes, and angles are normalized.
Fig. 10: Analysis of Angle Distribution Refinement, including initial and refined oriented boxes, external rectangles, vertex offsets, with unweighted and weighted distributions.
Fig. 11: Visualization of backbone output feature maps. The heatmaps show that our method yields more concentrated and object-aligned activations compared to other methods.
Fig. 13: Qualitative comparison between our method and other detectors on challenging scenarios, including large-scale objects (1st row), low-light conditions (2nd row), and densely distributed objects (3rd row). The confidence threshold is set to 0.3
Fig. 14: Visualization of sampling points in cross attention (a) Sampling points in Rotated Deformable DETR. (b) Rotated sampling points in O 2 -RTDETR. (c) Rotated uneven sampling points in O 2 -DFINE.
We present RiO-DETR: DETR for Real-time Oriented Object Detection, the first real-time oriented detection transformer to the best of our knowledge. Adapting DETR to oriented bounding boxes (OBBs) poses three challenges: semantics-dependent orientation, angle periodicity that breaks standard Euclidean refinement, and an enlarged search space that slows convergence. RiO-DETR resolves these issues with task-native designs while preserving real-time efficiency. First, we propose Content-Driven Angle Estimation by decoupling angle from positional queries, together with Rotation-Rectified Orthogonal Attention to capture complementary cues for reliable orientation. Second, Decoupled Periodic Refinement combines bounded coarse-to-fine updates with a Shortest-Path Periodic Loss for stable learning across angular seams. Third, Oriented Dense O2O injects angular diversity into dense supervision to speed up angle convergence at no extra cost. Extensive experiments on DOTA-1.0, DIOR-R, and FAIR-1M-2.0 demonstrate RiO-DETR establishes a new speed--accuracy trade-off for real-time oriented detection. GitHub Repository: https://github.com/RicePasteM/RiO-DETR.
Zhangchi Hu, Yifan Zhao, Yansong Peng +8
University of Science and Technology of China · Tsinghua University · Huawei Technologies Co., Ltd. +1
Oriented object detection in remote sensing images is a challenging task due to objects being distributed in multi-orientation. Recently, end-to-end transformer-based methods have achieved success by eliminating the need for post-processing operators compared to traditional CNN-based methods. However, directly extending transformers to oriented object detection presents three main issues: 1) objects rotate arbitrarily, necessitating the encoding of angles along with position and size; 2) the geometric relations of oriented objects are lacking in self-attention, due to the absence of interaction between content and positional queries; and 3) oriented objects cause misalignment, mainly between values and positional queries in cross-attention, making accurate classification and localization difficult. In this paper, we propose an end-to-end transformer-based oriented object detector, consisting of three dedicated modules to address these issues. First, Gaussian positional encoding is proposed to encode the angle, position, and size of oriented boxes using Gaussian distributions. Second, Wasserstein self-attention is proposed to introduce geometric relations and facilitate interaction between content and positional queries by utilizing Gaussian Wasserstein distance scores. Third, oriented cross-attention is proposed to align values and positional queries by rotating sampling points around the positional query according to their angles. Experiments on six datasets DIOR-R, a series of DOTA, HRSC2016 and ICDAR2015 show the effectiveness of our approach. Compared with previous end-to-end detectors, the OrientedFormer gains 1.16 and 1.21 AP50 on DIOR-R and DOTA-v1.0 respectively, while reducing training epochs from 3× to 1×. The codes are available at https://github.com/wokaikaixinxin/ai4rs and https://github.com/wokaikaixinxin/OrientedFormer.
Zeyu Ding, Yong Zhou, Jiaqi Zhao +4
School of Computer Science and Technology, China University of Mining and Technology, and with Mine Digitization Engineering Research Center of the Ministry of Education, and also with Innovation Research Center of Disaster Intelligent Prevention and Emergency Rescue, China University of Mining and Technology, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada
In remote sensing rotated object detection, mainstream methods suffer from two bottlenecks, directional incoherence at detector neck and task conflict at detecting head. Ulitising fourier rotation equivariance, we introduce Fourier Angle Alignment, which analyses angle information through frequency spectrum and aligns the main direction to a certain orientation. Then we propose two plug and play modules : FAAFusion and FAA Head. FAAFusion works at the detector neck, aligning the main direction of higher-level features to the lower-level features and then fusing them. FAA Head serves as a new detection head, which pre-aligns RoI features to a canonical angle and adds them to the original features before classification and regression. Experiments on DOTA-v1.0, DOTA-v1.5 and HRSC2016 show that our method can greatly improve previous work. Particularly, our method achieves new state-of-the-art results of 78.72% mAP on DOTA-v1.0 and 72.28% mAP on DOTA-v1.5 datasets with single scale training and testing, validating the efficacy of our approach in remote sensing object detection. The code is made publicly available at https://github.com/gcy0423/Fourier-Angle-Alignment .
Changyu Gu, Linwei Chen, Lin Gu +1
Beijing Institute of Technology · University of Hong Kong · The Hong Kong University of Science and Technology +2