Small Object Detection in Complex Backgrounds with Multi-Scale Attention and Global Relation Modeling
Authors: Wenguang Tao, Xiaotian Wang, Tian Yan, Yi Wang, Jie Yan
Organizations: Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China
Small object detection under complex backgrounds remains a challenging task due to severe feature degradation, weak semantic representation, and inaccurate localization caused by downsampling operations and background interference. Existing detection frameworks are mainly designed for general objects and often fail to explicitly address the unique characteristics of small objects, such as limited structural cues and strong sensitivity to localization errors. In this paper, we propose a multi-level feature enhancement and global relation modeling framework tailored for small object detection. Specifically, a Residual Haar Wavelet Downsampling module is introduced to preserve fine-grained structural details by jointly exploiting spatial-domain convolutional features and frequency-domain representations. To enhance global semantic awareness and suppress background noise, a Global Relation Modeling module is employed to capture long-range dependencies at high-level feature stages. Furthermore, a Cross-Scale Hybrid Attention module is designed to establish sparse and aligned interactions across multi-scale features, enabling effective fusion of high-resolution details and high-level semantic information with reduced computational overhead. Finally, a Center-Assisted Loss is incorporated to stabilize training and improve localization accuracy for small objects. Extensive experiments conducted on the large-scale RGBT-Tiny benchmark demonstrate that the proposed method consistently outperforms existing state-of-the-art detectors under both IoU-based and scale-adaptive evaluation metrics. These results validate the effectiveness and robustness of the proposed framework for small object detection in complex environments.
Figures & tables
Fig. 1: Overall Framework Diagram. The input image is fed into the backbone network for feature extraction after passing through the residual Haar wavelet downsampling module (RHWD). Subsequently, the P3, P4, and P5 features are respectively input into the global relation modeling module (GRM) and cross-scale hybrid attention module (CSHA) for feature enhancement, and the detection results are finally obtained via the feature pyramid network (FPN).
Fig. 4: Schematic diagram of sampling points in P3, P4, and P5 Layers (Only 4 Heads of 8 are Shown).
Methods
Year
Param.
AP
AP 50
AP 75
YOLO [ 13 ]
2016
61.5M
10.0
26.7
5.1
Cascade RCNN [ 14 ]
2018
68.9M
15.0
33.9
11.1
Dynamic RCNN [ 15 ]
2020
41.2M
14.5
33.5
10.7
Deformable DETR [ 16 ]
2020
39.8M
12.3
30.9
7.3
TOOD [ 17 ]
2021
31.8M
13.0
30.3
9.9
VarifocalNet [ 18 ]
2021
32.5M
13.0
29.4
10.1
TABLE I: IoU-based results comparison with other advanced methods on RGBT-Tiny using COCO metrics. The top two results are presented in bold and underlined format.
Methods
Year
Param.
AP
AP 50
AP 75
YOLO [ 13 ]
2016
61.5M
24.3
37.7
28.4
Cascade RCNN [ 14 ]
2018
68.9M
30.1
44.2
35.8
Dynamic RCNN [ 15 ]
2020
41.2M
29.4
44.0
34.2
Deformable DETR [ 16 ]
2020
39.8M
28.2
45.4
32.0
TOOD [ 17 ]
2021
31.8M
27.9
43.5
31.7
VarifocalNet [ 18 ]
2021
32.5M
26.9
41.6
30.1
TABLE II: SAFit-based results comparison with other advanced methods on RGBT-Tiny using COCO metrics. The top two results are presented in bold and underlined format.
Methods
Param.
GFLOPs
AP 50
AP
Baseline
46.1M
107.7
46.1
21.6
+ RHWD Module
46.1M
108.1
47.4
21.7
+ GRM Module
55.6M
113.1
47.9
22.1
+ CSHA Module
58.2M
125.1
48.2
22.9
+ Center-Assisted Loss
58.2M
125.1
48.4
23.0
TABLE III: IoU-based ablation study results on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
Large-kernel Convolution
46.1M
107.7
46.1
21.6
Focus
46.1M
107.7
41.7
19.5
RHWD Module
46.1M
108.1
47.4
21.7
TABLE IV: IoU-based results comparison of the RHWD Module with Large-kernel Convolution and Focus on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
without
46.1M
108.1
47.4
21.7
Multi-Head Self-Attention
52.4M
112.3
46.3
21.9
GRM Module
55.6M
113.1
47.9
22.1
TABLE V: IoU-based results comparison of the GRM Module with Multi-Head Self-Attention on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
without
46.1M
108.1
47.4
21.7
Before SPPF Module
55.6M
113.1
46.4
21.6
After SPPF Module
55.6M
113.1
47.9
22.1
TABLE VI: IoU-based results comparison of the GRM Module placed Before and After SPPF on RGBT-Tiny.
Fig. 5: Visualized comparison of results between the baseline and our method on the RGBT-Tiny dataset.
Shenzhen Campus of Sun Yat-sen University · Beijing University of Posts and Telecommunications, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China +2