Small Object Detection in Complex Backgrounds with Multi-Scale Attention and Global Relation Modeling
Authors: Wenguang Tao, Xiaotian Wang, Tian Yan, Yi Wang, Jie Yan
Organizations: Unmanned System Research Institute, Northwestern Polytechnical University, Xi’an, China · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China
Small object detection under complex backgrounds remains a challenging task due to severe feature degradation, weak semantic representation, and inaccurate localization caused by downsampling operations and background interference. Existing detection frameworks are mainly designed for general objects and often fail to explicitly address the unique characteristics of small objects, such as limited structural cues and strong sensitivity to localization errors. In this paper, we propose a multi-level feature enhancement and global relation modeling framework tailored for small object detection. Specifically, a Residual Haar Wavelet Downsampling module is introduced to preserve fine-grained structural details by jointly exploiting spatial-domain convolutional features and frequency-domain representations. To enhance global semantic awareness and suppress background noise, a Global Relation Modeling module is employed to capture long-range dependencies at high-level feature stages. Furthermore, a Cross-Scale Hybrid Attention module is designed to establish sparse and aligned interactions across multi-scale features, enabling effective fusion of high-resolution details and high-level semantic information with reduced computational overhead. Finally, a Center-Assisted Loss is incorporated to stabilize training and improve localization accuracy for small objects. Extensive experiments conducted on the large-scale RGBT-Tiny benchmark demonstrate that the proposed method consistently outperforms existing state-of-the-art detectors under both IoU-based and scale-adaptive evaluation metrics. These results validate the effectiveness and robustness of the proposed framework for small object detection in complex environments.
Figures & tables
Fig. 1: Overall Framework Diagram. The input image is fed into the backbone network for feature extraction after passing through the residual Haar wavelet downsampling module (RHWD). Subsequently, the P3, P4, and P5 features are respectively input into the global relation modeling module (GRM) and cross-scale hybrid attention module (CSHA) for feature enhancement, and the detection results are finally obtained via the feature pyramid network (FPN).
Fig. 4: Schematic diagram of sampling points in P3, P4, and P5 Layers (Only 4 Heads of 8 are Shown).
Methods
Year
Param.
AP
AP 50
AP 75
YOLO [ 13 ]
2016
61.5M
10.0
26.7
5.1
Cascade RCNN [ 14 ]
2018
68.9M
15.0
33.9
11.1
Dynamic RCNN [ 15 ]
2020
41.2M
14.5
33.5
10.7
Deformable DETR [ 16 ]
2020
39.8M
12.3
30.9
7.3
TOOD [ 17 ]
2021
31.8M
13.0
30.3
9.9
VarifocalNet [ 18 ]
2021
32.5M
13.0
29.4
10.1
TABLE I: IoU-based results comparison with other advanced methods on RGBT-Tiny using COCO metrics. The top two results are presented in bold and underlined format.
Methods
Year
Param.
AP
AP 50
AP 75
YOLO [ 13 ]
2016
61.5M
24.3
37.7
28.4
Cascade RCNN [ 14 ]
2018
68.9M
30.1
44.2
35.8
Dynamic RCNN [ 15 ]
2020
41.2M
29.4
44.0
34.2
Deformable DETR [ 16 ]
2020
39.8M
28.2
45.4
32.0
TOOD [ 17 ]
2021
31.8M
27.9
43.5
31.7
VarifocalNet [ 18 ]
2021
32.5M
26.9
41.6
30.1
TABLE II: SAFit-based results comparison with other advanced methods on RGBT-Tiny using COCO metrics. The top two results are presented in bold and underlined format.
Methods
Param.
GFLOPs
AP 50
AP
Baseline
46.1M
107.7
46.1
21.6
+ RHWD Module
46.1M
108.1
47.4
21.7
+ GRM Module
55.6M
113.1
47.9
22.1
+ CSHA Module
58.2M
125.1
48.2
22.9
+ Center-Assisted Loss
58.2M
125.1
48.4
23.0
TABLE III: IoU-based ablation study results on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
Large-kernel Convolution
46.1M
107.7
46.1
21.6
Focus
46.1M
107.7
41.7
19.5
RHWD Module
46.1M
108.1
47.4
21.7
TABLE IV: IoU-based results comparison of the RHWD Module with Large-kernel Convolution and Focus on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
without
46.1M
108.1
47.4
21.7
Multi-Head Self-Attention
52.4M
112.3
46.3
21.9
GRM Module
55.6M
113.1
47.9
22.1
TABLE V: IoU-based results comparison of the GRM Module with Multi-Head Self-Attention on RGBT-Tiny.
Methods
Param.
GFLOPs
AP 50
AP
without
46.1M
108.1
47.4
21.7
Before SPPF Module
55.6M
113.1
46.4
21.6
After SPPF Module
55.6M
113.1
47.9
22.1
TABLE VI: IoU-based results comparison of the GRM Module placed Before and After SPPF on RGBT-Tiny.
Fig. 5: Visualized comparison of results between the baseline and our method on the RGBT-Tiny dataset.
Small object detection remains a significant challenge due to feature degradation from downsampling, mutual occlusion in dense clusters, and complex background interference. To address these issues, this paper proposes FSDETR, a frequency-spatial feature enhancement framework built upon the RT-DETR baseline. By establishing a collaborative modeling mechanism, the method effectively leverages complementary structural information. Specifically, a Spatial Hierarchical Attention Block (SHAB) captures both local details and global dependencies to strengthen semantic representation. Furthermore, to mitigate occlusion in dense scenes, the Deformable Attention-based Intra-scale Feature Interaction (DA-AIFI) focuses on informative regions via dynamic sampling. Finally, the Frequency-Spatial Feature Pyramid Network (FSFPN) integrates frequency filtering with spatial edge extraction via the Cross-domain Frequency-Spatial Block (CFSB) to preserve fine-grained details. Experimental results show that with only 14.7M parameters, FSDETR achieves 13.9% APS on VisDrone 2019 and 48.95% AP50 tiny on TinyPerson, showing strong performance on small-object benchmarks. The code and models are available at https://github.com/YT3DVision/FSDETR.
Jianchao Huang, Fengming Zhang, Haibo Zhu +1
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China
Small Object Detection (SOD) is a fundamental yet challenging problem in computer vision due to its limited spatial resolution and weak visual cues. Although recent approaches have achieved remarkable advances, the background distractors in different frequency spectra still degrade the performance. In this paper, we propose a novel small object detection framework termed SFDNet, which is capable of detecting small objects via efficient spectrum-aware feature disentanglement. Specifically, we propose an Adaptive Spectrum Disentanglement (ASD) module that decomposes backbone features into multiple complementary spectral components, aiming to construct discriminative object-relevant representations by discarding the background distractors for each component. Afterwards, to strengthen the semantic consistency of the similar objects in the same class, we propose a Class-Wise Prototype Distillation (CPD) procedure, which establishes class prototypes for the object instances and enforces the compact representation by efficient prototype distillation. Extensive experiments on multiple challenging benchmarks show that SFDNet outperforms existing state-of-the-art methods by a large margin. Our code is available at https://github.com/ManOfStory/SFDNet.
Yang Guo, Zihan Yang, Feifei Kou +3
Shenzhen Campus of Sun Yat-sen University · Beijing University of Posts and Telecommunications, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China +2
Despite the remarkable progress over the past decades, accurately identifying small objects remains challenging because of their insufficient visual cues. Previous works typically attempt to construct discriminative representation of the small objects. However, the wide range frequency domain noises and label ambiguities have been greatly overlooked, which significantly hinders the accurate localization. To address these issues, we propose a novel small object detection (SOD) detector termed DyFrDet, which is able to precisely localize the small object by dynamically suppressing the background distractions in frequency domain. Specifically, we propose a Dynamic Frequency-aware Feature Pyramid Network (DyFrFPN) to adaptively suppress low-frequency redundancy and excessive high-frequency noises. The DyFrFPN transforms the hierarchical features into frequency domain representation, and introduces a Dynamic Band Predictor (DBP) to preserve the discriminative components for small object identification. Afterwards, we present a novel Label Disambiguation Module (LDM), which leverages probabilistic distributions to explicitly model and alleviate the inherent ambiguity of target labels, yielding efficient improvement in localization precision of the small objects with low-resolution. Extensive experiments demonstrate that DyFrDet achieves state-of-the-art performance across multiple benchmarks, indicating its effectiveness and robustness in various challenging scenarios. Our code is available at https://github.com/ManOfStory/DyFrDet.
Zihan Yang, Yang Guo, Hongxing Zhang +2
Beijing University of Posts and Telecommunications · Shenzhen Campus of Sun Yat-sen University