Small Target Detection Based on Mask-Enhanced Attention Fusion of Visible and Infrared Remote Sensing Images
Authors: Qianqian Zhang, Xiaolong Jia, Ahmed M. Abdelmoniem, Li Zhou, Junshe An
Organizations: National Space Science Center, Chinese Academy of Sciences, Beijing, 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · School of Astronomy and Space Science, University of Chinese Academy of Sciences, Beijing, China
Targets in remote sensing images are usually small, weakly textured, and easily disturbed by complex backgrounds, challenging high-precision detection with general algorithms. Building on our earlier ESM-YOLO, this work presents ESM-YOLO+ as a lightweight visible infrared fusion network. To enhance detection, ESM-YOLO+ includes two key innovations. (1) A Mask-Enhanced Attention Fusion (MEAF) module fuses features at the pixel level via learnable spatial masks and spatial attention, effectively aligning RGB and infrared features, enhancing small-target representation, and alleviating cross-modal misalignment and scale heterogeneity. (2) Training-time Structural Representation (SR) enhancement provides auxiliary supervision to preserve fine-grained spatial structures during training, boosting feature discriminability without extra inference cost. Extensive experiments on the VEDAI and DroneVehicle datasets validate ESM-YOLO+'s superiority. The model achieves 84.71% mAP on VEDAI and 74.0% mAP on DroneVehicle, while greatly reducing model complexity, with 93.6% fewer parameters and 68.0% lower GFLOPs than the baseline. These results confirm that ESM-YOLO+ integrates strong performance with practicality for real-time deployment, providing an effective solution for high-performance small-target detection in complex remote sensing scenes.
Figures & tables
Figure 1: Overall architecture of the ESM-YOLO+. It comprises three key components: 1) the Mask-Enhanced Attention Fusion (MEAF) Module; 2) Detection Backbone and detection head; and 3) EDSR-based super-resolution branch used only during training to enhance spatial learning and removed at inference for faster detection.
Figure 2: Mask-Enhanced Attention Fusion (MEAF) Module. Pixel-level fusion module in the ESM-YOLO+ model.
Figure 3: Distribution of target sizes in the datasets.
Method
Car
Pickup
Camping
Truck
Other
Tractor
Boat
Van
mAP50 ↑
YOLOv3 [ 20 ]
84.57
72.68
67.13
61.96
43.04
65.24
37.10
58.29
61.26
YOLOv4 [ 2 ]
85.46
72.84
72.38
62.82
48.94
68.99
34.28
54.66
62.55
YOLOv5s [ 27 ]
80.81
68.48
69.06
54.71
46.79
64.29
24.25
45.96
56.79
YOLOv5m [ 27 ]
82.53
72.32
68.41
59.25
46.20
66.23
33.51
57.11
60.69
YOLOv5l [ 27 ]
82.83
72.32
69.92
63.94
48.48
63.07
40.12
56.46
62.16
YOLOv5x [ 27 ]
84.33
72.95
70.09
61.15
49.94
67.35
38.71
56.65
62.65
Table 1: Comparative Experiments of the ESM-YOLO+ model on the VEDAI dataset.
Figure 4: Visual comparison between the ESM-YOLO Model and the ESM-YOLO+ Model. The red cycles represent the false alarms, the yellow ones denote the FP detection results, and the blue ones are FN detection results.
Figure 5: Comparison of P-R Curve for ESM-YOLO (a) and ESM-YOLO+ (b).
Figure 6: Comparison of F1 curves for ESM-YOLO (a) and ESM-YOLO+ (b).
Car
Pickup
Camping
Truck
Other
Tractor
Boat
Van
mAP50 ↑
Single-Mode (RGB)
89.31
87.31
69.71
83.34
61.04
80.08
54.17
75.52
75.06
Single-Mode (IR)
87.84
86.00
78.54
72.09
45.44
59.95
55.02
81.77
70.83
MultiModal (RGB+IR)
93.64
88.76
85.66
86.65
74.80
91.39
71.57
85.20
84.71
Table 2: Comparison of single modality and multimodalities (ESM-YOLO+) on the VEDAI dataset.
Figure 7: Visualization of BEF fusion and MEAF fusion results.
Method
Backbone
mAP50 ↑
Params(M) ↓
CFT [ 31 ]
Transformer
85.3
196.9
ICAFusion [ 22 ]
Transformer
76.6
120.2
Cross-Channel-ViT [ 1 ]
Transformer
78.5
-
MOD-YOLO [ 21 ]
Transformer + CNN
59.3
24.9
MOD-YOLO-Tiny [ 21 ]
Transformer + CNN
58.4
16.0
ESM-YOLO+
CNN
84.7
5.1
Table 3: Comparative Experiments of the ESM-YOLO+ model on the VEDAI dataset.
Method
Car
Pickup
Camping
Truck
Other
Tractor
Boat
Van
mAP50 ↑
MIR-YOLO [ 45 ]
88.8
77.2
70.8
68.7
71.0
72.8
60.8
47.8
69.7
ACDF-YOLO [ 9 ]
89.8
87.9
78.5
75.3
67.3
76.3
73.7
75.9
78.1
MMFDet [ 42 ]
88.3
78.5
81.6
59.8
63.5
86.2
76.0
88.3
77.9
FFCA-YOLO [ 41 ]
89.6
85.7
78.7
85.7
48.6
81.8
61.5
67.0
74.8
L-FFCA-YOLO [ 41 ]
91.3
85.5
72.8
79.7
47.3
79.0
56.1
73.9
73.3
ESM-YOLO+
93.6
88.8
85.7
86.7
74.8
91.4
71.6
85.2
84.7
Table 4: Comparison of Detection Accuracy Performance between Specialized Algorithms for Small Target Detection in Remote Sensing MultiModal Fusion and Our Method
Method
Architecture
Car
Truck
Bus
Van
Freight Car
mAP50 ↑
Params(M) ↓
GFLOPs(G) ↓
UA-CMDet [ 25 ]
CNN
87.5
60.7
87.1
37.9
46.8
64.0
234.0
-
MKD [ 11 ]
93.5
62.5
91.9
44.5
52.7
69.0
242.0
-
Oriented R-CNN [ 33 ]
89.9
56.6
89.6
46.9
54.4
67.5
41.1
107.3
TSFADet [ 36 ]
89.9
67.9
89.9
54.0
63.7
73.1
104.7
109.8
C 2 Former-S 2 ANet [ 37 ]
Transformer
90.2
68.3
89.8
58.5
64.4
74.2
132.5
100.9
ViT-B+RVSA [ 30 ]
89.7
52.3
88.0
44.4
51.0
65.1
114.4
416.6
Table 5: Comparative Experiments of the ESM-YOLO+ model on the DroneVehicle dataset.
Figure 8: The F1 curve and P-R curve obtained by the ESM-YOLO+ model validated on the DroneVehicle dataset.
Figure 9: The visualization shows the comparison results of ESM-YOLO+ and other algorithms verified on the DroneVehicle dataset [ 25 ] . The red cycles represent the false alarms, the yellow ones denote the FP detection results, and the blue ones are FN detection results.
No.
EB
EB (-SR)
BEF
MEAF
mAP50 ↑
Params(M) ↓
1
✓
-
-
-
72.68
80.16
2
✓
-
✓
-
82.42
80.18
3
-
✓
-
✓
84.71
5.14
Table 6: Ablation experiments on MEAF Module. EB: ESM-YOLO Backbone; EB (-SR): ESM-YOLO Backbone (SR removed during inference)
Infrared small target detection (IRSTD) is important for low-altitude perception, unmanned-system warning, and security monitoring. However, weak targets in infrared imagery usually occupy only a few pixels and are easily submerged by cloud clutter, ground edges, and bright noise, making it difficult for lightweight segmentation-based methods to preserve local target structures while suppressing background interference. To address these challenges, we propose LCMamNet, a lightweight cross-scale Mamba network that progressively enhances local target structures, interacts cross-scale context in a latent space, and restores spatial details with background suppression. Specifically, a compact hierarchical encoder with cross-shaped directional bottleneck residual (CDBR) blocks strengthens direction-sensitive target structures under a small computation budget. A latent dense cross-scale fusion (LDCF) module then performs dense all-level interaction through bidirectional Mamba modeling and reorganizes the interacted features into stable hierarchical semantics. Finally, a progressive decoder selectively recovers shallow spatial details while suppressing irrelevant background textures. Extensive experiments on IRSTD-1k, NUAA-SIRST, and NUDT-SIRST show that the proposed network achieves mIoU scores of 71.25%, 79.60%, and 95.58%, respectively, with only 1.175M parameters and 6.91 GFLOPs. It also runs with a mean inference latency of 6.62 ms, and deployment results on an NVIDIA Jetson Orin NX 16G SUPER further demonstrate its practical potential for real-time edge inference. The code and checkpoints are publicly available at https://github.com/Haoyu096/LCMamNet.
Yuhao Fan, Le Hui, Yuchao Dai
School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710072, China
Infrared small target detection (IRSTD) in high-resolution images is crucial for unmanned aerial vehicle (UAV) surveillance and UAV-based ground monitoring. However, small target size, weak features, and interference from complex dynamic backgrounds make IRSTD challenging. Existing methods incur redundant computation in non-target background regions and insufficiently exploit target context, limiting detection performance. To address these issues, we propose ECFNet, an efficient coarse-to-fine IRSTD framework with attention prior-guided knowledge distillation. In the coarse stage, we design a region binary classification network (RBCN) on grid-based multi-scale feature maps to efficiently identify target-containing context region proposals. A new denoising-assisted training strategy incorporates noisy ground-truth (GT) masks into RBCN feature maps and trains the network to reconstruct the original GT masks. This auxiliary task encourages explicit learning of target-background context to better distinguish target proposals from background regions. In the fine stage, we customize a lightweight target detector to the coarse-stage region proposals to balance accuracy and efficiency. Furthermore, we introduce a knowledge distillation strategy guided by a teacher-student cross-attention prior. This strategy directs the student to focus on critical target regions, enhancing discriminative feature representations for infrared small targets. Extensive experiments on three real infrared datasets demonstrate that ECFNet outperforms existing single-stage and two-stage approaches while maintaining high real-time processing efficiency. Code: https://github.com/IVPLabs/ECFNet.
Houzhang Fang, Ruixuan Huang, Qiuhuan Chen +3
Xidian University, Xi’an, China · Huazhong University of Science and Technology, Wuhan, China
Infrared small target detection (IRSTD) methods predominantly formulate the task as pixel-level segmentation, which requires costly dense annotations and is not well suited to tiny targets with weak texture and ambiguous boundaries. To address this issue, we propose Point-to-Mask, a framework that bridges low-cost point supervision and mask-level detection through two components: a Physics-driven Adaptive Mask Generation (PAMG) module that converts point annotations into compact target masks and geometric cues, and a lightweight Radius-aware Point Regression Network (RPR-Net) that reformulates IRSTD as target center localization and effective radius regression using spatiotemporal motion cues. The two modules form a closed loop: PAMG generates pseudo masks and geometric supervision during training, while the geometric predictions of RPR-Net are fed back to PAMG for pixel-level mask recovery during inference. To facilitate systematic evaluation, we further construct SIRSTD-Pixel, a sequential dataset with refined pixel-level annotations. Experiments show that the proposed framework achieves strong pseudo-label quality, high detection accuracy, and efficient inference, approaching full-supervision performance under point-supervised settings with substantially lower annotation cost. Code and datasets will be available at: https://github.com/GaoScience/point-to-mask.
Weihua Gao, Wenlong Niu, Jie Tang +3
Key Laboratory of Electronics and Information Technology for Space Systems, National Space Science Center, Chinese Academy of Sciences, Beijing 100190, China. · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, China.