cs.CVOct 21, 2026

Early versus Late RGB-Infrared Fusion for Counter-UAV Detection: A Comparative Study

Authors: Alberto De Zan, Denis Tavaris, Gian Luca Foresti, Ivan Scagnetto

Organizations: Department of Mathematics, Computer Science, and Physics, University of Udine, Via delle Scienze 206, 33100 Udine, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper presents a novel strategy for UAV detection through visible (RGB) and infrared (IR) fusion, in order to improve accuracy and precision in harsh environments and weather conditions. In this research, two opposite approaches have been studied and implemented: the first concerns image-level (early) fusion, which merges two different spectra into a single image to which one detector is applied. On the other hand, the second approach, called decision-level (late) fusion, focuses on blending together the output bounding boxes of two different detection pipelines, aiming to create a common result that can be applied to both spectra simultaneously. Both strategies have been extensively tested in a dedicated environment, which included alternating low and high brightness scenes, different backgrounds (sky and ground), distances (5, 25 and 60 m), altitudes and zoom levels (1X, 2X and 4X). These tests allowed for the assessment of actual detection capabilities and system reliability in real-world applications. Furthermore, the two fusion strategies were evaluated and compared over a set of aligned images in order to highlight the better approach of the two. The results indicate that early fusion achieves better raw performance, since it incorporates the thermal information from the beginning of the training process. However, late fusion presents an innovative and useful detection workflow that can be applied in cases of sensor or frame misalignment. Both positive and negative aspects have also been pointed out and discussed, in order to give a more comprehensive overview. Finally, the paper outlines implementation considerations and practical recommendations based on operational priorities. Overall, the findings contribute to the research in RGB-IR fusion modalities, with the goal of enhancing UAV detection capabilities.

Figures & tables

Fig. 1 : From left to right: red, green and blue channel’s intensity maps.
Fig. 2 : Graphical representation of how layers are overlapped by Pillow library in order to get the resulting image on the right.
Fig. 7 : Visible comparison between state-of-the-art fusion strategies, exploiting IAN framework and convolutional neural networks for multimodal image generation.
ChannelMain contributionLimitations / Remarks
GreenContributes the most to the perception of luminance and fine details.Provides high structural information.
RedCrucial for discriminating between minerals and soils.Complementary for material recognition.
BlueCaptures wavelengths in the 450–495 nm range, useful for identifying water bodies and atmospheric effects.Lower contrast, limited thermal and low-light contributions, sensitive to Rayleigh scattering, noisier in long-distance imagery.
TABLE I : Summary of the characteristics of the RGB channels [ 54 ] .
Fig. 8 : Early Fusion framework.
Fig. 9 : Channel shifting process. As it can be seen, there is a clear difference between CIR and RGT representations. Color-infrared imagery uses a portion of the electromagnetic spectrum known as near infrared, which ranges from 0.70μm0.70\mu m to 1.0μm1.0\mu m . RGT uses instead far infrared. In both cases, blue wavelengths are shifted out of the visible portion of the spectrum. However, only 3 channels are considered part of the RGT representation, without considering the non-visible part of the spectrum.
Fig. 10 : From the top: RGT, RBT and GBT fusion strategies. In all the cases only 3 of the 4 channels are considered for creating the fusions. For RGT the blue channel has been omitted in favour of the thermal one. A similar type of image is usually called CIR or false-color image [ 56 ] . The other types have only been considered for a technical comparison.
DatasetPrecisionRecallmAP@50mAP@50-95F1-Score
RGT95.9%94.6%97.8%56.2%95.2%
RBT96.9%93.8%96.9%55.6%95.3%
GBT93.2%85%92.4%53.3%88.9%
FCDFusion [ 41 ]98.9%91.8%96.8%56.9%95.2%
PIAFusion [ 39 ]96.2%76.4%84.6%42.2%85.2%
DenseFuse [ 38 ]95.5%84.7%91.5%50.3%89.8%
TABLE II : Performance comparison of YOLOv8s model (11.2M parameters, 28.6B FLOPs) over standard spectra (RGB, IR), literature fusions (FCD, PIA, Dense, IFCNN) and proposed early fusion approaches (RGT, RBT, GBT). The model has been trained for 300 epochs with imgsz=640 over the same dataset, only changing the color channels and the fusion strategies for the images inside it.
Fig. 11 : The bounding box is unique for both images. As it was seen in the experiments, it could work even with misaligned frames.
Fig. 12 : Late Fusion logic.
Fig. 13 : The outdoor testing field ©\copyright Google Earth Pro. The takeoff field is highlighted in green. Inside it, in the red area, the master drone used for detection takes off. In the rest of the yellow scenario, a target drone flies at different altitudes, in order to test the capabilities of the detection models.
Fig. 14 : Ground and sky comparisons at different times and resolutions. From the top left corner of the image, it is possible to see the first line, regarding Early Fusion RGT process on the ground at different zoom levels (1X, 2X, 4X) in different times of day (9:00, 12:00, 15:00). Following it, there is a second line showing the same fusion techniques applied for detecting over a sky background. Finally, the third and fourth lines contain images of tests conducted in the same conditions presented before while using Late Fusion instead.
TimeFusionZoomBackgroundDistanceAltitude
09:00Early1 ×\times , 2 ×\times , 4 ×\timesTrees5, 25, 60 m5, 10, 20 m
09:00Early1 ×\times , 2 ×\times , 4 ×\timesSky5, 25, 60 m5, 10, 20 m
09:00Late1 ×\times , 2 ×\times , 4 ×\timesTrees5, 25, 60 m5, 10, 20 m
09:00Late1 ×\times , 2 ×\times , 4 ×\timesSky5, 25, 60 m5, 10, 20 m
12:00Early1 ×\times , 2 ×\times , 4 ×\timesTrees5, 25, 60 m5, 10, 20 m
12:00Early1 ×\times , 2 ×\times , 4 ×\timesSky5, 25, 60 m5, 10, 20 m
TABLE III : Summary of the experimental evaluation conducted at different hours of the day, under different environmental conditions. The target drone was moving at different altitudes and distances from the master drone, while covering two main backgrounds: trees and sky. In the same way, three zoom levels have been tested.
Fusion MethodmAP@50mRmF1-score
Late-Fusion (RGB-IR)72%77%73.7%
Early-Fusion (RGT)75.7%83%77.8%
TABLE IV : Comparison of mean metrics between Late-Fusion (RGB-IR) and Early-Fusion (RGT) strategies (where mR is mean Recall and mF1 is mean F1).
Fig. 15 : Comparison between blue ground truth boxes and models predicted boxes (red for Late-Fusion and green for Early-Fusion).
Figure 16
Figure 17
Figure 18
Figure 19
More in this topic

UAV Detection

See all 26 papers

Explore similar work

CardsList
Mar 9, 2026

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors

Reliable unmanned aerial vehicle (UAV) detection is critical for autonomous airspace monitoring but remains challenging when integrating sensor streams that differ substantially in resolution, perspective, and field of view. Conventional fusion methods-such as wavelet-, Laplacian-, and decision-level approaches-often fail to preserve spatial correspondence across modalities and suffer from annotation of inconsistencies, limiting their robustness in real-world settings. This study introduces two fusion strategies, Registration-aware Guided Image Fusion (RGIF) and Reliability-Gated Modality-Attention Fusion (RGMAF), designed to overcome these limitations. RGIF employs Enhanced Correlation Coefficient (ECC)-based affine registration combined with guided filtering to maintain thermal saliency while enhancing structural detail. RGMAF integrates affine and optical-flow registration with a reliability-weighted attention mechanism that adaptively balances thermal contrast and visual sharpness. Experiments were conducted on the Multi-Sensor and Multi-View Fixed-Wing (MMFW)-UAV dataset comprising 147,417 annotated air-to-air frames collected from infrared, wide-angle, and zoom sensors. Among single-modality detectors, YOLOv10x demonstrated the most stable cross-domain performance and was selected as the detection backbone for evaluating fused imagery. RGIF improved the visual baseline by 2.13% mAP@50 (achieving 97.65%), while RGMAF attained the highest recall of 98.64%. These findings show that registration-aware and reliability-adaptive fusion provides a robust framework for integrating heterogeneous modalities, substantially enhancing UAV detection performance in multimodal environments.

Ishrat Jahan, Molla E Majid, M Murugappan +5

Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5

UAV DetectionMultimodal Sensor FusionAerial Object DetectionInfrared-Visible Image Fusion
AskPDF
May 20, 2026

LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection

Detecting small unmanned aerial vehicles from RGB-infrared remote-sensing pairs remains challenging due to tiny target scale, cluttered backgrounds, and spatial misalignment between heterogeneous sensors. Existing bimodal detectors often align or fuse features without assessing the reliability of local cross-sensor correspondence, allowing mismatch artifacts to propagate into the detection head. To address this issue, we propose LER-YOLO, a reliability-aware sparse mixture-of-experts framework for misaligned RGB-infrared UAV detection. LER-YOLO first introduces an Uncertainty-Aware Target Alignment module that resamples visible features toward the infrared reference and estimates a spatial reliability map. This reliability prior is then used by a Reliability-Guided Sparse MoE Fusion module to adaptively select k experts from RGB-dominant, infrared-dominant, and interactive fusion experts, enabling trustworthy cross-modal interaction while suppressing unreliable fusion. Experiments on the public MBU benchmark under a YOLOv5s-family protocol show that LER-YOLO achieves 89.7+/-0.2% AP50 over three independent seeds, with a best result of 89.9%. Extensive ablations, parameter-matched comparisons, synthetic-shift evaluations, and complexity analysis demonstrate that the gains mainly come from reliability-guided expert routing rather than increased model capacity.

Liming Hou, Yueping Peng, Hexiang Hao +6

Engineering University of PAP, Xi’an 710086, China · Unit Command Department, Officers College of PAP, Chengdu 610213, China

Infrared Object DetectionMultispectral Object DetectionCross-Modal AlignmentMixture of ExpertsMultimodal Sensor FusionUAV Remote Sensing+1
AskPDF
Aug 12, 2026

EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection

Joint use of RGB and infrared (IR) imagery can improve UAV-view object detection, but most existing methods fuse multimodal features with static or fixed weights and therefore overlook spatially varying modality reliability. We propose EGM-Det, an entropy-guided multimodal adaptive fusion framework for RGB-IR object detection. EGM-Det employs a dual-stream architecture to preserve modality-specific representations and introduces an Entropy Offset Gate Fusion module for adaptive multi-scale fusion. The module derives shallow entropy priors from input intensity, local entropy, and cross-modal discrepancy, and uses them to guide local offset alignment and spatial-channel gated fusion. It therefore selectively aggregates reliable RGB and infrared cues instead of uniformly combining heterogeneous features. We further introduce cross-modal distillation to regularize the learned fusion gates and reduce fusion degradation. Each student branch extracts complementary knowledge from the cross-modality teacher branch matched to the main branch, while entropy-adaptive supervision emphasizes uncertain modality decisions. Experiments on DroneVehicle, LLVIP, and VEDAI demonstrate state-of-the-art performance across all three benchmarks; in particular, EGM-Det outperforms prior approaches by more than 10 percentage points on VEDAI.

Cunzheng Fan, Dawei Yan, Guanlin Wang +4

School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · School of Automation and Software Engineering, Shanxi University, Taiyuan 030006, China

Infrared Object DetectionMulti-Scale Feature FusionUncertainty-Aware Multimodal FusionMultimodal FusionAerial Object Detection
AskPDF