Early versus Late RGB-Infrared Fusion for Counter-UAV Detection: A Comparative Study

Alberto De Zan; Denis Tavaris; Gian Luca Foresti; Ivan Scagnetto

cs.CVOct 21, 2026

Early versus Late RGB-Infrared Fusion for Counter-UAV Detection: A Comparative Study

Authors: Alberto De Zan, Denis Tavaris, Gian Luca Foresti, Ivan Scagnetto

Organizations: Department of Mathematics, Computer Science, and Physics, University of Udine, Via delle Scienze 206, 33100 Udine, Italy

Abstract

This paper presents a novel strategy for UAV detection through visible (RGB) and infrared (IR) fusion, in order to improve accuracy and precision in harsh environments and weather conditions. In this research, two opposite approaches have been studied and implemented: the first concerns image-level (early) fusion, which merges two different spectra into a single image to which one detector is applied. On the other hand, the second approach, called decision-level (late) fusion, focuses on blending together the output bounding boxes of two different detection pipelines, aiming to create a common result that can be applied to both spectra simultaneously. Both strategies have been extensively tested in a dedicated environment, which included alternating low and high brightness scenes, different backgrounds (sky and ground), distances (5, 25 and 60 m), altitudes and zoom levels (1X, 2X and 4X). These tests allowed for the assessment of actual detection capabilities and system reliability in real-world applications. Furthermore, the two fusion strategies were evaluated and compared over a set of aligned images in order to highlight the better approach of the two. The results indicate that early fusion achieves better raw performance, since it incorporates the thermal information from the beginning of the training process. However, late fusion presents an innovative and useful detection workflow that can be applied in cases of sensor or frame misalignment. Both positive and negative aspects have also been pointed out and discussed, in order to give a more comprehensive overview. Finally, the paper outlines implementation considerations and practical recommendations based on operational priorities. Overall, the findings contribute to the research in RGB-IR fusion modalities, with the goal of enhancing UAV detection capabilities.

Figures & tables

Panel of Figure 1 — Fig. 1 : From left to right: red, green and blue channel’s intensity maps.

Fig. 2 : Graphical representation of how layers are overlapped by Pillow library in order to get the resulting image on the right.

Panel of Figure 3 — Fig. 7 : Visible comparison between state-of-the-art fusion strategies, exploiting IAN framework and convolutional neural networks for multimodal image generation.

Channel	Main contribution	Limitations / Remarks
Green	Contributes the most to the perception of luminance and fine details.	Provides high structural information.
Red	Crucial for discriminating between minerals and soils.	Complementary for material recognition.
Blue	Captures wavelengths in the 450–495 nm range, useful for identifying water bodies and atmospheric effects.	Lower contrast, limited thermal and low-light contributions, sensitive to Rayleigh scattering, noisier in long-distance imagery.

TABLE I : Summary of the characteristics of the RGB channels [ 54 ] .

Fig. 9 : Channel shifting process. As it can be seen, there is a clear difference between CIR and RGT representations. Color-infrared imagery uses a portion of the electromagnetic spectrum known as near infrared, which ranges from $0.70\mu m$ to $1.0\mu m$ . RGT uses instead far infrared. In both cases, blue wavelengths are shifted out of the visible portion of the spectrum. However, only 3 channels are considered part of the RGT representation, without considering the non-visible part of the spectrum.

Panel of Figure 7 — Fig. 10 : From the top: RGT, RBT and GBT fusion strategies. In all the cases only 3 of the 4 channels are considered for creating the fusions. For RGT the blue channel has been omitted in favour of the thermal one. A similar type of image is usually called CIR or false-color image [ 56 ] . The other types have only been considered for a technical comparison.

Dataset	Precision	Recall	mAP@50	mAP@50-95	F1-Score
RGT	95.9%	94.6%	97.8%	56.2%	95.2%
RBT	96.9%	93.8%	96.9%	55.6%	95.3%
GBT	93.2%	85%	92.4%	53.3%	88.9%
FCDFusion [ 41 ]	98.9%	91.8%	96.8%	56.9%	95.2%
PIAFusion [ 39 ]	96.2%	76.4%	84.6%	42.2%	85.2%
DenseFuse [ 38 ]	95.5%	84.7%	91.5%	50.3%	89.8%

TABLE II : Performance comparison of YOLOv8s model (11.2M parameters, 28.6B FLOPs) over standard spectra (RGB, IR), literature fusions (FCD, PIA, Dense, IFCNN) and proposed early fusion approaches (RGT, RBT, GBT). The model has been trained for 300 epochs with imgsz=640 over the same dataset, only changing the color channels and the fusion strategies for the images inside it.

Fig. 11 : The bounding box is unique for both images. As it was seen in the experiments, it could work even with misaligned frames.

Fig. 13 : The outdoor testing field $\copyright$ Google Earth Pro. The takeoff field is highlighted in green. Inside it, in the red area, the master drone used for detection takes off. In the rest of the yellow scenario, a target drone flies at different altitudes, in order to test the capabilities of the detection models.

Panel of Figure 12 — Fig. 14 : Ground and sky comparisons at different times and resolutions. From the top left corner of the image, it is possible to see the first line, regarding Early Fusion RGT process on the ground at different zoom levels (1X, 2X, 4X) in different times of day (9:00, 12:00, 15:00). Following it, there is a second line showing the same fusion techniques applied for detecting over a sky background. Finally, the third and fourth lines contain images of tests conducted in the same conditions presented before while using Late Fusion instead.

Time	Fusion	Zoom	Background	Distance	Altitude
09:00	Early	1 $\times$ , 2 $\times$ , 4 $\times$	Trees	5, 25, 60 m	5, 10, 20 m
09:00	Early	1 $\times$ , 2 $\times$ , 4 $\times$	Sky	5, 25, 60 m	5, 10, 20 m
09:00	Late	1 $\times$ , 2 $\times$ , 4 $\times$	Trees	5, 25, 60 m	5, 10, 20 m
09:00	Late	1 $\times$ , 2 $\times$ , 4 $\times$	Sky	5, 25, 60 m	5, 10, 20 m
12:00	Early	1 $\times$ , 2 $\times$ , 4 $\times$	Trees	5, 25, 60 m	5, 10, 20 m
12:00	Early	1 $\times$ , 2 $\times$ , 4 $\times$	Sky	5, 25, 60 m	5, 10, 20 m

TABLE III : Summary of the experimental evaluation conducted at different hours of the day, under different environmental conditions. The target drone was moving at different altitudes and distances from the master drone, while covering two main backgrounds: trees and sky. In the same way, three zoom levels have been tested.

Fusion Method	mAP@50	mR	mF1-score
Late-Fusion (RGB-IR)	72%	77%	73.7%
Early-Fusion (RGT)	75.7%	83%	77.8%

TABLE IV : Comparison of mean metrics between Late-Fusion (RGB-IR) and Early-Fusion (RGT) strategies (where mR is mean Recall and mF1 is mean F1).

Panel of Figure 15 — Fig. 15 : Comparison between blue ground truth boxes and models predicted boxes (red for Late-Fusion and green for Early-Fusion).

UAV Detection

See all 26 papers

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors
Mar 9, 2026Ishrat Jahan, Molla E Majid, M Murugappan +5UAV Detection Multimodal Sensor Fusion
Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5
LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection
May 20, 2026Liming Hou, Yueping Peng, Hexiang Hao +6Infrared Object Detection Multispectral Object Detection
Engineering University of PAP, Xi’an 710086, China · Unit Command Department, Officers College of PAP, Chengdu 610213, China
EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection
Aug 12, 2026Cunzheng Fan, Dawei Yan, Guanlin Wang +4Infrared Object Detection Multi-Scale Feature Fusion
School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · School of Automation and Software Engineering, Shanxi University, Taiyuan 030006, China

Mar 9, 2026J/K move · Enter open · S save

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors

Ishrat Jahan, Molla E Majid, M Murugappan, Muhammad E. H. Chowdhury+4

Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait+5

UAV Detection Multimodal Sensor Fusion Aerial Object Detection Infrared-Visible Image Fusion

Open paper AskPDF arXiv

Reliable unmanned aerial vehicle (UAV) detection is critical for autonomous airspace monitoring but remains challenging when integrating sensor streams that differ substantially in resolution, perspective, and field of view. Conventional fusion methods-such as wavelet-, Laplacian-, and decision-level approaches-often fail to preserve spatial correspondence across modalities and suffer from annotation of inconsistencies, limiting their robustness in real-world settings. This study introduces two fusion strategies, Registration-aware Guided Image Fusion (RGIF) and Reliability-Gated Modality-Attention Fusion (RGMAF), designed to overcome these limitations. RGIF employs Enhanced Correlation Coefficient (ECC)-based affine registration combined with guided filtering to maintain thermal saliency while enhancing structural detail. RGMAF integrates affine and optical-flow registration with a reliability-weighted attention mechanism that adaptively balances thermal contrast and visual sharpness. Experiments were conducted on the Multi-Sensor and Multi-View Fixed-Wing (MMFW)-UAV dataset comprising 147,417 annotated air-to-air frames collected from infrared, wide-angle, and zoom sensors. Among single-modality detectors, YOLOv10x demonstrated the most stable cross-domain performance and was selected as the detection backbone for evaluating fused imagery. RGIF improved the visual baseline by 2.13% mAP@50 (achieving 97.65%), while RGMAF attained the highest recall of 98.64%. These findings show that registration-aware and reliability-adaptive fusion provides a robust framework for integrating heterogeneous modalities, substantially enhancing UAV detection performance in multimodal environments.

Abstract

UAV Detection

Explore similar work

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors

LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection

EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection