Uncertainty-Aware Multimodal Anti-UAV Detection via Evidential Fusion and Conflict-Discounted Belief Aggregation

Sharanda Suttorp; Seyed Sahand Mohammadi Ziabari; Ali Mohammed Mansour Alsahag

cs.CVAug 29, 2026

Uncertainty-Aware Multimodal Anti-UAV Detection via Evidential Fusion and Conflict-Discounted Belief Aggregation

Authors: Sharanda Suttorp, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansour Alsahag

Organizations: University of Amsterdam Amsterdam, The Netherlands · Department of Computer Science and Technology, SUNY Empire State University Saratoga Springs, NY, USA

Abstract

Anti-UAV perception systems must remain reliable when sensor streams degrade under occlusion, fast motion, or modality-specific failure. Existing multimodal anti-UAV systems fuse RGB and thermal streams deterministically, without modeling predictive uncertainty, and cannot express doubt when streams disagree. Evidential Deep Learning (EDL) produces calibrated per-class uncertainty in a single forward pass. EDTC already exploits this for thermal-only perception, yet cross-modal evidential fusion remains unaddressed. This paper extends EDTC to multimodal RGB-Thermal perception via Discounted Belief Fusion (DBF), which converts inter-modal conflict into uncertainty mass before aggregating stream opinions. Bounding boxes are resolved by selecting the lower-uncertainty modality. On the Anti-UAV benchmark, multimodal fusion consistently outperforms either single stream (test Acc 0.670 vs. 0.604 IR, 0.598 RGB) at real-time speed (at least 38 FPS). However, DBF is empirically indistinguishable from undiscounted averaging: near-zero inter-modal conflict on this presence-dominated benchmark leaves the discounting step inert. The fused uncertainty is well-calibrated (ECE 0.057) yet expectedly a weaker localization failure detector than spatial variance (AUROC 0.626 vs. 0.739). The null result is structural: the benchmark's near-universal presence and vacuous miss-encoding jointly suppress inter-modal conflict, a diagnosis that delimits where conflict-aware fusion provides measurable benefit.

Figures & tables

Panel of Figure 1 — Figure 1. Representative paired frames from the Anti-UAV dataset with ground-truth bounding boxes (green). The infrared (left) and RGB (right) streams differ in resolution ( $640\!\times\!512$ vs. $1920\!\times\!1080$ ) and are spatially unaligned, making direct cross-modal bounding box fusion ill-posed.

Figure 2. Multimodal uncertainty-aware detection framework. The detection branch (YOLOv5s) processes RGB and infrared streams independently, each producing per-modality semantic uncertainty via an evidential head and spatial uncertainty via a Gaussian head. Semantic opinions are aggregated across modalities via Discounted Belief Fusion (DBF), and the bounding box is resolved by selecting the lower-spatial-uncertainty modality (Dynamic Modality Selection). The fused opinion is passed through an uncertainty-gated output decision: a box is emitted only when the fused belief favors the UAV class ( $b_{\text{fused}}(\text{UAV})>b_{\text{fused}}(\text{bg})$ ) and the fused uncertainty falls below the gate threshold ( $u_{\text{fused}}<\theta_{\text{cls}}$ ); otherwise the target is reported absent.

Split	Type	Model	[email protected] $\uparrow$	mAP $\uparrow$	mIoU $\uparrow$	EDCT’s Acc $\uparrow$	AUSE σ $\downarrow$
Val.	Proposed (Det-only)	IR	$0.944\pm 0.022$	$0.531\pm 0.014$	$0.765\pm 0.017$	$0.732\pm 0.027$	$0.0593\pm 0.0005$
		RGB	$0.907\pm 0.043$	$0.596\pm 0.034$	$0.782\pm 0.033$	$0.730\pm 0.057$	$0.0443\pm 0.0011$
		ABF	$\mathbf{0.960\pm 0.011}$	$\mathbf{0.598\pm 0.015}$	$\mathbf{0.806\pm 0.009}$	$\mathbf{0.779\pm 0.020}$	$\mathbf{0.0434\pm 0.0028}$
		DBF	$\mathbf{0.960\pm 0.011}$	$\mathbf{0.598\pm 0.015}$	$\mathbf{0.806\pm 0.009}$	$\mathbf{0.779\pm 0.020}$	$\mathbf{0.0434\pm 0.0028}$
		Naive	$\mathbf{0.960\pm 0.011}$	$0.597\pm 0.016$	$\mathbf{0.806\pm 0.009}$	$\mathbf{0.779\pm 0.020}$	$\mathbf{0.0434\pm 0.0028}$
		Gated	$0.958\pm 0.008$	$0.594\pm 0.010$	$\mathbf{0.806\pm 0.009}$	$\mathbf{0.779\pm 0.020}$	$\mathbf{0.0434\pm 0.0029}$

Table 1. Localization performance and benchmark comparability (mean

\pm

std over seeds 0–2). [email protected] and mAP@[0.5:0.95] are the primary detection metrics; mIoU is mean IoU over present frames; Acc is the EDTC accuracy metric included for comparability with the original thermal-only baselines (det+track); AUSE σ is the sparsification error using

-\hat{\sigma}^{2}_{best}

(lower is better). Bold = best within each group per split.

Figure 3. Sparsification curves on the test set (signal: $-\hat{\sigma}^{2}$ , mean $\pm$ std over seeds 0-2). Each curve shows the gap between an IoU-ordered oracle and the model when frames are removed in order of decreasing spatial uncertainty. Area under this gap is $\text{AUSE}_{\sigma}$ (lower is better). IR’s Gaussian head is a substantially weaker localization failure predictor across the retention range (AUSE 0.0615), reflecting its weaker spatial uncertainty signal (Table 5 ). All four fusion operators are indistinguishable (AUSE = 0.0497), confirming the near-zero inter-modal conflict reported in Table 2 . DMS is marginally worse (0.0517).

Situation	Share of frames	Conflict $C$
Both streams detect	71.5–79.1%	$\leq 1.2\times 10^{-7}$
One stream detects	16.2–21.5%	$0$
Neither detects	4.7–7.2%	—

Table 2. Frame taxonomy and inter-modality conflict

C=\sum_{j\neq k}b^{\mathrm{rgb}}_{j}\,b^{\mathrm{ir}}_{k}

, computed from the per-stream beliefs entering fusion (test split; values are the maximum over frames in each row, with ranges spanning seeds 0–2).

		Presence discrimination				Uncertainty quality
Split	Model	Prec $\uparrow$	Recall $\uparrow$	Spec $\uparrow$	BalAcc $\uparrow$	ECE $\downarrow$	AUROC $\uparrow$
Validation	IR	0.9988 $\pm$ 0.0004	0.9719 $\pm$ 0.0163	0.9497 $\pm$ 0.0195	0.9608 $\pm$ 0.0059	0.0213 $\pm$ 0.0078	0.8770 $\pm$ 0.0562
	RGB	0.9900 $\pm$ 0.0005	0.9451 $\pm$ 0.0423	0.8494 $\pm$ 0.0084	0.8973 $\pm$ 0.0197	0.0333 $\pm$ 0.0022	0.7777 $\pm$ 0.1320
	ABF	0.9950 $\pm$ 0.0005	0.9929 $\pm$ 0.0041	0.7820 $\pm$ 0.0192	0.8874 $\pm$ 0.0085	0.0218 $\pm$ 0.0019	0.7557 $\pm$ 0.0439
	DBF	0.9950 $\pm$ 0.0004	0.9929 $\pm$ 0.0041	0.7820 $\pm$ 0.0192	0.8874 $\pm$ 0.0085	0.0218 $\pm$ 0.0019	0.7558 $\pm$ 0.0439
	Naive	0.9950 $\pm$ 0.0004	0.9929 $\pm$ 0.0041	0.7820 $\pm$ 0.0192	0.8874 $\pm$ 0.0085	0.0305 $\pm$ 0.0097	0.7621 $\pm$ 0.0468

Table 3. Fused-belief quality, evaluated independently of the box output. Presence-discrimination metrics use the fused semantic belief only (IoU-free) under heavy class imbalance (val

\approx

44:1, test

\approx

73:1 present:absent). ECE measures calibration of the belief values; AUROC measures failure (misclassification) discrimination. DMS produces no fused belief and is not assessed here. Results are mean

\pm

std over seeds 0-2. Bold = best per column within each split.

Figure 4. Reliability diagrams on the validation set (mean $\pm$ std over seeds 0-2). The gap between each bar and the diagonal equals the per-bin calibration error; ECE is the weighted mean gap. All methods concentrate mass in two bins: a mid-confidence cluster $(\approx 0.55)$ from harder frames where the detector confidence is lower, and a high-confidence cluster $(\approx 0.95-1.0)$ from the dominant presence frames. At the mid-confidence bin, IR is overconfident (bar below diagonal), while fusion operators are underconfident (bar above diagonal); RGB is best calibrated there. Naive exhibits an additional bin near 0.75 from missing-detection frames encoded with $u\approx 0.5$ (Table 4 ). DBF and ABF are identical.

Model	median $u_{\mathrm{fused}}$	max $u_{\mathrm{fused}}$	$\theta_{\mathrm{cls}}$ effect
DBF	0.009–0.011	0.046	inert
ABF	0.009–0.011	0.046	inert
Gated	0.009–0.010	0.023	inert
Naive	0.009–0.011	0.51	active ( $\theta^{\star}\!\geq\!0.55$ )

Table 4.

\theta_{\mathrm{cls}}

sweep (validation).

u_{\mathrm{fused}}

statistics on gate-eligible frames (

n\approx 60

k); ranges span seeds

0

–

2

. For DBF, ABF and Gated the threshold is inert (gate

\equiv\arg\max b_{\mathrm{fused}}

) and Acc is constant over the entire grid. For Naive the threshold is active but yields no gain: Acc plateaus at the

\arg\max

value for

\theta_{\mathrm{cls}}\geq 0.55

and degrades below it.

	Failure-detection AUROC $\uparrow$
Model	$u_{\text{fused}}$	$\hat{\sigma}^{2}_{\text{best}}$
IR	$\mathbf{0.757\pm 0.083}$	$0.560\pm 0.015$
RGB	$0.657\pm 0.066$	$\mathbf{0.748\pm 0.060}$
DBF	$0.626\pm 0.062$	$\mathbf{0.739\pm 0.045}$
ABF	$0.626\pm 0.062$	$\mathbf{0.739\pm 0.045}$
Naive	$0.635\pm 0.070$	$\mathbf{0.739\pm 0.045}$

Table 5. Failure-detection AUROC on the validation set, evaluated per signal across all six models. Evaluation is restricted to present frames conditioned on

\arg\max=\text{UAV}

; a failure is defined as

\text{IoU}<0.5

. AUROC

>0.5

indicates the signal ranks failure frames above success frames; AUROC

=0.5

is random; AUROC

<0.5

indicates the signal ranks failures below successes. Bold denotes the higher AUROC per configuration.

	EDTC’s Acc $\uparrow$			ECE $\downarrow$			$\rho(u,\text{IoU})$ $\downarrow$			$\bar{u}$ (diag.)
Attr	IR	RGB	ABF/DBF	IR	RGB	ABF/DBF	IR	RGB	ABF/DBF	IR	RGB	ABF/DBF
ALL	0.605	0.598	0.670	0.081	0.057	0.057	$-0.339$	$\mathbf{-0.442}$	$-0.288$	0.126	0.211	0.073
TC	0.560	0.553	0.632	0.110	0.083	0.077	$-0.425$	$\mathbf{-0.524}$	$-0.347$	0.156	0.247	0.092
FM	0.618	0.576	0.660	0.078	0.070	0.066	$-0.135$	$\mathbf{-0.460}$	$-0.242$	0.101	0.200	0.066
SV	0.533	0.508	0.584	0.108	0.086	0.094	$-0.371$	$\mathbf{-0.593}$	$-0.381$	0.181	0.325	0.129
LI	0.681	0.603	0.705	0.045	0.062	0.045	$+0.283$	$\mathbf{-0.400}$	$-0.036$	0.035	0.129	0.017

Table 6. Attribute-based robustness on the Anti-UAV test set. Means over seeds 0-2.

\rho

is Spearman correlation between per-frame

u_{\text{fused}}

and IoU (more negative = uncertainty better tracks localization failure).

\bar{u}

is mean

u_{\text{fused}}

over all frames (including non-detections), reported as a diagnostic of each method’s uncertainty scale (not a quality metric; not scored). Best per row within each scored metric group in bold .

Model	Params (M)	GFLOPs	FPS (val)	FPS (test)
IR	7.95	28.40	$82.56\pm 0.91$	$81.05\pm 4.07$
RGB	7.95	28.40	$77.16\pm 1.79$	$77.75\pm 1.44$
Naive	15.89	56.80	$38.39\pm 0.98$	$40.69\pm 0.94$
ABF	15.89	56.80	$38.98\pm 0.70$	$39.30\pm 0.58$
DBF	15.89	56.80	$39.67\pm 0.07$	$40.01\pm 0.95$
DMS	15.89	56.80	$39.13\pm 0.57$	$39.99\pm 1.21$

Table 7. Computational efficiency. Parameters and GFLOPs are deterministic; FPS is reported as mean

\pm

std over seeds 0-2 on an NVIDIA A100 GPU (Snellius).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Abbreviation	Definition
Task & Dataset
UAV	Unmanned Aerial Vehicle
IR	Infrared
RGB	Red-Green-Blue
TC	Thermal Crossover
FM	Fast Motion

Appendix

Table 8. List of abbreviations used throughout this paper.

Figure 5. Distribution of log $\hat{\sigma}^{2}_{\text{best}}$ conditioned on localization outcome (success: $IoU\geq 0.5$ , failure: $0<IoU<0.5$ ) for IR-only (left) and RGB-only (right) configurations, pooled across seeds 0–2 on the validation set. IR shows heavily overlapping distributions (overlap coefficient 0.800, AUROC 0.598), while RGB shows clearly separated distributions (overlap 0.552, AUROC 0.760). The contrast confirms that IR’s Gaussian head produces outcome-insensitive variance, explaining the reversal of the spatial vs. semantic uncertainty ordering observed in Table 5 .

Modality	Bin	Size range	$n_{\text{succ}}$	$n_{\text{fail}}$	$\bar{\sigma}^{2}_{\text{succ}}$	$\bar{\sigma}^{2}_{\text{fail}}$	OVL	AUROC
		( $\times 10^{-3}$ )
IR	small	$0.73$ – $3.81$	57476	1076	0.023	0.033	0.836	0.606
	medium	$3.81$ – $5.40$	58111	526	0.023	0.029	0.821	0.580
	large	$5.40$ – $30.4$	57855	516	0.022	0.028	0.729	0.619
RGB	small	$0.23$ – $2.52$	53663	1272	0.031	0.059	0.543	0.752
	medium	$2.52$ – $4.74$	54488	452	0.022	0.026	0.527	0.647

Appendix

Table 9. Target-size-stratified discriminability of the spatial uncertainty

\hat{\sigma}^{2}_{\text{best}}

for the IR-only and RGB-only detectors, on detected present frames (IoU

>

0; success: IoU

\geq

0.5, failure:

0<

IoU

<

0.5), validation set, seeds 0–2 pooled. Frames are divided into three equal-sized groups by normalized predicted target size

(w\!\cdot\!h)/A_{\text{frame}}

, with group boundaries computed per modality. OVL is the histogram overlap coefficient of

\log\hat{\sigma}^{2}_{\text{best}}

between the success and failure distributions (1 = indistinguishable, 0 = disjoint); AUROC scores

\hat{\sigma}^{2}_{\text{best}}

as a failure predictor. The IR

-

RGB gap is constant across sizes, isolating the thermal modality rather than target size as the cause.

Figure 6. Multimodal uncertainty-aware detection-tracking framework. Both the global detection branch (YOLOv5s) and local tracking branch (SiamCAR) process RGB and infrared streams independently, each producing per-modality semantic uncertainty via an evidential head and spatial uncertainty via a Gaussian head. Semantic predictions are aggregated across modalities via Discounted Belief Fusion (DBF). Spatial predictions are resolved by selecting the bounding box from the lower-uncertainty modality (Dynamic Modality Selection). State-dependent switching between detection and tracking requires both semantic ( $u_{\text{fused}}$ ) and spatial ( $\hat{\sigma}^{2}_{\text{best}}$ ) confidence to satisfy their respective thresholds simultaneously.

Configuration	Method	Acc
Detect-track	IR only	0.756
	RGB only	0.714
	DBF	0.797
Detector-only	IR only	0.771
	RGB only	0.811
	DBF	0.802

Appendix

Table 10. Best-run Acc per method under the preliminary detect-track protocol (validation set, seed 0). Each row is the best threshold configuration from a sequential calibration sweep; intermediate runs are not shown. The detector-only rows use the same protocol for direct comparability. Neither group is comparable to Table 1 , which uses the corrected protocol.

Panel of Figure 7 — Figure 7. IR-only configuration. Histograms of the three candidate switching signals across DETECT (top row) and TRACK (bottom row) modes on the validation set. Each subcaption reports the AUROC of the signal as a failure predictor.

Panel of Figure 8 — Figure 8. RGB-only configuration. Histograms of the three candidate switching signals across DETECT (top row) and TRACK (bottom row) modes on the validation set. Each subcaption reports the AUROC of the signal as a failure predictor.

Panel of Figure 9 — Figure 9. DBF configuration. Histograms of the three candidate switching signals across DETECT (top row) and TRACK (bottom row) modes on the validation set. Each subcaption reports the AUROC of the signal as a failure predictor.

Uncertainty-Aware Multimodal Fusion

Uncertainty-aware multimodal fusion combines data sources according to their reliability to improve robust, calibrated predictions.

See all 82 papers

Sep 1, 2026

Evidential Deep Learning for Multi-Modal Anti-UAV Detection

Anti-UAV systems increasingly fuse multiple sensors, yet their detection heads provide no per-modality reliability signal. This study evaluates whether evidential deep learning (EDL) heads, Dempster-Shafer (DS) evidence fusion, and uncertainty-driven temporal sensor gating improve anti-UAV detection through a controlled ablation on three benchmarks: thermal tracking (AntiUAV600), RGB-audio-RF classification (TRIDENT), and RGB-IR tracking (MM-UAV). The EDL training objective improves accuracy over retrained sigmoid baselines (+5.9 percentage points in accuracy and a tripled tracker-on-absent rate in E1; +4.8 percentage points in classification accuracy in E2, surviving a clip-clustered bootstrap, p = 0.011) and ranks classification errors substantially better (entropy UAUC approximately 0.94 vs. 0.51). The remaining components do not support their respective hypotheses. DS fusion does not outperform simple probability averaging. Dirichlet vacuity adds no ranking power beyond predictive entropy and inverts at the detection level, where extreme background imbalance causes it to encode class membership rather than error likelihood, a failure also observed for entropy and sigmoid confidence. Temporal gating preserves accuracy only when nearly inactive and yields no realised latency saving on shared-backbone hardware. The benefit of evidential learning therefore arises primarily from its training objective rather than its uncertainty estimate; a crop-level control further localises the detection-level breakdown to anchor-level evaluation rather than the learned representation.

Dmitry Golovchits, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

University of Amsterdam Amsterdam, The Netherlands

Evidential Deep Learning Dempster-Shafer Theory Object Detection Aerial Object Detection Multimodal Sensor Fusion UAV Detection

Ask PDF

Apr 17, 2026

Tri-Modal Fusion Transformers for UAV-based Object Detection

Reliable UAV object detection requires robustness to illumination changes, motion blur, and scene dynamics that suppress RGB cues. Thermal long-wave infrared (LWIR) sensing preserves contrast in low light, and event cameras retain microsecond-level temporal edges, but integrating all three modalities in a unified detector has not been systematically studied. We present a tri-modal framework that processes RGB, thermal, and event data with a dual-stream hierarchical vision transformer. At selected encoder depths, a Modality-Aware Gated Exchange (MAGE) applies inter-sensor channel and spatial gating, and a Bidirectional Token Exchange (BiTE) module performs bidirectional token-level attention with depthwise-pointwise refinement, producing resolution-preserving fused maps for a standard feature pyramid and two-stage detector. We introduce a 10,489-frame UAV dataset with synchronized and pre-aligned RGB-thermal-event streams and 24,223 annotated vehicles across day and night flights. Through 61 controlled ablations, we evaluate fusion placement, mechanism (baseline MAGE+BiTE, CSSA, GAFF), modality subsets, and backbone capacity. Tri-modal fusion improves over all dual-modal baselines, with fusion depth having a significant effect and a lightweight CSSA variant recovering most of the benefit at minimal cost. This work provides the first systematic benchmark and modular backbone for tri-modal UAV-based object detection.

Craig Iaboni, Pramod Abichandani

New Jersey Institute of Technology

Infrared Object Detection Multimodal Object Detection Object Detection Aerial Object Detection Multimodal Sensor Fusion

Ask PDF

Mar 9, 2026

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors

Reliable unmanned aerial vehicle (UAV) detection is critical for autonomous airspace monitoring but remains challenging when integrating sensor streams that differ substantially in resolution, perspective, and field of view. Conventional fusion methods-such as wavelet-, Laplacian-, and decision-level approaches-often fail to preserve spatial correspondence across modalities and suffer from annotation of inconsistencies, limiting their robustness in real-world settings. This study introduces two fusion strategies, Registration-aware Guided Image Fusion (RGIF) and Reliability-Gated Modality-Attention Fusion (RGMAF), designed to overcome these limitations. RGIF employs Enhanced Correlation Coefficient (ECC)-based affine registration combined with guided filtering to maintain thermal saliency while enhancing structural detail. RGMAF integrates affine and optical-flow registration with a reliability-weighted attention mechanism that adaptively balances thermal contrast and visual sharpness. Experiments were conducted on the Multi-Sensor and Multi-View Fixed-Wing (MMFW)-UAV dataset comprising 147,417 annotated air-to-air frames collected from infrared, wide-angle, and zoom sensors. Among single-modality detectors, YOLOv10x demonstrated the most stable cross-domain performance and was selected as the detection backbone for evaluating fused imagery. RGIF improved the visual baseline by 2.13% mAP@50 (achieving 97.65%), while RGMAF attained the highest recall of 98.64%. These findings show that registration-aware and reliability-adaptive fusion provides a robust framework for integrating heterogeneous modalities, substantially enhancing UAV detection performance in multimodal environments.

Ishrat Jahan, Molla E Majid, M Murugappan +5

Department of Computer Science and Engineering, Shahjalal University of Science & Technology, Sylhet, Bangladesh · Computer Applications Department, Academic Bridge Program, Qatar Foundation, Doha, Qatar. · Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha, Kuwait +5

UAV Detection Multimodal Sensor Fusion Aerial Object Detection Infrared-Visible Image Fusion

Ask PDF

Abstract

Uncertainty-Aware Multimodal Fusion

Explore similar work

Evidential Deep Learning for Multi-Modal Anti-UAV Detection

Tri-Modal Fusion Transformers for UAV-based Object Detection

Alignment-Aware and Reliability-Gated Multimodal Fusion for Unmanned Aerial Vehicle Detection Across Heterogeneous Thermal-Visual Sensors