Threat-Aware Energy-Efficient Deployment for Dynamic UAV Networks: A Multi-Agent RL Approach

Faisal Al-Kamali; Hussein A. Ammar; Francois Chan; James H. Bayes; Yasser Gadallah; Mohamed H. Ahmed

cs.ITSep 25, 2026

Threat-Aware Energy-Efficient Deployment for Dynamic UAV Networks: A Multi-Agent RL Approach

Authors: Faisal Al-Kamali, Hussein A. Ammar, Francois Chan, James H. Bayes, Yasser Gadallah, Mohamed H. Ahmed

Organizations: School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada · Department of Electrical Engineering, Ibb University, Ibb, Yemen · Department of Electrical and Computer Engineering, Royal Military College, Kingston, ON, K7K 7B4, Canada · Department of Electronics and Communications Engineering, The American University in Cairo, New Cairo 11835, Egypt

Abstract

Ensuring operational safety in threat-prone environments remains a critical challenge for multi-UAV networks serving as aerial base stations. This paper proposes an efficient framework to maximize global energy efficiency (EE) while promoting safe operation through threat-aware clustering and reward-based safety enforcement. The proposed framework is executed in three steps. First, a threat-aware K-means (TAKM) algorithm determines the minimum required UAVs and computes safe initial placements. Second, an optimal matching stage assigns physical UAVs to these centroids to minimize energy expenditure. Third, a threat-aware multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm dynamically optimizes trajectories, power, and user associations. Simulation results show that the proposed framework achieves zero observed safety violations in the considered scenarios while achieving superior EE and faster convergence than other learning methods and non-clustering baselines. Compared to heuristic optimization, the proposed framework outperforms the greedy particle swarm optimization (GPSO) and achieves performance comparable to that of the optimized PSO (OPSO), while incurring significantly lower online deployment computational complexity. Furthermore, the proposed framework demonstrates effective generalization to unseen user distributions, large UAV fleets, and different threat geometries, while maintaining zero safety violations.

Figures & tables

Aspect	[ 7 ]	[ 15 ]	[ 16 ]	[ 17 ]	[ 18 ]	This Work
User distribution	Uniform	Uniform	Uniform	Uniform	Real-world	Non-uniform
Threat-zone awareness	✗	✗	✗	✗	✗	✓
UAV count optimization	✗	✗	✗	✗	✗	✓
User association strategy	Dynamic	Fixed	Dynamic	Fixed	Fixed	Dynamic
Clustering scheme	None	MEM	None	Hotspot-based	GAK-means	TAKM
Operational area	$1000\times 1000$ m 2	$1000\times 1000$ m 2	$100\times 100$ m 2	$500\times 500$ m 2	$2000\times 2000$ m 2	$4000\times 4000$ m 2

TABLE I : Comparison Between the Proposed Framework and Related Works

Symbol	Description	Symbol	Description	Symbol	Description
$a,b$	Environment-dependent parameters	$a_{m,k}[t]$	User association indicator	$\mathbf{A}[t]$	User association matrix
$\mathbf{a}_{m}[t]$	Action of UAV $m$	$A_{r}$	Rotor disc area	$B$	System bandwidth
$c_{1},c_{2}$	PSO Acceleration coefficients	$\mathbf{C}_{m}$	TAKM cluster centroid $m$	$C_{m}[t]$	Total power consumption of UAV $m$
$d_{m,k}[t]$	Distance between UAV $m$ and user $k$	$d_{\min}$	Minimum separation distance	$d_{\mathrm{safe}}$	Safety margin around threat zones
$d_{f}$	Fuselage drag ratio	$D_{n}$	Minimum distance to threat $n$	$\mathbf{g}_{n}$	Center of threat zone $n$
$g_{m,k}[t]$	Channel power gain	$H_{m}$	Altitude of UAV $m$	$I$	Max iterations in TAKM

TABLE II : List of Notations

\textbf{a}_{m}[t]=\mathrm{clip}\big(\pi_{\mathbf{U}_{m}}(o_{m})+\mathcal{N}(0,\sigma_{e}^{2}),0,1\big)

Algorithm 3 The Proposed MATD3 with TAKM clustering and pre-placement

Algorithm	Computational Complexity	Approximate Operations
MATD3	$\mathcal{O}(MKI)+\mathcal{O}(M^{3})+\mathcal{O}\!\left(Lt_{\max}\!\left(2M\sum\limits_{i=0}^{L_{a}}N_{a}^{(i)}N_{a}^{(i+1)}+4\sum\limits_{j=0}^{L_{c}}N_{c}^{(j)}N_{c}^{(j+1)}\right)\right)+\mathcal{O}(Lt_{\max}MK)$	Offline Training: $6.4\times 10^{10}$ , Online Deployment: $4.7\times 10^{5}$
MADDPG	$\mathcal{O}(MKI)+\mathcal{O}(M^{3})+\mathcal{O}\!\left(Lt_{\max}\!\left(2M\sum\limits_{i=0}^{L_{a}}N_{a}^{(i)}N_{a}^{(i+1)}+2\sum\limits_{j=0}^{L_{c}}N_{c}^{(j)}N_{c}^{(j+1)}\right)\right)+\mathcal{O}(Lt_{\max}MK)$	Offline Training: $5.8\times 10^{10}$ , Online Deployment: $4.7\times 10^{5}$
PSO	$\mathcal{O}(MKI)+\mathcal{O}(M^{3})+\mathcal{O}\!\left(I_{\text{PSO}}N_{p}(MK+M^{2}+MN)t_{\max}\right)$	Online Deployment: $1.6\times 10^{10}$

TABLE III : Computational Complexity Comparison of Different Algorithms

Parameter	$B$	$K$	$N$	$f$
Value	10 MHz	400	2	1 GHz
Parameter	$\sigma_{n}^{2}$	$P_{\max}$	$V_{\max}$	$\delta_{t}$
Value	-111 dBm	33 dBm	10 m/s	300 ms
Parameter	$d_{safe}$	$\zeta_{saf}$	$d_{\min}$	$H$
Value	10 m	5	10 m	300 m
Parameter	$a$	$b$	$\gamma_{\text{LoS}}$	$\gamma_{\text{NLoS}}$

TABLE IV : Simulation Parameters

Fig. 3 : Clustering using the CKM Algorithm.

Fig. 4 : Clustering using the proposed TAKM Algorithm.

Cluster	CKM		TAKM
Cluster	Threat 1	Threat 2	Threat 1	Threat 2
Cluster 1	3269.50	1134.33	3274.17	1138.50
Cluster 2	2754.85	2139.65	2766.54	2150.54
Cluster 3	151.31 (UNSAFE)	2285.86	510	1784.66
Cluster 4	2018.21	1668.77	2240.12	1785.94

TABLE V : Distance Between CKM/TAKM Cluster Centroids and Threats

Cluster	Cluster Centroid		Number of GUs
Cluster	CKM	TAKM	CKM	TAKM
Cluster 1	[3224.35, 3080.82]	[3224.35, 3088.92]	65	65
Cluster 2	[3239.34, 602.82]	[3249.8, 596]	200	198
Cluster 3	[909.04, 1145]	[1060, 1170]	52	55
Cluster 4	[739.47, 3209.3]	[727.2, 3433.1]	83	82

TABLE VI : Cluster Position and Number of GUs

Parameter	Value
Number of episodes ( $L$ )	400
Episode duration	30 s (100 time slots)
Discount factor $\alpha$	$0.95$
Soft updated factor $\tau$	$0.005$
Actor network	3 hidden layers, 256 neurons each
Critic network	3 hidden layers, 256 neurons each

TABLE VII : MARL Simulation Parameters

Fig. 5 : Safety violations versus Episodes for MATD3 with CKM clustering and Pre-placement.

Fig. 6 : The average reward comparison between the proposed scheme and the benchmark schemes.

Panel of Figure 15 — Fig. 7 : Safety violations versus Episodes for the proposed scheme and the benchmark schemes.

Fig. 8 : Rate violations percentage vs. episodes for the proposed scheme and benchmark schemes.

Fig. 9 : The average reward performance with different altitudes for the proposed scheme.

Fig. 10 : Number of served users at different altitudes for the proposed scheme.

Fig. 11 : The average reward performance with different number of users for the proposed scheme.

Method	Cluster 1	Cluster 2	Cluster 3	Cluster 4	SDNU
CKM	65	200	52	83	58.78
TAKM	65	198	55	82	57.40
MATD3	64	183	67	86	48.66

TABLE VIII : Comparison of User Distribution.

Fig. 12 : The trajectories of UAVs for the proposed scheme.

Fig. 13 : Distribution of user rates at final UAV positions.

Fig. 14 : Energy efficiency versus time steps for the final trajectories in the proposed and PSO schemes.

Scenario	User Distribution	EE (b/J)
Training	rng (1)	$4.48\times 10^{6}$
Testing (Same)	rng (1)	$4.24\times 10^{6}$
Testing (Different)	rng (4)	$4.10\times 10^{6}$
Testing (Different)	rng (10)	$4.06\times 10^{6}$
Testing (Different)	rng (11)	$4.12\times 10^{6}$
Testing (Different)	rng (14)	$4.22\times 10^{6}$

TABLE IX : Reproducibility and Robustness Evaluation of the MATD3 Framework

Fig. 15 : UAV trajectories under a new threat configuration with different zone centers and radii.

Fig. 16 : Average reward versus episode for the proposed scheme at different observation scenarios.

UAV Communication

See all 8 papers

Apr 2, 2026

PRO-SPECT: Probabilistically Safe Scalable Planning for Energy-Aware Coordinated UAV-UGV Teams in Stochastic Environments

We consider energy-aware planning for an unmanned aerial vehicle (UAV) and unmanned ground vehicle (UGV) team operating in a stochastic environment. The UAV must visit a set of air points in minimum time while respecting energy constraints, relying on the UGV as a mobile charging station. Unlike prior work that assumed deterministic travel times or used fixed robustness margins, we model travel times as random variables and bound the probability of failure (energy depletion) across the entire mission to a user-specified risk level. We formulate the problem as a Mixed-Integer Program and propose PRO-SPECT, a polynomial-time algorithm that generates risk-bounded plans. The algorithm supports both offline planning and online re-planning, enabling the team to adapt to disturbances while preserving the risk bound. We provide theoretical results on solution feasibility and time complexity. We also demonstrate the performance of our method via numerical comparisons and simulations.

Roger Fowler, Cahit Ikbal Er, Benjamin Johnsenberg +1

Khoury Department of Computer Sciences, Northeastern University, Boston, MA · The Charles Stark Draper Laboratory, Inc., Cambridge, MA · Department of Mechanical and Industrial Engineering, Northeastern University, Boston, MA +1

Motion Planning Multi-Robot Coordination

Ask PDF

Apr 30, 2026

Dynamic-TD3: A Novel Algorithm for UAV Path Planning with Dynamic Obstacle Trajectory Prediction

Deep reinforcement learning (DRL) finds extensive application in autonomous drone navigation within complex, high-risk environments. However, its practical deployment faces a safety-exploration dilemma: soft penalty mechanisms encourage risky trial-and-error, while most constraint-based methods suffer degraded performance under sensor noise and intent uncertainty. We propose Dynamic-TD3, a physically enhanced framework that enforces strict safety constraints while maintaining maneuverability by modeling navigation as a Constrained Markov Decision Process (CMDP). This framework integrates an Adaptive Trajectory Relational Evolution Mechanism (ATREM) to capture long-range intentions and employs a Physically Aware Gated Kalman Filter (PAG-KF) to mitigate non-stationary observation noise. The resulting state representation drives a dual-criterion policy that balances mission efficiency against hard safety constraints via Lagrangian relaxation. In experiments with aggressive dynamic threats, this approach demonstrates superior collision avoidance performance, reduced energy consumption, and smoother flight trajectories.

Wentao Chen, Jingtang Chen, Mingjian Fu +4

Reinforcement Learning UAV Navigation Constrained RL Dynamic Obstacle Avoidance Trajectory Prediction

Ask PDF

Mar 17, 2026

Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment

Autonomous Unmanned Aerial Vehicle (UAV) swarms are increasingly used as rapidly deployable aerial relays and sensing platforms, yet practical deployments must operate under partial observability and intermittent peer-to-peer connectivity. We present a graph-based multi-agent reinforcement learning framework trained under centralized training with decentralized execution (CTDE): a centralized critic and global state are available only during training, while each UAV executes a shared policy using local observations and messages from nearby neighbors. Under restricted communication, neighbor relations are induced by an SNR-threshold connectivity graph. Our architecture encodes local agent state and nearby entities with an agent-entity attention module and aggregates inter-UAV messages with neighbor self-attention over a signal-quality-limited communication graph defined by a channel model. We evaluate the framework on a cooperative relay-deployment task, DroneConnect. Experimental results show that the proposed method achieves an approximately 12% increase in target coverage over MAPPO under restricted communication and partial observability, while remaining competitive with a mixed-integer linear programming (MILP)-based offline upper bound with full node observability.

Enguang Fan, Yifan Chen, Zihan Shan +3

University of Illinois at Urbana-Champaign · Boeing Research and Technology

Cooperative MARL Decentralized MARL Multi-Agent Communication Partially Observable RL

Ask PDF

Abstract

UAV Communication

Explore similar work

PRO-SPECT: Probabilistically Safe Scalable Planning for Energy-Aware Coordinated UAV-UGV Teams in Stochastic Environments

Dynamic-TD3: A Novel Algorithm for UAV Path Planning with Dynamic Obstacle Trajectory Prediction

Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment