Cyber Deception for Mission Surveillance via Hypergame-Theoretic Deep Reinforcement Learning
Authors: Zelin Wan, Jin-Hee Cho, Mu Zhu, Ahmed H. Anwar, Charles Kamhoua, Munindar P. Singh
Organizations: Department of Computer Science, Virginia Tech, Falls Church, VA, USA · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · US Army Research Laboratory, Adelphi, MD, USA · Department of Computer Science, North Carolina State University, Raleigh, NC 27695
Unmanned Aerial Vehicles (UAVs) are valuable for mission-critical systems like surveillance, rescue, or delivery. Not surprisingly, such systems attract cyberattacks, including Denial-of-Service (DoS) attacks to overwhelm the resources of mission drones (MDs). How can we defend UAV mission systems against DoS attacks? We adopt cyber deception as a defense strategy, in which honey drones (HDs) are proposed to bait and divert attacks. The attack and deceptive defense hinge upon radio signal strength: The attacker selects victim MDs based on their signals, and HDs attract the attacker from afar by emitting stronger signals, despite this reducing battery life. We formulate an optimization problem for the attacker and defender to identify their respective strategies for maximizing mission performance while minimizing energy consumption. To address this problem, we propose a novel approach, called HT-DRL. HT-DRL identifies optimal solutions without a long learning convergence time by taking the solutions of hypergame theory into the neural network of deep reinforcement learning. This achieves a systematic way to intelligently deceive attackers. We analyze the performance of diverse defense mechanisms under different attack strategies. Further, the HT-DRL-based HD approach outperforms existing non-HD counterparts up to two times better in mission performance while incurring low energy consumption.
Figures & tables
Fig. 1: (1) Example drone fleet with multi-hop communications consisting of MDs, HDs, RLD, and GCS. (2) Example of the Neighbor Table (NT) and Fleet Table (FT) where NT provides information about the neighboring drones of a given drone, while FT tracks the membership of the mission team, including drones executing missions, leaving the team to recharge batteries, or compromised by the attacker. Drones with a battery level lower than the energy threshold Te will leave the mission team and move to the charging station.
Fig. 2: The procedures generating the solutions by a HT-DRL agent: St is the state at round t and π(at∣St) is the probability of all actions.
Fig. 3: Hypergame-Theoretic Strategy Selection Methods by the Attacker and Defender.
Fig. 4: Conceptual model of the proposed honey-drone mission system in which both the DoS attacker and the defender employ Deep Reinforcement Learning to select their optimal signal strengths. This figure is adapted from our prior design [ 11 ] .
Symbol
Meaning
Default
ρ
Signal strength decrement interval
5
TC
Time duration of battery being charged
30
TMmax
Maximum mission duration
150
[τl,τu]
The maximum number of MDs that an HD can protect simultaneously
[2,4]
EP
Energy consumption rate by a drone
7,900 mW
EC
Energy consumption rate by a camera
4 mW
TABLE I: Key Design Parameters and Default Values
Fig. 5: Performance comparison of the considered defense approaches with respect to the ratio of completed mission tasks ( RMC ), given an attack strategy.
Fig. 6: Performance analysis of HD-HT-DRL, IDS, CD, and fixed defense, given an attack strategy with respect to energy consumption ( EC ).
Fig. 7: Performance analysis of honey drone defense strategies with respect to the ratio of completed mission tasks ( RMC ), given an attack strategy.
Fig. 8: Performance analysis of honey drone defense strategies with respect to energy consumption ( EC ), given an attack strategy.
Fig. 1: Performance analysis of HD-HT-DRL, IDS, CD, and fixed defense, given an attack strategy, with respect to the number of active, connected drones ( NAC ).
Fig. 2: Performance analysis of HD-HT-DRL, IDS, CD, and fixed defense, given an attack strategy, with respect to the attacker’s accumulated reward ( GA ).
Fig. 3: Performance analysis of HD-HT-DRL, IDS, CD, and fixed defense, given an attack strategy, with respect to the defender’s accumulated reward ( GD ).
Fig. 4: Performance analysis of honey done-based defenses with respect to the number of active, connected drones ( NAC ), under a given attack.
Fig. 5: Performance analysis of honey done-based defenses with respect to the attacker’s accumulated reward ( GA ), under a given attack.
Fig. 6: Performance analysis of honey done-based defenses with respect to the defender’s accumulated reward ( GD ), under a given attack.
Fig. 7: Effect of varying the attack budget ( ζ ) on the ratio of completed mission tasks ( RMC ).
Fig. 8: Effect of varying the attack budget ( ζ ) on energy consumption ( EC ).
Fig. 9: Effect of varying the attack budget ( ζ ) on the number of active, connected drones ( NAC ).
Fig. 10: Effect of varying the attack budget ( ζ ) on the attacker’s accumulated reward ( GA ).
Fig. 11: Effect of varying the attack budget ( ζ ) on the defender’s accumulated reward ( GD ).
Fig. 12: Probability distributions of the strategy selection in HT, DRL, and HT-DRL after the training with the two episodes.
Fig. 13: Probability distributions of the strategy selection in HT, DRL, and HT-DRL after the models are fully trained.
Autonomous UAV infiltration in dynamic contested environments remains a significant challenge due to the partially observable nature of threats and the conflicting objectives of mission efficiency versus survivability. Traditional Reinforcement Learning (RL) approaches often suffer from myopic decision-making and struggle to balance these trade-offs in real-time. To address these limitations, this paper proposes an Intent-Context Synergy Reinforcement Learning (ICS-RL) framework. The framework introduces two core innovations: (1) An LSTM-based Intent Prediction Module that forecasts the future trajectories of hostile units, transforming the decision paradigm from reactive avoidance to proactive planning via state augmentation; (2) A Context-Analysis Synergy Mechanism that decomposes the mission into hierarchical sub-tasks (safe cruise, stealth planning, and hostile breakthrough). We design a heterogeneous ensemble of Dueling DQN agents, each specialized in a specific tactical context. A dynamic switching controller based on Max-Advantage values seamlessly integrates these agents, allowing the UAV to adaptively select the optimal policy without hard-coded rules. Extensive simulations demonstrate that ICS-RL significantly outperforms baselines (Standard DDQN) and traditional methods (PSO, Game Theory). The proposed method achieves a mission success rate of 88% and reduces the average exposure frequency to 0.24 per episode, validating its superiority in ensuring robust and stealthy penetration in high-dynamic scenarios.
Jiahao Fu, Feng Yang
National Elite Institute of Engineering,Northwestern Polytechnical University, Xi’an, Shanxi, China · School of Automation, Northwestern Polytechnical University, Xi’an, Shanxi, China
As backdoor attacks in UAV-based decentralized federated learning (DFL) grow increasingly stealthy and sophisticated, existing defenses have likewise escalated in complexity. Yet these defenses, which rely heavily on outlier detection, remain vulnerable to carefully crafted backdoors. In UAV-DFL, the lack of global coordination and limited resources further render outlier-based defenses impractical. Against this backdrop, gradient spectral analysis offers a promising alternative. While prior work primarily leverages low-frequency coefficients for pairwise comparisons, it neglects to analyze the intrinsic spectral characteristics of backdoor gradients. Through empirical analysis of existing stealthy attacks, we reveal a key insight: the more effort attackers invest in mimicking benign behaviors, the more distinct the spectral concentration becomes. Motivated by this, we propose Task-Aware Spectral Energy Refine (TASER) -- a decentralized defense framework. To our knowledge, this is the first efficient backdoor defense that utilizes spectral concentration instead of complex outlier detection, enabling mitigation of stealthy attacks by structurally disrupting the backdoor task. To suppress the backdoor task, TASER preserves main-task-relevant frequency coefficients and discards others. We provide theoretical guarantees and demonstrate through experiments that TASER remains effective against stealthy backdoor attacks that bypass outlier-based defenses, achieving attack success rate below 20% and accuracy loss under 5%.
Ensuring operational safety in threat-prone environments remains a critical challenge for multi-UAV networks serving as aerial base stations. This paper proposes an efficient framework to maximize global energy efficiency (EE) while promoting safe operation through threat-aware clustering and reward-based safety enforcement. The proposed framework is executed in three steps. First, a threat-aware K-means (TAKM) algorithm determines the minimum required UAVs and computes safe initial placements. Second, an optimal matching stage assigns physical UAVs to these centroids to minimize energy expenditure. Third, a threat-aware multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm dynamically optimizes trajectories, power, and user associations. Simulation results show that the proposed framework achieves zero observed safety violations in the considered scenarios while achieving superior EE and faster convergence than other learning methods and non-clustering baselines. Compared to heuristic optimization, the proposed framework outperforms the greedy particle swarm optimization (GPSO) and achieves performance comparable to that of the optimized PSO (OPSO), while incurring significantly lower online deployment computational complexity. Furthermore, the proposed framework demonstrates effective generalization to unseen user distributions, large UAV fleets, and different threat geometries, while maintaining zero safety violations.
Faisal Al-Kamali, Hussein A. Ammar, Francois Chan +3
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada · Department of Electrical Engineering, Ibb University, Ibb, Yemen · Department of Electrical and Computer Engineering, Royal Military College, Kingston, ON, K7K 7B4, Canada +1