cs.LGJun 26, 2025

Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Authors: Deepak Kumar Panda, Adolfo Perrusquia, Weisi Guo

Organizations: the Faculty of Engineering and Applied Sciences, Cranfield University, MK43 0AL Cranfield, U.K

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous unmanned aerial vehicles (UAVs) increasingly rely on reinforcement learning (RL) for navigation, yet global navigation satellite system (GNSS) spoofing attacks can induce out-of-distribution observation shifts that corrupt value estimation and degrade mission performance. Existing robust RL methods often lack generalization to unseen attacks. To address this, we propose a curriculum-guided adaptation framework that progressively exposes a robust policy to gradient-based adversarial observation perturbations of increasing intensity while aligning temporal-difference (TD) error distributions across curriculum stages. By preserving TD-error consistency rather than targeting specific attack models, the framework promotes resilience across attack conditions. We further derive a TD-space generalization certificate that bounds performance degradation when the TD-error distribution under a test-time attack remains sufficiently close to that of the final curriculum stage. We evaluate the framework in a UAV deconfliction environment with dynamic 3D obstacles under unseen fixed and dynamic GNSS spoofing attacks. Under fixed spoofing, the curriculum-adapted (CurAd) policy achieves approximately 99-100% mission success across evaluated traffic densities, compared with 20-24% for the best-performing robust baseline, while also outperforming the non-robust vanilla baseline. Under dynamic obstacle-luring spoofing, CurAd achieves the highest episodic rewards and reduces median mission completion steps by up to 22% relative to the best-performing robust RL baseline. These findings demonstrate that adversarial curriculum adaptation transfers resilience from synthetic perturbations to unseen GNSS spoofing attacks without attack-specific retraining, enhancing robust autonomous UAV navigation.

Figures & tables

No.Existing MethodsDescriptionResearch Gap Addressed in the Paper
1Robust RL [ 42 , 43 ]Hardens policies against adversarial disturbances using uncertainty sets, adversarial training, or worst-case optimization.Typically assumes fixed or bounded disturbances and does not account for TD-value drift under unseen attacks, which can lead to catastrophic forgetting. Moreover, no test-time robustness certificates are provided.
2Continual RL [ 30 , 38 ]Adapts policies to evolving environments while mitigating catastrophic forgetting across tasks.Primarily focuses on task transitions rather than maintaining value-distribution consistency under adversarial attacks.
3Existing Curriculum RL and Unsupervised Domain Adaptation Techniques [ 14 , 45 ]Improves learning by progressively increasing task difficulty while incorporating distribution alignment.Does not provide generalization guarantees against unseen domains or adversarial attacks.
4Model-based safe RL [ 1 , 44 ]Learns dynamics and safety models with uncertainty estimates to support safe exploration.Primarily addresses model uncertainty and safety constraints but does not directly characterize robustness against unseen adversarial attacks.
5Existing GNSS Spoofing Attack Defenses [ 7 , 36 ]Detects or mitigates spoofing using sensor redundancy, filtering, or signal-level analysis.Often attack-specific and reliant on predefined spoofing signatures or additional sensing modalities.
TABLE I: Comparison with existing approaches and research gaps addressed by the proposed framework.
Fig. 1: Curriculum guided adaptation scheme for the action robust RL policies against incremental adversarial observations.
 {\color[rgb]{0,0,0}\mathcal{L}_{\textup{crit}}=\mathcal{W}_{1}\left(\mathbf{TD}_{\textup{exp}},\mathbf{TD}_{\textup{ad}}\right).} 
Algorithm 1 TD-Error Guided Curriculum Adaptation
Fig. 2: The mean and uncertainty of the convergence episodic reward plot for action robust RL with different α\alpha when run with 10 Monte Carlo runs.
Fig. 3: Density plot of the converged rewards from previous 100 episodes from different α\alpha from action robust RL algorithms over 10 Monte Carlo runs.
Fig. 4: Forgetting value fπϵ{f}^{\epsilon}_{\pi} of vanilla and action robust RL with different α\alpha under PGD attack when the value of ϵ\epsilon is varied from 1.0 to 5.0.
Policyϵ\epsilonL^R\widehat{L}_{R}L^T\widehat{L}_{T}L^Q\widehat{L}_{Q}P95 ⁣(∥∇sQ∥2)P_{95}\!\left(\lVert\nabla_{s}Q\rVert_{2}\right)
Vanilla DDPG0.000.4141.5090.5151.097
Vanilla DDPG0.250.6161.1270.4370.995
Action Robust DDPG0.000.5051.4206.0478.069
Action Robust DDPG0.250.2391.0945.5848.402
TABLE II: Comparison of the Lipschitz constants for DDPG and action robust DDPG
Fig. 5: Convergence plot of the value distribution bound with the increase in ϵ\epsilon to ensure adaptation.
Fig. 6: Reward distributions under fixed GNSS spoofing attacks, illustrating superior reward retention of the curriculum-adapted policy across increasing aerial traffic densities with the simulations conducted across 100 testing episodes.
Fig. 7: Mission success rates under fixed GNSS spoofing attacks with increasing aerial traffic, where failure corresponds to exhausting the 500-step flight budget before reaching the destination with simulations conducted across 100 testing episodes
Fig. 8: Reward distributions under dynamic obstacle-luring GNSS spoofing attacks across increasing aerial traffic densities showing superior reward retention and lower performance variability for the curriculum-adapted policy with simulations conducted across 100 testing episodes.
Fig. 9: Mission completion steps under dynamic obstacle-luring GNSS spoofing attacks across increasing aerial traffic densities, where fewer steps indicate greater navigation efficiency with simulations conducted across 100 testing episodes.
Fig. 10: Perception-reality divergence under dynamic obstacle-luring GNSS spoofing attacks, measured as the difference between minimum perceived and true obstacle clearance, where lower values indicate greater resistance to spoofing-induced perception errors.
Fig. 11: Representative UAV trajectories, altitude profiles, and minimum obstacle clearances under a dynamic obstacle-luring spoofing attack, illustrating the ability of the curriculum-adapted policy to maintain stable navigation despite corrupted obstacle perception.
Fig. 12: Comparing the conflict distribution with expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.
Fig. 13: Sensitivity analysis comparing the performance of the curriculum adaptation trained with different adversarial strength. The analysis is conducted against spoofing attack over 100 testing episodes.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 14: Comparing the test-time reward distribution with expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.
Fig. 15: Comparing the total number of non zero conflicts obtained from expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.
Fig. 16: Computational time for each curriculum stage (at each ϵ\epsilon ) across all episodes in seconds.
Fig. 17: Boxplot of the steps the episodes for each curriculum stage.
More in this topic

Robust RL

Robust reinforcement learning develops agents that maintain performance and safety under uncertainty, disturbances, attacks, and distribution shifts.

See all 221 papers

Explore similar work

CardsList
Mar 30, 2026

Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing

We address robust separation assurance for small Unmanned Aircraft Systems (sUAS) under GPS degradation and spoofing via Multi-Agent Reinforcement Learning (MARL). In cooperative surveillance, each aircraft (or agent) broadcasts its GPS-derived position; when such position broadcasts are corrupted, the entire observed air traffic state becomes unreliable. We cast this state observation corruption as a zero-sum game between the agents and an adversary: with probability R, the adversary perturbs the observed state to maximally degrade each agent's safety performance. We derive a closed-form expression for this adversarial perturbation, bypassing the iterative inner optimization of adversarial training entirely and enabling linear-time evaluation in the state dimension. We show that this expression approximates the exact minimizer of the value function over the modeled uncertainty set with second-order accuracy. We further bound the safety performance gap between clean and corrupted observations, showing that it degrades at most linearly with the corruption probability under Kullback-Leibler regularization. Finally, we integrate the closed-form adversarial policy into a MARL policy gradient algorithm to obtain a robust counter-policy for the agents. In a high-density sUAS simulation, we observe near-zero collision rates under corruption levels up to 35%, outperforming a baseline policy trained without adversarial perturbations.

Alex Zongo, Filippos Fotiadis, Ufuk Topcu +1

A. Zongo and P. Wei are with the Department of Mechanical and Aerospace Engineering, George Washington University, Washington, D.C. 20052, USA. · F. Fotiadis and U. Topcu are with the Oden Institute for Computational Engineering & Sciences, University of Texas at Austin, Austin, TX 78712, USA.

Robust RLCollision AvoidanceAdversarial PerturbationsAdversarial TrainingCooperative MARL
AskPDF
Sep 17, 2026

ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learning

Reinforcement learning (RL) controllers have been recently adopted for Unmanned Aerial Vehicles (UAV) navigation and control. However, they are susceptible to action-space attacks that overwrite the action commands after the policy generates them and before the actuators execute them. While most existing defenses target attacks on the policy's inputs, those addressing action-space attacks retrain the policy at training time and are not resilient to corrupted actions at runtime. We propose ASGARD, a two-phase teacher-student pipeline for making RL-based UAV control resilient to action-space attacks. In the teacher phase, an encoder combines the UAV's physical state with action-attack-related privileged information to produce an action-attack-aware latent that trains the RL control policy and a monitor that outputs corrected action commands to the actuators. In the student phase, both the encoder and the monitor are trained via supervised learning from their teacher counterparts to run on-board using only the UAV's physical state history. We evaluate ASGARD across attack scenarios targeting different action commands on UAV. We find that ASGARD is resilient to action-space attacks and completes the missions despite the attack. We further find that ASGARD generalizes to unseen attacks and remains resilient against stealthy attacks.

Mohsen Salehi, Karthik Pattabiraman

The University of British Columbia Vancouver, Canada

Robust RLAdversarial AttacksRL ControlUAV Navigation
AskPDF
Oct 14, 2026

Adversarial Reinforcement Learning for Robust Planetary Landing

Deep reinforcement learning (DRL) policies for planetary landing and other aerospace guidance, navigation, and control tasks are commonly trained under randomized disturbances to improve robustness. However, typical stochastic perturbation sampling assumes independent variations, whereas real mission disturbances often exhibit spatial and temporal correlation and cannot be reproduced through independent random draws. This work investigates adversarial reinforcement learning to discover challenging off-nominal conditions. A secondary agent generates physically constrained perturbations that degrade the performance of a landing controller, forming an asymmetric sequential zero-sum game. Training proceeds in three stages: baseline controller learning, robustness enhancement with stochastic perturbations, and an iterative adversarial phase in which the controller and disturbance generator are alternately optimized. Preliminary experiments in a three-degree-of-freedom landing environment show that adversarial disturbances reveal vulnerabilities missed by conventional randomization and provide early indications that, with a more extensive training campaign, the approach can yield improved robustness and stronger generalization under previously unseen perturbation regimes.

Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio +2

ETSI Aeronáutica y del Espacio, Universidad Politécnica de Madrid, Madrid, Spain · Indra Sistemas, Madrid, Spain · Department of Systems and Industrial Engineering, University of Arizona, Tucson, USA +2

Adversarial MARLReinforcement LearningRobust RLAdversarial Perturbations
AskPDF