Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Deepak Kumar Panda; Adolfo Perrusquia; Weisi Guo

cs.LGJun 26, 2025

Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Authors: Deepak Kumar Panda, Adolfo Perrusquia, Weisi Guo

Organizations: the Faculty of Engineering and Applied Sciences, Cranfield University, MK43 0AL Cranfield, U.K

Abstract

Autonomous unmanned aerial vehicles (UAVs) increasingly rely on reinforcement learning (RL) for navigation, yet global navigation satellite system (GNSS) spoofing attacks can induce out-of-distribution observation shifts that corrupt value estimation and degrade mission performance. Existing robust RL methods often lack generalization to unseen attacks. To address this, we propose a curriculum-guided adaptation framework that progressively exposes a robust policy to gradient-based adversarial observation perturbations of increasing intensity while aligning temporal-difference (TD) error distributions across curriculum stages. By preserving TD-error consistency rather than targeting specific attack models, the framework promotes resilience across attack conditions. We further derive a TD-space generalization certificate that bounds performance degradation when the TD-error distribution under a test-time attack remains sufficiently close to that of the final curriculum stage. We evaluate the framework in a UAV deconfliction environment with dynamic 3D obstacles under unseen fixed and dynamic GNSS spoofing attacks. Under fixed spoofing, the curriculum-adapted (CurAd) policy achieves approximately 99-100% mission success across evaluated traffic densities, compared with 20-24% for the best-performing robust baseline, while also outperforming the non-robust vanilla baseline. Under dynamic obstacle-luring spoofing, CurAd achieves the highest episodic rewards and reduces median mission completion steps by up to 22% relative to the best-performing robust RL baseline. These findings demonstrate that adversarial curriculum adaptation transfers resilience from synthetic perturbations to unseen GNSS spoofing attacks without attack-specific retraining, enhancing robust autonomous UAV navigation.

Figures & tables

No.	Existing Methods	Description	Research Gap Addressed in the Paper
1	Robust RL [ 42 , 43 ]	Hardens policies against adversarial disturbances using uncertainty sets, adversarial training, or worst-case optimization.	Typically assumes fixed or bounded disturbances and does not account for TD-value drift under unseen attacks, which can lead to catastrophic forgetting. Moreover, no test-time robustness certificates are provided.
2	Continual RL [ 30 , 38 ]	Adapts policies to evolving environments while mitigating catastrophic forgetting across tasks.	Primarily focuses on task transitions rather than maintaining value-distribution consistency under adversarial attacks.
3	Existing Curriculum RL and Unsupervised Domain Adaptation Techniques [ 14 , 45 ]	Improves learning by progressively increasing task difficulty while incorporating distribution alignment.	Does not provide generalization guarantees against unseen domains or adversarial attacks.
4	Model-based safe RL [ 1 , 44 ]	Learns dynamics and safety models with uncertainty estimates to support safe exploration.	Primarily addresses model uncertainty and safety constraints but does not directly characterize robustness against unseen adversarial attacks.
5	Existing GNSS Spoofing Attack Defenses [ 7 , 36 ]	Detects or mitigates spoofing using sensor redundancy, filtering, or signal-level analysis.	Often attack-specific and reliant on predefined spoofing signatures or additional sensing modalities.

TABLE I: Comparison with existing approaches and research gaps addressed by the proposed framework.

Fig. 1: Curriculum guided adaptation scheme for the action robust RL policies against incremental adversarial observations.

{\color[rgb]{0,0,0}\mathcal{L}_{\textup{crit}}=\mathcal{W}_{1}\left(\mathbf{TD}_{\textup{exp}},\mathbf{TD}_{\textup{ad}}\right).}

Algorithm 1 TD-Error Guided Curriculum Adaptation

Fig. 2: The mean and uncertainty of the convergence episodic reward plot for action robust RL with different $\alpha$ when run with 10 Monte Carlo runs.

Fig. 3: Density plot of the converged rewards from previous 100 episodes from different $\alpha$ from action robust RL algorithms over 10 Monte Carlo runs.

Fig. 4: Forgetting value

{f}^{\epsilon}_{\pi}

of vanilla and action robust RL with different

\alpha

under PGD attack when the value of

\epsilon

is varied from 1.0 to 5.0.

Policy	$\epsilon$	$\widehat{L}_{R}$	$\widehat{L}_{T}$	$\widehat{L}_{Q}$	$P_{95}\!\left(\lVert\nabla_{s}Q\rVert_{2}\right)$
Vanilla DDPG	0.00	0.414	1.509	0.515	1.097
Vanilla DDPG	0.25	0.616	1.127	0.437	0.995
Action Robust DDPG	0.00	0.505	1.420	6.047	8.069
Action Robust DDPG	0.25	0.239	1.094	5.584	8.402

TABLE II: Comparison of the Lipschitz constants for DDPG and action robust DDPG

Fig. 5: Convergence plot of the value distribution bound with the increase in

\epsilon

to ensure adaptation.

Fig. 6: Reward distributions under fixed GNSS spoofing attacks, illustrating superior reward retention of the curriculum-adapted policy across increasing aerial traffic densities with the simulations conducted across 100 testing episodes.

Fig. 7: Mission success rates under fixed GNSS spoofing attacks with increasing aerial traffic, where failure corresponds to exhausting the 500-step flight budget before reaching the destination with simulations conducted across 100 testing episodes

Fig. 8: Reward distributions under dynamic obstacle-luring GNSS spoofing attacks across increasing aerial traffic densities showing superior reward retention and lower performance variability for the curriculum-adapted policy with simulations conducted across 100 testing episodes.

Fig. 9: Mission completion steps under dynamic obstacle-luring GNSS spoofing attacks across increasing aerial traffic densities, where fewer steps indicate greater navigation efficiency with simulations conducted across 100 testing episodes.

Fig. 10: Perception-reality divergence under dynamic obstacle-luring GNSS spoofing attacks, measured as the difference between minimum perceived and true obstacle clearance, where lower values indicate greater resistance to spoofing-induced perception errors.

Fig. 11: Representative UAV trajectories, altitude profiles, and minimum obstacle clearances under a dynamic obstacle-luring spoofing attack, illustrating the ability of the curriculum-adapted policy to maintain stable navigation despite corrupted obstacle perception.

Fig. 12: Comparing the conflict distribution with expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.

Fig. 13: Sensitivity analysis comparing the performance of the curriculum adaptation trained with different adversarial strength. The analysis is conducted against spoofing attack over 100 testing episodes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 14: Comparing the test-time reward distribution with expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.

Fig. 15: Comparing the total number of non zero conflicts obtained from expert policy and the curriculum update stopping condition against spoofing attack and 8 dynamic obstacles across 100 episodes.

Fig. 16: Computational time for each curriculum stage (at each

\epsilon

) across all episodes in seconds.

Fig. 17: Boxplot of the steps the episodes for each curriculum stage.

Robust RL

Robust reinforcement learning develops agents that maintain performance and safety under uncertainty, disturbances, attacks, and distribution shifts.

See all 221 papers

Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing
Mar 30, 2026Alex Zongo, Filippos Fotiadis, Ufuk Topcu +1Robust RL Collision Avoidance
A. Zongo and P. Wei are with the Department of Mechanical and Aerospace Engineering, George Washington University, Washington, D.C. 20052, USA. · F. Fotiadis and U. Topcu are with the Oden Institute for Computational Engineering & Sciences, University of Texas at Austin, Austin, TX 78712, USA.
ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learning
Sep 17, 2026Mohsen Salehi, Karthik PattabiramanRobust RL Adversarial Attacks
The University of British Columbia Vancouver, Canada
Adversarial Reinforcement Learning for Robust Planetary Landing
Oct 14, 2026Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio +2Adversarial MARL Reinforcement Learning
ETSI Aeronáutica y del Espacio, Universidad Politécnica de Madrid, Madrid, Spain · Indra Sistemas, Madrid, Spain · Department of Systems and Industrial Engineering, University of Arizona, Tucson, USA +2

Mar 30, 2026J/K move · Enter open · S save

Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing

Alex Zongo, Filippos Fotiadis, Ufuk Topcu, Peng Wei

A. Zongo and P. Wei are with the Department of Mechanical and Aerospace Engineering, George Washington University, Washington, D.C. 20052, USA. · F. Fotiadis and U. Topcu are with the Oden Institute for Computational Engineering & Sciences, University of Texas at Austin, Austin, TX 78712, USA.

Robust RL Collision Avoidance Adversarial Perturbations Adversarial Training Cooperative MARL

Open paper AskPDF arXiv

We address robust separation assurance for small Unmanned Aircraft Systems (sUAS) under GPS degradation and spoofing via Multi-Agent Reinforcement Learning (MARL). In cooperative surveillance, each aircraft (or agent) broadcasts its GPS-derived position; when such position broadcasts are corrupted, the entire observed air traffic state becomes unreliable. We cast this state observation corruption as a zero-sum game between the agents and an adversary: with probability R, the adversary perturbs the observed state to maximally degrade each agent's safety performance. We derive a closed-form expression for this adversarial perturbation, bypassing the iterative inner optimization of adversarial training entirely and enabling linear-time evaluation in the state dimension. We show that this expression approximates the exact minimizer of the value function over the modeled uncertainty set with second-order accuracy. We further bound the safety performance gap between clean and corrupted observations, showing that it degrades at most linearly with the corruption probability under Kullback-Leibler regularization. Finally, we integrate the closed-form adversarial policy into a MARL policy gradient algorithm to obtain a robust counter-policy for the agents. In a high-density sUAS simulation, we observe near-zero collision rates under corruption levels up to 35%, outperforming a baseline policy trained without adversarial perturbations.

Abstract

Robust RL

Explore similar work

Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing

ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learning

Adversarial Reinforcement Learning for Robust Planetary Landing