This article presents a solution to intercept an agile drone by another agile drone carrying a catching net. We formulate the interception as a Competitive Reinforcement Learning problem, where the interceptor and the target drone are controlled by separate policies trained with Proximal Policy Optimization (PPO). We introduce a high-fidelity simulation environment that integrates a realistic quadrotor dynamics model and a low-level control architecture implemented in JAX, which allows for fast parallelized execution on GPUs. We train the agents using low-level control, collective thrust and body rates, to achieve agile flights both for the interceptor and the target. We compare the performance of the trained policies in terms of catch rate, time to catch, and crash rate, against common heuristic baselines and show that our solution outperforms these baselines for interception of agile targets. Finally, we demonstrate the performance of the trained policies in a scaled real-world scenario using agile drones inside an indoor flight arena.
Figures & tables
Fig. 1 : A competitive reinforcement learning approach to train both a pursuer and an evader drone for agile interception tasks. Both agents learn low-level control policies that enable them to perform dynamic maneuvers in a high-fidelity simulation environment.
Fig. 2 : Control architecture used for the quadrotor dynamics simulation.
Fig. 3 : Schematic of the interception problem. Capture happens when the distance d between the evader’s centre comes within a capture distance of the pursuer’s net.
Coefficient
Value
Coefficient
Value
λcatch
10.0
λcoll
0.1
λdist
0.001
λfail
30.0
λcmd
2e-04
λbnd
1.0
TABLE I : Reward coefficients.
Hyperparameter
Value
Number of parallel environments (per agent)
1024
Rollout length
128
Learning rate
5×10−4
Discount factor
0.99
Number of PPO epochs per training data batch
15
Number of minibatches per PPO epoch
1
TABLE II : Training hyperparameters.
Fig. 4 : Comparison of learning curves and average episode length.
Pursuer mode
Evader Mode
Small arena
Large arena
PP
Hov.
APF
DRL
Hov.
APF
DRL
Catch Rate (%)
96.4
19.5
58.3
100
15.9
24.6
Evade Rate (%)
0.0
47.9
40.0
0.0
66.6
74.6
of which timeout
0.0
0.0
0.8
0.0
0.0
1.1
Crash rates (%)
TABLE III : Performances of the pursuer and the evader in a 40x40x14m (Large) and a 8x8x5m (Small) arena.
Figure 8
Fig. 7 : Simulation results: A full pursuit-evasion episode using the trained policies. The trajectory on the right is a shortened version of the full episode for more visibility. We can see the pursuer first missing the evader in a first attempt, then successfully catching it after a second attempt.
Fig. 8 : A portion of the real-world flight trajectory. The pursuer (blue) successfully caught the evader (green).
This article presents a solution to intercept an agile drone by a team of agile drone carrying catching nets. We formulate the problem as a competitive Multi-Agent Reinforcement Learning (MARL) task. To address the problem of nonstationarity and catastrophic forgetting of agents overfitting to the current opponent strategy, we train the pursuers and the evader using Multi-Agent Proximal Policy Optimization (MAPPO) with Prioritized Fictitious Self Play (PFSP). We train the agents in a high-fidelity simulator using low-level control commands, collective thrust and body rates (CTBR), to achieve agile flights for both the pursuers and the evader. We compare the performance of the trained policies in terms of catch rate, time to catch and crash rates, against heuristic baselines and show that our solution outperforms them. Ablation studies show that PFSP lead to more robust policies that can adapt to different opponent strategies, and that a low-level control commands are crucial for learning performing strategies in the pursuit-evasion task. Finally, a qualitative analysis of the learned behaviours highlights the emergence of cooperative tactics among the pursuers.
Timothée Gavin, Murat Bronz
ENAC-LAB, LAAS-RIS · *IAS, Thales LAS, Rungis, France · †Dynamic Systems, OPTIM, Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, France +2
This paper presents a methodology for learning a control policy to intercept an intruder using the 3D direction unit vector to the intruder and the interceptor state. Prior deep reinforcement learning approaches assume either relative position or distance to the intruder is available, but this information is not readily accessible in real-world applications that employ passive, monocular camera sensors. Instead, we propose a solution that leverages an analytical policy gradient method using differentiable quadrotor dynamics to learn agile interception at speeds up to 10 m/s. The proposed approach outperforms baseline methods that utilize simplified point mass dynamics by an average of 30%.
Michael Anoruo, Xiaoyu Tian, Abhishek Rathod +5
Carnegie Mellon University · Artificial Intelligence Integration Center
Medium altitude, long endurance unmanned aerial systems such as Shahed-type strike drones are an unanswered threat in modern conflicts. Conventional air defense systems are too expensive to provide a feasible large-scale solution. Quadcopter interceptors are a cheap, battle-proven alternative, but their extremely limited flight time restricts them to low altitude targets at short range, whereas fixed-wing platforms have the endurance to reach higher altitude targets from longer distances. However, such interceptors are still operated manually. Recent developments in learning-based controllers show potential for automation. In this work, a fixed-wing interceptor is presented that is controlled by a neural network that maps filtered monocular vision observations directly to actuator commands. The policy was trained in a custom Python environment using reinforcement learning and proximal policy optimization. The policy was then deployed on a fixed-wing platform and tested in real interception scenarios. As a baseline, two classical guidance controllers were also developed and tested in simulation. The policy is able to intercept targets both in simulation and in real flight. In simulation, the policy outperforms the classical controllers under most conditions. The interceptor was ultimately successfully validated in real flight by intercepting maneuvering fixed-wing targets. This novel system shows the potential of artificial intelligence for autonomous control of fixed-wing interceptor drones.