Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games
Organizations: Robotics, Perception and Learning Lab., School of Electrical Engineering and Computer Science, Royal Institute of Technology (KTH), SE-100 44 Stockholm, Sweden.
Abstract
We study one-versus-one (1v1) quadrotor pursuit-evasion, where a pursuer and an evader are trained with reinforcement learning (RL). Direct adversarial RL tends to be unstable because opponent policies change during training, causing non-stationarity, forgetting, and strategy cycling. We propose Asynchronous Multi-Stage Population-Based training, inspired by Policy-Space Response Oracles (PSRO). Proximal Policy Optimization (PPO) trains responses against previously learned opponents held fixed during each stage. The uniform variant, AMSPB-U, samples opponents equally, while AMSPB-P combines hard prioritized fictitious self-play (PFSP) with uniform exploration to emphasize currently difficult opponents. In cross-evaluations against learned opponents, both variants improve pursuit-evasion balance over latest-opponent training. AMSPB-P scores highest while retaining competence against earlier strategies. Across varied starting positions and after larger-arena retraining, both population variants retain higher average success across the two roles than latest-opponent training. Hardware experiments with two 10 cm, approximately 40 g Crazyflie brushless quadrotors demonstrate zero-shot sim-to-real transfer against baseline controllers and learned adversaries across 42 flights. Recorded flights include captures of evaders reaching robust peak speeds of 3.31 m/s and accelerations of 17.95 m/s^2, with pursuer flight speeds up to 4.87 m/s.
Robot Policy Learning
Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.
See all 402 papers