cs.ROOct 25, 2023

Multi-Agent Reinforcement Learning-Based UAV Pathfinding for Obstacle Avoidance in Stochastic Environment

Authors: Qizhen Wu, Kexin Liu, Lei Chen, Jinhu Lü

Organizations: School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China. · Advanced Research Institute of Multidisciplinary Sciences and State Key Laboratory of CNS/ATM, Beijing Institute of Technology, Beijing 100081, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Traditional methods plan feasible paths for multiple agents in the stochastic environment. However, the methods' iterations with the changes in the environment result in computation complexities, especially for the decentralized agents without a centralized planner. Although reinforcement learning provides a plausible solution because of the generalization for different environments, it struggles with enormous agent-environment interactions in training. Here, we propose a novel centralized training with decentralized execution method based on multi-agent reinforcement learning, which is improved based on the idea of model predictive control. In our approach, agents communicate only with the centralized planner to make decentralized decisions online in the stochastic environment. Furthermore, considering the communication constraint with the centralized planner, each agent plans feasible paths through the extended observation, which combines information on neighboring agents based on the distance-weighted mean field approach. Inspired by the rolling optimization approach of model predictive control, we conduct multi-step value convergence in multi-agent reinforcement learning to enhance the training efficiency, which reduces the expensive interactions in convergence. Experiment results in both comparison, ablation, and real-robot studies validate the effectiveness and generalization performance of our method.

Figures & tables

Figure 1: Overview of the proposed method. (a) Framework of RL-based decision-making for UAV pathfinding. (b) Illustration of model-based uncertainty mitigation during policy learning. (c) Comparison of unweighted mean-field and proximity-weighted mean-field in extended observations.
Figure 2: Experiment and implementation setup. (a) Simulations. (b) Indoor experiments. (c) Outdoor experiments. (d) Long-exposure photo of a flight with multiple UAVs.
ParametersValues
Episode number EkE_{k}60
Episode length KK500
Time interval τ\tau0.1 s
UAV radius ρu,i\rho_{u,i}0.5 m
Hazard sphere radius ρj,m\rho_{j,m}[0.5, 1.5] m
Hazards number, Sphere number MjM_{j}10, 5
Table I: Parameter Settings.
NetworkInput ShapeHidden LayersOutput Shape
Policymom_{o}4 layers, 128 units eachmam_{a}
Criticmo+mam_{o}+m_{a}4 layers, 256 units each1
PNNmo+mam_{o}+m_{a}4 layers, 256 units each2 mom_{o}
Table II: Network Architectures
Figure 3: Learning curves of baselines and our Method. The figures plot episode returns versus the number of training episodes. Solid curves represent the mean values across all instances, and the shaded regions indicate the standard deviation.
Hazard FrequencyMethodU3U6U9
Ti.( ×10−3\times 10^{-3} s)Re.Ra.( %\% )Ti.( ×10−3\times 10^{-3} s)Re.Ra.( %\% )Ti.( ×10−3\times 10^{-3} s)Re.Ra.( %\% )
0.05 HzRRT*3.37 ±\pm 0.31-365 ±\pm 43100 ±\pm 06.57 ±\pm 0.62-812 ±\pm 8795 ±\pm 213.40 ±\pm 1.23-1583 ±\pm 15285 ±\pm 4
ACO5.23 ±\pm 0.48-355 ±\pm 39100 ±\pm 012.93 ±\pm 1.15-788 ±\pm 8296 ±\pm 231.30 ±\pm 2.54-1418 ±\pm 14687 ±\pm 3
Dec-DDPG1.63 ±\pm 0.18-930 ±\pm 9774 ±\pm 63.13 ±\pm 0.31-2241 ±\pm 21367 ±\pm 86.93 ±\pm 0.55-4383 ±\pm 41835 ±\pm 9
MAPPO-728 ±\pm 7685 ±\pm 4-1275 ±\pm 13483 ±\pm 5-2331 ±\pm 22474 ±\pm 6
MADDPG-607 ±\pm 6490 ±\pm 3-1272 ±\pm 12783 ±\pm 4-2059 ±\pm 20378 ±\pm 5
Table III: Performance Comparison of Algorithms Across Varying Swarm Sizes and Hazard Frequencies.
Figure 4: Ablation study results of our method.
Figure 5: Scalability analysis with varying hazard frequencies. The figures illustrate success rates across scales U10 to U30 for F0.05, F0.1, and F0.2 scenarios, comparing our method against several baselines.
Figure 6: Testing instance of pathfinding in the U12 & F0.1 scenario.
Figure 7: Robustness to communication uncertainty. The figures plot success rate versus message loss rate, with the legend denoting different levels of Gaussian sensor noise.
Figure 8: Pathfinding process for UAVs in the indoor real-robot system.
Figure 9: Pathfinding process for UAVs in the outdoor real-robot system.
Figure 10: Diagram of the UAVs and hazardous areas.
Figure 14
Figure 15
Figure 16
Figure 17
More in this topic

Mean-Field MARL

See all 10 papers

Explore similar work

CardsList

  1. Decentralized End-to-End Multi-AAV Pursuit Using Predictive Spatio-Temporal Observation via Deep Reinforcement Learning

    Mar 25, 2026Yude Li, Zhexuan Zhou, Huizhe Li +6Multi-Agent SystemsMulti-Robot Coordination

    The authors are with the School of Intelligence Science and Engineering, the Guangdong Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics, and the Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China

  2. Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning

    Jul 28, 2026Yu Su, Nabil AoufLearning from DemonstrationCurriculum Learning

    School of Science and Technology, City St George’s University of London, London, United Kingdom

  3. Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment

    Mar 17, 2026Enguang Fan, Yifan Chen, Zihan Shan +3Cooperative MARLDecentralized MARL

    University of Illinois at Urbana-Champaign · Boeing Research and Technology