We present a learning-based approach to kinodynamic online motion planning for an Unmanned Aerial Vehicle (UAV) operating at a fixed altitude in unknown dynamic environments, where real-time avoidance of both static and dynamic obstacles must be achieved under conditions of extreme partial observability. The UAV is controlled with a single degree of freedom (yaw only), resulting in constrained, nonholonomic motion similar to fixed-wing platforms. The proposed framework integrates a behavior grid map representation with Deep Reinforcement Learning (DRL), using Proximal Policy Optimization (PPO) for stable policy learning in continuous control. The key idea is the co-design of a state representation and control policy that enables reliable navigation using only three low-cost directional range sensors, without reliance on dense sensing modalities such as LiDAR or vision-based systems. The behavior grid map dynamically aggregates sparse measurements into a structured local representation that supports real-time decision-making for obstacle avoidance and target reaching. Extensive simulations across environments of varying sizes and obstacle densities demonstrate that the proposed standard and enhanced methods achieve higher success rates than PPO variants and Model Predictive Control (MPC) (94% vs. 79--90% in small-scale high-congestion scenarios, and 83% vs. 62--71% in large-scale high-congestion scenarios), while maintaining real-time performance. Real-world experiments across four scenarios further confirm practical feasibility, with consistent target-reaching behaviour and no collisions under the tested conditions.
Figures & tables
Category
Pros
Cons
Open Problem
• UAV Path Planning and Obstacle Avoidance Ait Saadi et al. (2022) ; Lin and Peng (2021) ; Ranjbar Divkoti and Nouri-Baygi (2025) ; Qu et al. (2020) ; Ge et al. (2020) ; Zhu and Zhang (2021)
• Comprehensive analyses of path-planning approaches. • Innovations in obstacle detection for UAV.
• Challenges in dynamic obstacle avoidance. • Limitations of traditional online motion planning algorithms.
• Develop real-time adaptive solutions to dynamic obstacles.
• DRL-Based Motion Planning and Navigation Aradi (2020) ; Liu et al. (2020) ; Yan et al. (2020)
• Potential for complex online motion planning with DRL. • Integration of sensor fusion and optimal control.
• Challenges in accounting for environmental disturbances. • Issues with poor generalization in DRL.
• Integrate low-cost hardware sensors with smart software to enhance environmental understanding.
Table 1: A summary of key research papers and their contributions
Ref (Year)
Sensors
Perception / Input
Map / Representation
DRL Algorithm
Environment / Hardware
Fan et al. (2025) Fan et al. (2025)
3D LiDAR
Point-cloud
2D obstacle map via LiDAR encoder
PPO-/end-to-end
Simulated and real UAV, dynamic environment
Liu et al. (2024) Liu et al. (2024)
Airborne LiDAR
3D scans
Occupancy grid for formation
PPO-style
Simulated, multi-UAV
Huang et al. (2025) Huang et al. (2025)
Vision, sensors, and LiDAR
Multimodal (RGB, depth)
Transformer-enhanced perception
SAC and GTrXL
Simulated, dynamic environment
Almahamid (2024) AlMahamid and Grolinger (2024)
RGB and depth camera
Voxel grids
3D voxel map
Off-policy DRL
3D environments simulated
Wang et al. (2024) Wang et al. (2024)
2D LiDAR
Relative distances
End-to-end and potential field
PF-SAC
Wheeled robot, but LiDAR-based
This Work (2025)
3 rangefinders (front, left, and right)
Sparse range readings
Local behavior grid map
PPO
Simulated and real UAV, 3 sensors
Table 2: Comparison of DRL-based UAV online motion planning methods under different sensor modalities
Figure 1: Illustration of the setup: the drone is equipped with three rangefinder sensors (forward, left, and right), shown as red lines. The right sensor detects only the blue part of the ball as an obstacle, while all sensors fail to detect the box, as it is positioned in the drone’s blind spot.
Figure 2: Driving car perspective of the grid map G .
Figure 3: The red arrow indicates the robot’s direction. A normal distribution with mean μ=[μiμj]t(f) at the location of the obstacle and variance σ2 is formed, representing its presence.
Figure 4: An example of grid map G (left) and down-scaled grid map g (right).
Figure 5: Selected path steps showing an example of a UAV’s environment (left) and behavior grid map (right) during navigation. The circular obstacles are moving, directly influencing the shape of the generated path.
Figure 6: The proposed UAV online motion planning architecture integrates a behavior grid map algorithm, PPO with actor-critic neural networks, and a PID controller. Extracted states, including outputs from the behavior grid map and parameters ℓ and Ψ , serve as inputs to the PPO model. The actor-network outputs the desired angular velocity r , and the desired linear velocities are set as u=constant , v=0 , and w=0 . These desired velocities are then fed into the PID controller.
Parameter
Value
Simulation Setup
Simulation platform
Webots 2024a
Simulation area
20×20 meters
Number of obstacles
16 cylinders
Radius of cylinder
0.2 meters
Radius of target area
0.1 meters
Table 3: Simulation and PPO Training Parameters
Figure 7: A different perspective of one scene of the Webots simulation setup, featuring static and dynamic obstacles, the Crazyflie drone, and a rubber duck as the target area.
Figure 8: Comparison between the actual and desired angular velocities ( r ).
Figure 9: PPO training metrics over 100,000 steps. The red dashed line at step 55,296 indicates the selected model checkpoint, before which all metrics demonstrate stable convergence. The shaded region highlights post-selection degradation in training dynamics.
σ
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
2
77
80.77
9.30 ± 1.06
1.78 ± 0.36
3.00 ± 1.06
16
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
64
81
84.24
9.70 ± 1.38
1.79 ± 0.35
4.91 ± 2.48
Table 4: Ablation Study on Gaussian Spread ( σ ) in Grid Map
ρ
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
0.1
77
80.92
9.39 ± 1.07
1.78 ± 0.35
3.65 ± 1.59
0.5
82
81.04
9.51 ± 1.16
1.76 ± 0.36
4.07 ± 2.17
0.95
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
Table 5: Ablation Study on Forgetting Factor ( ρ )
Grid Coverage Area ( m2 )
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
1 × 1
82
80.45
9.37 ± 1.09
1.75 ± 0.37
3.42 ± 1.44
2 × 2
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
4 × 4
76
81.16
9.47 ± 1.19
1.81 ± 0.31
5.70 ± 2.22
Table 6: Ablation Study on Grid Coverage Area Around Robot
Grid Size
σ
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
32 × 32
4
81
42.53
9.47 ± 1.19
1.75 ± 0.37
3.83 ± 2.19
128 × 128
16
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
256 × 256
32
91
193.03
9.58 ± 1.11
1.75 ± 0.35
5.87 ± 2.25
Table 7: Ablation Study on Grid Resolution
Reduction Size
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
4 × 4
84
81.46
9.60 ± 1.08
1.75 ± 0.35
10.87 ± 2.12
8 × 8
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
16 × 16
67
81.76
9.16 ± 0.96
1.81 ± 0.34
3.21 ± 1.58
Table 8: Ablation Study on Down-Scaled Grid Map Size
Operator ( F )
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
Max
81
80.49
9.57 ± 1.17
1.77 ± 0.36
4.63 ± 2.65
Mean
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
Table 9: Ablation Study on Grid Reduction Operator
c1
c2
c3
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
0.1
0.1
0.8
90
76.39
10.02 ± 1.03
1.82 ± 0.43
5.68 ± 2.39
0.1
0.8
0.1
66
79.30
9.27 ± 0.99
1.79 ± 0.35
6.56 ± 2.41
0.8
0.1
0.1
84
78.40
9.50 ± 1.11
1.76 ± 0.36
3.49 ± 1.69
0.33
0.33
0.33
90
77.77
9.37 ± 1.20
1.70 ± 0.36
4.51 ± 2.44
Table 10: Ablation Study on Reward Function Coefficients (Goal Weight ( c1 ), Heading Weight ( c2 ), and Grid Map Weight ( c3 ))
Environment Size ( m2 )
Scenario
# Static Obstacles
# Dynamic Obstacles
5 × 5
No obstacle
0
0
Low
1
1
Moderate
2
2
High
4
4
20 × 20
No obstacle
0
0
Low
4
4
Table 11: Test environments with increasing levels of congestion used for evaluation.
Figure 10: Test environments with increasing size and congestion levels. Subfigures (a)-(d) correspond to 5×5 m 2 environments, while subfigures (e)-(h) correspond to 20×20 m 2 environments. Within each row, obstacle density increases from left to right. The blue cross denotes the drone’s start position, and the purple triangle indicates its initial heading direction. The filled green circle represents the target position. Gray filled circles denote static obstacles, whereas red filled circles denote dynamic obstacles. The orange line associated with each dynamic obstacle indicates its movement range.
Env
Method
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
No obstacle
PPO-NoMap-NoSshape
100
101.84
2.59 ± 0.48
N/A
4.01 ± 1.14
PPO-NoMap-WithSshape
100
101.85
2.59 ± 0.48
N/A
4.02 ± 1.11
PPO-WithMap-NoSshape
100
166.69
2.57 ± 0.48
N/A
2.34 ± 1.06
PPO-WithMap-WithSshape
100
166.30
2.57 ± 0.48
N/A
2.61 ± 1.02
MPC
100
33.20
2.44 ± 0.32
N/A
2.35 ± 1.07
Low
PPO-NoMap-NoSshape
94
102.05
2.57 ± 0.48
1.66 ± 0.31
3.97 ± 1.15
Table 12: Performance Comparison Across Different Scenarios for 5 × 5 Environment
Env
Method
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
No obstacle
PPO-NoMap-NoSshape
100
33.07
9.23 ± 1.04
N/A
12.17 ± 2.22
PPO-NoMap-WithSshape
100
33.21
9.22 ± 1.05
N/A
12.24 ± 1.94
PPO-WithMap-NoSshape
100
80.58
9.15 ± 1.07
N/A
2.74 ± 0.92
PPO-WithMap-WithSshape
100
80.20
9.15 ± 1.06
N/A
5.67 ± 1.14
MPC
100
35.79
8.97 ± 1.00
N/A
2.66 ± 1.18
Low
PPO-NoMap-NoSshape
87
33.19
9.11 ± 1.02
3.07 ± 0.48
12.21 ± 2.15
Table 13: Performance Comparison Across Different Scenarios for 20 × 20 Environment
Figure 11: Plot of the shape of the paths of different methods. a) PPO-WithMap-WithSshape (enhanced) and PPO-WithMap-NoSshape (standard) methods are achieved to the target, and other methods intersect with the obstacle. b) All methods could achieve the target except MPC.
A
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
Avg. Battery Consumption (J)
0
80
78.28
9.74 ± 1.28
1.30 ± 0.26
6.14 ± 3.35
397.53 ± 39.06
0.5
86
78.08
9.86 ± 1.41
1.29 ± 0.26
8.12 ± 3.02
401.46 ± 42.96
1.0
82
78.31
9.86 ± 1.38
1.30 ± 0.26
9.71 ± 2.75
401.77 ± 42.14
Table 14: Average battery consumption and safety comparison for different A values in high-congestion 20×20 m 2 environments.
ω
Success Rate (%)
Avg. Plan Time per Step (ms)
Avg. Path Length (m)
Avg. Clearance (m)
Avg. Smoothness
Avg. Battery Consumption (J)
(0.1)2π
79
78.45
9.74 ± 1.24
1.29 ± 0.26
6.73 ± 2.98
397.80 ± 37.82
(0.8)2π
86
78.08
9.86 ± 1.41
1.29 ± 0.26
8.12 ± 3.02
401.46 ± 42.96
(1.6)2π
80
78.10
9.71 ± 1.14
1.31 ± 0.26
7.97 ± 2.39
396.88 ± 34.73
Table 15: Average battery consumption and safety comparison for different ω values in high-congestion 20×20 m 2 environments.
Metric
Value
Network Architecture
MLP [64, 64]
Total Parameters
17,283
Model Size (Memory Footprint)
0.0659 MB
Observation Space Dimension
68
Action Space Dimension
1 (Continuous)
Average Inference Latency
0.81 ms
Table 16: Computational Resource Analysis of the Trained PPO Policy
Figure 12: Real-world experimental setup: (a) Crazyflie 2.1 nano quadcopter, (b) Crazyradio PA communication module, and experimental environments including (c) obstacle-free, (d) one static obstacle, (e) four static obstacles, and (f) one dynamic obstacle scenarios.
Episode
Path Length (m)
Avg. Plan Time (ms)
Smoothness
a
0.686
33.29
1.337
b
1.115
33.00
2.186
c
1.297
32.02
1.716
Table 17: Quantitative results of real-world UAV navigation experiments in an obstacle-free environment.
Figure 13: The plots of real-world UAV trajectories in the obstacle-free environment for different experimental episodes.
Figure 14: Experimental results for the obstacle-free scenario: (a) Composite snapshot tracking the drone’s sequential positions from takeoff to landing against a fixed background, and (b) the corresponding 2D trajectory plot of the full episode.
Episode
Path Length (m)
Avg. Clearance (m)
Avg. Plan Time (ms)
Smoothness
a
1.684
0.464
33.262
4.313
b
1.340
0.413
31.662
4.434
c
1.449
0.508
31.062
5.009
Table 18: Quantitative results of real-world UAV navigation experiments in the presence of one static obstacle.
Figure 15: The plots of real-world UAV trajectories in the environment containing one static obstacle.
Figure 16: Experimental results with a single static obstacle: (a) Composite snapshot tracking the drone’s sequential positions from takeoff to landing against a fixed background, and (b) the corresponding 2D trajectory plot of the full episode.
Episode
Goal
Path Length (m)
Avg. Clearance (m)
Avg. Plan Time (ms)
Smoothness
a
Yes
1.302
0.270
36.00
5.959
b
No
1.410
0.235
40.841
7.034
c
Yes
1.817
0.253
38.98
7.237
Table 19: Quantitative results of real-world UAV navigation experiments in the presence of four static obstacles.
Figure 17: The plots of real-world UAV trajectories in the environment containing four static obstacles.
Figure 18: Experimental results with four static obstacles: (a) Composite snapshot tracking the drone’s sequential positions from takeoff to landing against a fixed background, and (b) the corresponding 2D trajectory plot of the full episode.
Episode
Path Length (m)
Avg. Clearance (m)
Avg. Plan Time (ms)
Smoothness
a
1.336
0.400
38.39
5.234
b
0.900
0.288
42.48
5.660
c
1.393
0.420
32.96
3.977
Table 20: Quantitative results of real-world UAV navigation experiments in the presence of one dynamic obstacle.
Figure 19: The plots of real-world UAV trajectories in the environment containing one dynamic obstacle.
Figure 20: Experimental results with a dynamic obstacle: (a) Time-lapse composite snapshot tracking the drone’s position relative to the obstacle’s pendular workspace (indicated by the blue curve), and (b) the corresponding measured trajectory plot.
Figure 21: Drone Battery Voltage Discharge: Standard vs Enhanced Method
Obstacle avoidance in unmanned aerial vehicles (UAVs), as a fundamental capability, has gained increasing attention with the growing focus on spatial intelligence. However, current obstacle-avoidance methods mainly depend on limited field-of-view sensors and are ill-suited for UAV scenarios which require full-spatial awareness when the movement direction differs from the UAV's heading. This limitation motivates us to explore omnidirectional obstacle avoidance for panoramic drones with full-view perception. We first study an under explored problem setting in which a UAV must generate collision-free motion in environments with obstacles from arbitrary directions, and then construct a benchmark that consists of three representative flight tasks. Based on such settings, we propose Fly360, a two-stage perception-decision pipeline with a fixed random-yaw training strategy. At the perception stage, panoramic RGB observations are input and converted into depth maps as a robust intermediate representation. For the policy network, it is lightweight and used to output body-frame velocity commands from depth inputs. Extensive simulation and real-world experiments demonstrate that Fly360 achieves stable omnidirectional obstacle avoidance and outperforms forward-view baselines across all tasks. Our model is available at https://zxkai.github.io/fly360/
Xiangkai Zhang, Dizhe Zhang, WenZhuo Cao +5
Institute of Automation Chinese Academy of Sciences, Beijing, China · Insta360 Research, Shenzhen, China · Wuhan University,Wuhan, China.
Deep reinforcement learning (DRL) finds extensive application in autonomous drone navigation within complex, high-risk environments. However, its practical deployment faces a safety-exploration dilemma: soft penalty mechanisms encourage risky trial-and-error, while most constraint-based methods suffer degraded performance under sensor noise and intent uncertainty. We propose Dynamic-TD3, a physically enhanced framework that enforces strict safety constraints while maintaining maneuverability by modeling navigation as a Constrained Markov Decision Process (CMDP). This framework integrates an Adaptive Trajectory Relational Evolution Mechanism (ATREM) to capture long-range intentions and employs a Physically Aware Gated Kalman Filter (PAG-KF) to mitigate non-stationary observation noise. The resulting state representation drives a dual-criterion policy that balances mission efficiency against hard safety constraints via Lagrangian relaxation. In experiments with aggressive dynamic threats, this approach demonstrates superior collision avoidance performance, reduced energy consumption, and smoother flight trajectories.
This paper presents an observation-only autonomy framework for Unmanned Underwater Vehicles (UUVs) navigation in dynamic underwater environments that integrates persistent occupancy mapping, global clearance-aware planning, and risk-aware local control. The proposed pipeline constructs occupancy maps solely from onboard sonar and depth image observations, adapts a clearance-constrained global planner (GP) to provide long-horizon structure, and integrates a reinforcement learning (RL) policy to handle short-range tracking and reactive avoidance. To further support decision-making under partial observability, the system learns a compact latent state representation from onboard sensor data, encoding environmental structure, obstacle dynamics, and uncertainty. Behavior tree (BT) distillation with staged supervision is introduced to improve safety and training stability, while an uncertainty-calibrated distillation mechanism reweights teacher guidance using online latent-model uncertainty, emphasizing uncertain regimes during learning, with time-to-collision (TTC) and clearance cues remaining explicit in planning and local policy features. To demonstrate the efficacy of the framework, a reproducible multi-seed evaluation protocol is established in high-fidelity GPU-accelerated simulation using NVIDIA Isaac Sim, and performance is benchmarked against BT-only and standard RL baselines. The results obtained demonstrate improved robustness and safety under dynamic conditions, thus providing a general pipeline with a unified hybrid planning learning architecture and a reproducible methodology for robust UUV autonomy under partial observability.
Md Ether Deowan, Eleni Kelasidi
Department of Mechanical and Industrial Engineering, Faculty of Engineering, NTNU, Trondheim, Norway.