math.OCApr 14, 2026

A Comparison of Reinforcement Learning and Optimal Control Methods for Path Planning

Authors: Qiang Le, Yaguang Yang, Isaac E. Weintraub

Organizations: Department of Electrical and Computer Engineering, Hampton University Hampton, USA · Air Warfare Directorate, Air Force Research Laboratory Wright-Patterson, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Path-planning for autonomous vehicles in threat-laden environments is a fundamental challenge. While traditional optimal control methods can find ideal paths, the computational time is often too slow for real-time decision-making. To solve this challenge, we propose a method based on Deep Deterministic Policy Gradient (DDPG) and model the threat as a simple, circular no-go' zone. A mission failure is claimed if the vehicle enters this no-go' zone at any time or does not reach a neighborhood of the destination. The DDPG agent is trained to learn a direct mapping from its current state (position and velocity) to a series of feasible actions that guide the agent to safely reach its goal. A reward function and two neural networks, critic and actor, are used to describe the environment and guide the control efforts. The DDPG trains the agent to find the largest possible set of starting points (feasible set'') wherein a safe path to the goal is guaranteed. This provides critical information for mission planning, showing beforehand whether a task is achievable from a given starting point, assisting pre-mission planning activities. The approach is validated in simulation. A comparison between the DDPG method and a traditional optimal control (pseudo-spectral) method is carried out. The results show that the learning-based agent may produce effective paths while being significantly faster, making it a better fit for real-time applications. However, there are areas (infeasible set'') where the DDPG agent cannot find paths to the destination, and the paths in the feasible set may not be optimal. These preliminary results guide our future research: (1) improve the reward function to enlarge the DDPG feasible set, (2) examine the feasible set obtained by the pseudo-spectral method, and (3) investigate the arc-search IPM method for the path planning problem.

Figures & tables

Figure 1: (a) Actor network (b) Critic network
Figure 2: Heat map of the reward function where w3w_{3} =0
w1w_{1}w2w_{2}w3w_{3}σ\sigma
30005001.5(40,00010010040,000)\begin{pmatrix}40,000&100\\ 100&40,000\\ \end{pmatrix}
Table 1: Hyperparameters in reward function
Figure 3: (a) Obtained path by DDPG with the standard deviation 5 degrees in the action noise, the resulting travel time 23×0.25=5.7523\times 0.25=5.75 (b) Instant Reward (c) Action over time steps
Figure 4: (a) Obtained path by further trained DDPG agent with low standard deviation in the action noise, the resulting travel time 22×0.25=5.522\times 0.25=5.5 (b) Instant Reward (c) Action over time steps
Figure 5: Feasible area of the specialized agent who is trained using [400,400]⊤[400,400]^{\top} as the starting point
Figure 6: Feasible area of the general agent who is trained using random initial points
Figure 7: Testing examples of the specialized DDPG agent
Figure 8: Testing examples of the general DDPG agent
Figure 9: Obtained path by PS, Elapsed time ≈8\approx 8 seconds (a) constraining speed ∣V∣=200|V|=200 , (b) no speed constraint
More in this topic

Safe Motion Planning

Safe motion planning develops methods for robots and vehicles to reach goals while avoiding collisions and respecting dynamic and uncertainty constraints.

See all 215 papers

Explore similar work

CardsList
Jun 5, 2026

Path Planning Using Deep Deterministic Policy Gradient: A Reinforcement Learning Approach

Path-planning for autonomous vehicles in threat-laden environments is a fundamental challenge because the problem is nonlinear and nonconvex even in simplest scenarios. While traditional optimal control methods can be used to find ideal paths, the computational time is often too slow for real-time decision-making. To solve this challenge, we propose a method based on Deep Deterministic Policy Gradient (DDPG) and model the threat as possibly multiple circular 'no-go' zones. A mission is regarded as a failure if the vehicle enters this restricted zone at any time or does not reach a neighborhood of the destination. The DDPG agent is trained through trial and error in a simulated environment, learning a direct mapping from its current state (position and heading) to a series of feasible actions that guide the agent to safely reach its destination. The reword function has three parts: (a) an attractive field centered at the final destination, (b) some repulsive fields centered at the origins of circular obstacles, and (c) a penalty of control energy consumption (the magnitude of heading change) that indirectly in favor for straight path. The DDPG trains the agent using these incentives to find the largest possible set of starting points wherein a safe path to the destination is guaranteed. This provides critical information for mission planning, showing beforehand whether a task is achievable from a given starting point, assisting pre-mission planning activities. The approach is validated in simulation. A comparison between the DDPG method and a traditional optimal control (pseudo-spectral) method is carried out. The results show that the learning-based agent produces effective paths while being significantly faster, making it a better fit for real-time applications.

Qiang Le, Yaguang Yang, Isaac E. Weintraub

Hampton University, Hampton, VA 23669, USA · Air Force Research Laboratory, Wright-Patterson AFB, OH 45433, USA

Constrained Motion PlanningSafe Motion PlanningAutonomous Driving PlanningRL for Autonomous DrivingDeep Deterministic Policy Gradient
AskPDF
Jul 5, 2026

Integrated Graph Search and Model Predictive Control for Smooth and Efficient Path Planning in Autonomous Vehicles

Path planning is a fundamental component of autonomous vehicles, where achieving safe, comfortable, and dynamically feasible paths while ensuring computational efficiency remains a significant challenge. This paper presents a sequential path planning framework in which a rough path obtained from graph search is explicitly exploited to guide a Model Predictive Control (MPC)-based path refinement. A rough path is first obtained via Dijkstra search on a discretized grid and is then used to construct a spatially varying convex lateral safety corridor that explicitly captures obstacle avoidance constraints, transforming discrete obstacle avoidance decisions into continuous feasibility constraints for optimization. Within this corridor, an MPC problem is formulated to refine the path, enabling efficient optimization while maintaining path smoothness by penalizing the third-order spatial derivative of the lateral offset over a prediction horizon. The proposed algorithm is evaluated in multiple overtaking scenarios on both straight and curved roads, including cases with single and multiple target vehicles, using high-fidelity environment simulations (i.e., CarMaker). Compared with the previous study, which used polynomial fitting and a quadratic programming method, the proposed approach consistently achieves lower lateral acceleration, curvature, and jerk while reducing computational cost by 28.08% on straight roads and 29.52% on curved roads. These results demonstrate that exploiting graph-search structure within an MPC formulation provides an effective balance between path smoothness and computational efficiency for autonomous vehicles in structured driving environments.

Duc-Tien Bui, Ngoc Thinh Nguyen, Hung Duy Nguyen +3

Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria · Drone Engineering, University of Applied Science Kufstein Tirol, Austria · Automation and Control Institute (ACIN), Vienna University of Technology, Austria

Model Predictive ControlSafe Motion PlanningAutonomous Driving Planning
AskPDF
Oct 8, 2026

Real-Time Motion Planning with Dynamic Hazards: Classical vs. Learning-Based Methods

We study real-time motion planning in dynamic hazard fields through a controlled comparison between classical planning and learning-based methods. Rather than introducing a new planner, we construct a unified benchmark in which representative classical and learning-based methods face the same environments, motion constraints, information assumptions, and evaluation metrics. The test environment consists of planar domains populated with rotating sprinkler-like hazards that generate time-varying forbidden regions via sweeping angular sectors. Our results show a clear regime shift. In deterministic environments, classical planners achieve near-perfect success and higher-quality paths, though sometimes at the cost of substantial planning or replanning time. Under stochastic obstacle dynamics, however, online search becomes strongly budget-sensitive: low budgets lead to frequent failure, while high budgets improve success at the cost of latency and longer trajectories. PPO-based policies, trained under the same scenario distribution, consistently outperform in latency, success rate, and path quality in these stochastic regimes. Overall, the results indicate that uncertainty in obstacle evolution, more than partial observability, is the dominant factor determining which planning paradigm is practically effective for the problem at hand.

Eran Iceland, Alexander Tuisov, Oren Gal +2

School of Engineering and Computer Science, The Hebrew University of Jerusalem, Jerusalem, Israel · Faculty of Data and Decision Science, Technion Israeli Institute of Technology, Haifa, Israel · Hatter Department of Marine Technologies, University of Haifa, Haifa, Israel +1

Motion PlanningDynamic Obstacle AvoidanceRobot Policy Learning
AskPDF