cs.ROOct 28, 2026

Value Function-Guided MPPI for Hierarchical Planning and Control in Off-Road Autonomous Navigation

Authors: Anoushka Alavilli, Raymond Song, Guanya Shi, Jeff Schneider

Organizations: The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous navigation in off-road terrain is commonly achieved using hierarchical control and planning architectures that rely on explicit cost maps and one-way information flow from a high-level planner to a low-level controller, decoupling planning from the true execution cost experienced during control. We propose a hierarchical planning and control framework in which information flows bidirectionally, allowing the low-level controller to guide trajectory selection at the planning level. The highest level of the hierarchy is a Model Predictive Path Integral (MPPI) control-style planner that determines optimal trajectories and passes these waypoints to a pretrained low-level reinforcement learning (RL) controller. Rather than drawing from an explicit cost map, our MPPI planner leverages the value function of the RL controller as an implicit cost representation. In this formulation, MPPI serves as a trajectory exploration mechanism, while the low-level controller's value function guides the search. To account for the distribution shift seen by the value function between training and evaluation, we propose an online policy evaluation process that adapts the value function to unseen driving experiences, and we demonstrate this adaptation in unseen off-road simulation environments. We evaluate our approach in a high-fidelity off-road driving simulator across a range of challenging terrain conditions and compare its performance against a baseline MPPI controller. The results demonstrate that our method achieves safer and more effective navigation to goal locations while eliminating the need for explicit cost maps and enabling tighter coupling between planning and control.

Figures & tables

Fig. 1 : Overview of our value-guided planning framework. We pre-train RL policies to provide both an actor πθ(s)\pi_{\theta}(s) and a value function Vϕ(s)V_{\phi}(s) . MPPI samples candidate trajectories and evaluates them using a cost that combines goal progress with the negative value estimate, treating −Vϕ(s)-V_{\phi}(s) as a control-aware cost. The selected trajectory is executed by the policy, and the resulting experience can be used to update the value function online. This enables planning to be guided by the expected performance of the controller without requiring an explicit cost map.
Fig. 2 : Comparison between baseline MPPI, our algorithm, and our algorithm with online adaptation. All trajectories were unseen during training. In (A), our method incurs damage midway through the trajectory, while the online adaptation variant avoids this region through online updates alone, without any prior knowledge that this region would cause damage. In (A), (B), and (E), the baseline MPPI algorithm becomes stuck and is unable to complete its planned route. In (C), our algorithm demonstrates its ability to follow existing trails, which provide safer traversal than off-trail terrain. In (D), our algorithm with online adaptation reaches the maximum time limit and does not complete the trajectory.
Fig. 3 : Comparison of pitch and vertical jerk violations across all 64 evaluation trajectories for the MPPI baseline ((A), (B)), our algorithm ((C), (D)), and our algorithm with online adaptation ((E), (F)). Panels (A), (C), and (E) show pitch violations exceeding 0.40.4 rad ( 22.9∘22.9^{\circ} ), while (B), (D), and (F) show vertical jerk violations exceeding 10e−210\text{e}^{-2} m/ s3\text{s}^{3} . The violations are colored by their severity and are no more than 0.7 rad. and 2 m/s 3 for pitch and jerk respectively. The MPPI baseline includes an explicit pitch cost term, whereas our method is trained with a jerk penalty in the reward function. When divided into equally-spaced buckets of low, medium, and high violation severity, the MPPI baseline recorded 317, 51, and 19 jerk violations and 84, 0, and 0 pitch violations, respectively; ours recorded 22, 1, and 0 jerk violations and 11, 1, and 0 pitch violations; and ours with online adaptation recorded 31, 0, and 0 jerk violations and 1, 1, and 3 pitch violations.
MetricMPPIOursOurs Online
Task Performance
Success (sum)35 (55%)55 (86%)52 (81%)
Failure29 (45%)9 (14%)12 (19%)
Stuck2876
Timeout126
Dist. Traveled (m)549592521.29×1051.29\times 10^{5}
TABLE I: Comparison of navigation performance and terrain interaction metrics across methods. Each algorithm is evaluated on 64 randomly sampled start–goal pairs, all unseen during training. Episodes terminate after a maximum of 50 seconds (250 control steps) and are marked as failures if the goal is not reached within this budget (denoted “timeout”). Other episodes terminate with the vehicle stuck (on a steep slope, for example), and these failures are denoted “stuck.” Reported values reflect trajectory statistics aggregated over all trials; lower values in Z-axis motion and orientation indicate smoother traversal. Note: p10 and p90 represent the 10th and 90th percentile of values, respectively.
Fig. 4 : (A) Visualization of training data coverage and evaluation start–goal waypoints. (B) Trajectories traversed by the online adaptation variant of our algorithm, colored by the incurred reward as defined in Eq. 1 . (C) and (D) show value estimates of the traversed terrain during execution for our algorithm without and with online adaptation, respectively. These value maps can be interpreted as implicit cost maps over the visited regions. After online adaptation, regions associated with lower reward decrease in value (i.e., appear lighter), reflecting updated terrain estimates. Despite this, the algorithm continues to favor on-trail paths, as they are predicted to have higher value than off-trail terrain. All colorbars are scaled between the 10th and 90th percentiles of the plotted values.
HyperparameterValue
PPO Policy Training
Training data coverage (m)3.5×1033.5\times 10^{3}
Waypoint spacing (m)4
Waypoint acceptance thresh. (m)3
Reward weight β1\beta_{1}1000
Reward weight β2\beta_{2}-1000
TABLE II: Hyperparameters used in all implementations. PPO policy training is performed using the standard Stable-Baselines3 implementation [ 6 ] .
More in this topic

Off-Road Robot Navigation

Off-road robot navigation studies how ground robots perceive terrain, assess traversability, and plan safe routes through unstructured environments.

See all 54 papers

Explore similar work

CardsList
Jun 15, 2026

HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization

Robots deployed in the real world must plan motions across diverse scenarios without per-scenario retuning. End-to-end reinforcement learning (RL) can generalize across scenarios but often becomes brittle under distribution shift, reward misspecification, and stochastic interactions. Model predictive path integral (MPPI) control enables strong real-time refinement without gradients, but its performance depends on a well-shaped sampling prior, while manually designing the priors does not scale to multi-scenario deployment. We present HOLO-MPPI (High-level Offline, Low-level Online MPPI), a multi-scenario motion planning framework that combines high-level policy learning with low-level stochastic optimal control. Offline, we learn a high-level policy that proposes scenario-robust plans in an abstract action space, with a learned world model for online rollout. Online, the policy serves as a data-driven prior generator that parameterizes MPPI's sampling distribution conditioned on the current observation and goal. MPPI then optimizes low-level control sequences around this prior in real time to adapt to local disturbances. We instantiate HOLO-MPPI in autonomous driving by designing an effective high-level action space and tailored model architectures. Our evaluation across diverse driving scenarios shows that HOLO-MPPI improves upon MPPI and end-to-end RL baselines while maintaining real-time control.

Youngjae Min, Jovin D'sa, Faizan M. Tariq +3

1Massachusetts Institute of Technology, Cambridge, MA 02139. · 2Honda Research Institute, USA, San Jose, CA, 95134.

Hierarchical RLMotion PlanningMPPI ControlAutonomous Driving PlanningRobot Policy GeneralizationMulti-Objective Motion Planning
AskPDF
Mar 6, 2026

TADPO: Reinforcement Learning Goes Off-road

Off-road autonomous driving poses significant challenges such as navigating unmapped, variable terrain with uncertain and diverse dynamics. Addressing these challenges requires effective long-horizon planning and adaptable control. Reinforcement Learning (RL) offers a promising solution by learning control policies directly from interaction. However, because off-road driving is a long-horizon task with low-signal rewards, standard RL methods are challenging to apply in this setting. We introduce TADPO, a novel policy gradient formulation that extends Proximal Policy Optimization (PPO), leveraging off-policy trajectories for teacher guidance and on-policy trajectories for student exploration. Building on this, we develop a vision-based, end-to-end RL system for high-speed off-road driving, capable of navigating extreme slopes and obstacle-rich terrain. We demonstrate our performance in simulation and, importantly, zero-shot sim-to-real transfer on a full-scale off-road vehicle. To our knowledge, this work represents the first deployment of RL-based policies on a full-scale off-road platform.

Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3

Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201

End-to-End Autonomous DrivingAutonomous DrivingProximal Policy OptimizationRL for Autonomous DrivingOff-Policy RL
AskPDF
Mar 20, 2026

High-Speed, All-Terrain Autonomy: Ensuring Safety at the Limits of Mobility

A novel local trajectory planner, capable of controlling an autonomous off-road vehicle on rugged terrain at high-speed is presented. Autonomous vehicles are currently unable to safely operate off-road at high-speed, as current approaches either fail to predict and mitigate rollovers induced by rough terrain or are not real-time feasible. To address this challenge, a novel model predictive control (MPC) formulation is developed for local trajectory planning. A new dynamics model for off-road vehicles on rough, non-planar terrain is derived and used for prediction. Extreme mobility, including tire liftoff without rollover, is safely enabled through a new energy-based constraint. The formulation is analytically shown to mitigate rollover types ignored by many state-of-the-art methods, and real-time feasibility is achieved through parallelized GPGPU computation. The planner's ability to provide safe, extreme trajectories is studied through both simulated trials and full-scale physical experiments. The results demonstrate fewer rollovers and more successes compared to a state-of-the-art baseline across several challenging scenarios that push the vehicle to its mobility limits.

James R. Baxter, Bogdan I. Epureanu, Paramsothy Jayakumar +1

Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI · U.S. Army Ground Vehicle Systems Center, Warren, MI

Safe Motion PlanningOff-Road Robot NavigationModel Predictive ControlVehicle Dynamics
AskPDF