Value Function-Guided MPPI for Hierarchical Planning and Control in Off-Road Autonomous Navigation

Anoushka Alavilli; Raymond Song; Guanya Shi; Jeff Schneider

cs.ROOct 28, 2026

Value Function-Guided MPPI for Hierarchical Planning and Control in Off-Road Autonomous Navigation

Authors: Anoushka Alavilli, Raymond Song, Guanya Shi, Jeff Schneider

Organizations: The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.

Abstract

Autonomous navigation in off-road terrain is commonly achieved using hierarchical control and planning architectures that rely on explicit cost maps and one-way information flow from a high-level planner to a low-level controller, decoupling planning from the true execution cost experienced during control. We propose a hierarchical planning and control framework in which information flows bidirectionally, allowing the low-level controller to guide trajectory selection at the planning level. The highest level of the hierarchy is a Model Predictive Path Integral (MPPI) control-style planner that determines optimal trajectories and passes these waypoints to a pretrained low-level reinforcement learning (RL) controller. Rather than drawing from an explicit cost map, our MPPI planner leverages the value function of the RL controller as an implicit cost representation. In this formulation, MPPI serves as a trajectory exploration mechanism, while the low-level controller's value function guides the search. To account for the distribution shift seen by the value function between training and evaluation, we propose an online policy evaluation process that adapts the value function to unseen driving experiences, and we demonstrate this adaptation in unseen off-road simulation environments. We evaluate our approach in a high-fidelity off-road driving simulator across a range of challenging terrain conditions and compare its performance against a baseline MPPI controller. The results demonstrate that our method achieves safer and more effective navigation to goal locations while eliminating the need for explicit cost maps and enabling tighter coupling between planning and control.

Figures & tables

Fig. 1 : Overview of our value-guided planning framework. We pre-train RL policies to provide both an actor $\pi_{\theta}(s)$ and a value function $V_{\phi}(s)$ . MPPI samples candidate trajectories and evaluates them using a cost that combines goal progress with the negative value estimate, treating $-V_{\phi}(s)$ as a control-aware cost. The selected trajectory is executed by the policy, and the resulting experience can be used to update the value function online. This enables planning to be guided by the expected performance of the controller without requiring an explicit cost map.

Fig. 2 : Comparison between baseline MPPI, our algorithm, and our algorithm with online adaptation. All trajectories were unseen during training. In (A), our method incurs damage midway through the trajectory, while the online adaptation variant avoids this region through online updates alone, without any prior knowledge that this region would cause damage. In (A), (B), and (E), the baseline MPPI algorithm becomes stuck and is unable to complete its planned route. In (C), our algorithm demonstrates its ability to follow existing trails, which provide safer traversal than off-trail terrain. In (D), our algorithm with online adaptation reaches the maximum time limit and does not complete the trajectory.

Fig. 3 : Comparison of pitch and vertical jerk violations across all 64 evaluation trajectories for the MPPI baseline ((A), (B)), our algorithm ((C), (D)), and our algorithm with online adaptation ((E), (F)). Panels (A), (C), and (E) show pitch violations exceeding $0.4$ rad ( $22.9^{\circ}$ ), while (B), (D), and (F) show vertical jerk violations exceeding $10\text{e}^{-2}$ m/ $\text{s}^{3}$ . The violations are colored by their severity and are no more than 0.7 rad. and 2 m/s 3 for pitch and jerk respectively. The MPPI baseline includes an explicit pitch cost term, whereas our method is trained with a jerk penalty in the reward function. When divided into equally-spaced buckets of low, medium, and high violation severity, the MPPI baseline recorded 317, 51, and 19 jerk violations and 84, 0, and 0 pitch violations, respectively; ours recorded 22, 1, and 0 jerk violations and 11, 1, and 0 pitch violations; and ours with online adaptation recorded 31, 0, and 0 jerk violations and 1, 1, and 3 pitch violations.

Task Performance
Metric	MPPI	Ours	Ours Online
Success (sum)	35 (55%)	55 (86%)	52 (81%)
Failure	29 (45%)	9 (14%)	12 (19%)
Stuck	28	7	6
Timeout	1	2	6
Dist. Traveled (m)	5495	9252	$1.29\times 10^{5}$

TABLE I: Comparison of navigation performance and terrain interaction metrics across methods. Each algorithm is evaluated on 64 randomly sampled start–goal pairs, all unseen during training. Episodes terminate after a maximum of 50 seconds (250 control steps) and are marked as failures if the goal is not reached within this budget (denoted “timeout”). Other episodes terminate with the vehicle stuck (on a steep slope, for example), and these failures are denoted “stuck.” Reported values reflect trajectory statistics aggregated over all trials; lower values in Z-axis motion and orientation indicate smoother traversal. Note: p10 and p90 represent the 10th and 90th percentile of values, respectively.

Fig. 4 : (A) Visualization of training data coverage and evaluation start–goal waypoints. (B) Trajectories traversed by the online adaptation variant of our algorithm, colored by the incurred reward as defined in Eq. 1 . (C) and (D) show value estimates of the traversed terrain during execution for our algorithm without and with online adaptation, respectively. These value maps can be interpreted as implicit cost maps over the visited regions. After online adaptation, regions associated with lower reward decrease in value (i.e., appear lighter), reflecting updated terrain estimates. Despite this, the algorithm continues to favor on-trail paths, as they are predicted to have higher value than off-trail terrain. All colorbars are scaled between the 10th and 90th percentiles of the plotted values.

PPO Policy Training
Hyperparameter	Value
Training data coverage (m)	$3.5\times 10^{3}$
Waypoint spacing (m)	4
Waypoint acceptance thresh. (m)	3
Reward weight $\beta_{1}$	1000
Reward weight $\beta_{2}$	-1000

TABLE II: Hyperparameters used in all implementations. PPO policy training is performed using the standard Stable-Baselines3 implementation [ 6 ] .

Off-Road Robot Navigation

Off-road robot navigation studies how ground robots perceive terrain, assess traversability, and plan safe routes through unstructured environments.

See all 54 papers

HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization
Jun 15, 2026Youngjae Min, Jovin D'sa, Faizan M. Tariq +3Hierarchical RL Motion Planning
1Massachusetts Institute of Technology, Cambridge, MA 02139. · 2Honda Research Institute, USA, San Jose, CA, 95134.
TADPO: Reinforcement Learning Goes Off-road
Mar 6, 2026Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3End-to-End Autonomous Driving Autonomous Driving
Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201
High-Speed, All-Terrain Autonomy: Ensuring Safety at the Limits of Mobility
Mar 20, 2026James R. Baxter, Bogdan I. Epureanu, Paramsothy Jayakumar +1Safe Motion Planning Off-Road Robot Navigation
Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI · U.S. Army Ground Vehicle Systems Center, Warren, MI

Jun 15, 2026J/K move · Enter open · S save

HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization

Youngjae Min, Jovin D'sa, Faizan M. Tariq, David Isele+2

1Massachusetts Institute of Technology, Cambridge, MA 02139. · 2Honda Research Institute, USA, San Jose, CA, 95134.

Hierarchical RL Motion Planning MPPI Control Autonomous Driving Planning Robot Policy Generalization Multi-Objective Motion Planning

Open paper AskPDF arXiv

Robots deployed in the real world must plan motions across diverse scenarios without per-scenario retuning. End-to-end reinforcement learning (RL) can generalize across scenarios but often becomes brittle under distribution shift, reward misspecification, and stochastic interactions. Model predictive path integral (MPPI) control enables strong real-time refinement without gradients, but its performance depends on a well-shaped sampling prior, while manually designing the priors does not scale to multi-scenario deployment. We present HOLO-MPPI (High-level Offline, Low-level Online MPPI), a multi-scenario motion planning framework that combines high-level policy learning with low-level stochastic optimal control. Offline, we learn a high-level policy that proposes scenario-robust plans in an abstract action space, with a learned world model for online rollout. Online, the policy serves as a data-driven prior generator that parameterizes MPPI's sampling distribution conditioned on the current observation and goal. MPPI then optimizes low-level control sequences around this prior in real time to adapt to local disturbances. We instantiate HOLO-MPPI in autonomous driving by designing an effective high-level action space and tailored model architectures. Our evaluation across diverse driving scenarios shows that HOLO-MPPI improves upon MPPI and end-to-end RL baselines while maintaining real-time control.

Abstract

Off-Road Robot Navigation

Explore similar work

HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization

TADPO: Reinforcement Learning Goes Off-road

High-Speed, All-Terrain Autonomy: Ensuring Safety at the Limits of Mobility