Reinforcement learning (RL) and model predictive control (MPC) offer complementary strengths, yet existing MPC-RL frameworks commonly rely on gradient-based optimization and therefore cannot handle the non-differentiable dynamics and objectives common in RL. We propose Soft MPCritic, an MPC-RL framework that learns a soft value function and uses sampling-based planning both for online control and for generating value targets. Specifically, Soft MPCritic instantiates MPC with model predictive path integral control (MPPI) and learns a terminal Q-function through fitted value iteration. Because the value targets are computed by the same planner used for control, the learned value function is aligned with the planner and implicitly extends its effective horizon. To keep target generation tractable, we introduce an amortized warm-start strategy that reuses the open-loop action sequences planned during online control to initialize batched MPPI computations of value targets, reducing computational cost without sacrificing solution quality. For robustness to model error, Soft MPCritic plans over scenarios drawn from an ensemble of dynamics models trained for next-step prediction. On both classical and constrained control tasks, Soft MPCritic learns effective policies with non-differentiable objectives, particularly where finite-horizon MPPI alone falls short. Soft MPCritic thus offers a practical blueprint for synthesizing MPC policies directly from black-box objectives, without differentiable surrogates.
Figures & tables
Fig. 1: soft MPCritic exists entirely in value space, combining soft value functions and model predictive path integral control for autonomous self-improvement. During online control, the value-space planner updates the initial open-loop actions υ to align with the soft minimum of the Q -function. The updated sequence υ+ , along with transition (s,a,ℓ,s′) , is stored in the replay buffer to warm-start the planner’s value targets VMPPI . After a Q -iteration step, the planner is updated with the new Q , and the further-refined solution υ+ is stored for later reuse.
Fig. 2: Warm- vs. cold-starting model predictive path integral control targets with a single dynamics model or an ensemble. Lines represent the median of 10 seeds, shading the interquartile range. The dashed line marks soft actor-critic’s median after 106 time steps.
MPPI target computation iterations (SPS)
Warm
Cold
Ntarget
1 iter.
1 iter.
3 iters.
5 iters.
20
184 ± 2.80
181 ± 4.69
110 ± 1.71
78 ± 0.95
60
124 ± 2.92
125 ± 3.55
61 ± 0.66
40 ± 0.39
100
95 ± 1.92
95 ± 1.25
42 ± 0.76
27 ± 0.21
TABLE I: Mean and standard deviation of environment steps per second (SPS) during online learning for varying model predictive path integral control samples and iterations for computing targets; higher is better.
Fig. 3: Performance of soft MPCritic with a single model or an ensemble of dynamics models, compared with model predictive path integral control using oracle dynamics and soft actor-critic. Lines show the median over 10 seeds with the interquartile range shaded; model predictive path integral control is shown as the median over 105 time steps in closed-loop. The purple dashed line marks soft actor-critic’s median after 2×106 time steps.
Fig. 4: Closed-loop soft MPCritic trajectories from challenging initial conditions, colored by Qϕ(s,a) (lower is better). Insets: trajectory predictions for action sequences randomly sampled via model predictive path integral control from each end state, colored by their importance-weighted cost.
Fig. 5: Sensitivity of soft MPCritic to the horizon (left, 1000 model predictive path integral control samples) and model predictive path integral control samples (right, horizon of 20 ). Lines show the median over 10 seeds, with the interquartile range shaded; the dashed line marks the median for a horizon of 5 after 2×106 time steps.
Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.
Philipp Schoch, Markus Ryll
Autonomous Aerial Systems Lab Technical University of Munich
Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on balance-hard, where Total Average Return (TAR) increases from 98±18 to 387±255, and hurdle, from 199±13 to 466±200; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.
Effective exploration in high-dimensional continuous control remains a central challenge in reinforcement learning. Planning-based methods address this by combining online planning with learned policies and value functions, but their components can become misaligned during training: learned sampling policies may diverge from planner behavior, while planning distributions stored in replay become stale as the model and value function evolve. Reanalysis can refresh these targets, but at substantial computational cost. We propose GEM-MPC, an MPPI-based reinforcement learning method that improves the interaction between planning and learning. GEM-MPC uses MPPI to combine a policy trained to clone the planner with a KL-regularized policy that explores around it, providing complementary exploitation and guided exploration within planning. We further introduce Gated Prior Distillation, which selectively learns from stored planning distributions only when they provide a better target than the current prior, reducing the impact of stale planning data without requiring full reanalysis. Across continuous-control benchmarks, GEM-MPC consistently outperforms existing planning-based baselines under lower computational budgets.
Alvaro Serra-Gomez, Thomas Moerland
Leiden Institute of Advanced Computer Science, Leiden University, 2311 EZ Leiden, The Netherlands