Reinforcement learning (RL) and model predictive control (MPC) offer complementary strengths, yet existing MPC-RL frameworks commonly rely on gradient-based optimization and therefore cannot handle the non-differentiable dynamics and objectives common in RL. We propose Soft MPCritic, an MPC-RL framework that learns a soft value function and uses sampling-based planning both for online control and for generating value targets. Specifically, Soft MPCritic instantiates MPC with model predictive path integral control (MPPI) and learns a terminal Q-function through fitted value iteration. Because the value targets are computed by the same planner used for control, the learned value function is aligned with the planner and implicitly extends its effective horizon. To keep target generation tractable, we introduce an amortized warm-start strategy that reuses the open-loop action sequences planned during online control to initialize batched MPPI computations of value targets, reducing computational cost without sacrificing solution quality. For robustness to model error, Soft MPCritic plans over scenarios drawn from an ensemble of dynamics models trained for next-step prediction. On both classical and constrained control tasks, Soft MPCritic learns effective policies with non-differentiable objectives, particularly where finite-horizon MPPI alone falls short. Soft MPCritic thus offers a practical blueprint for synthesizing MPC policies directly from black-box objectives, without differentiable surrogates.
Figures & tables
Fig. 1: soft MPCritic exists entirely in value space, combining soft value functions and model predictive path integral control for autonomous self-improvement. During online control, the value-space planner updates the initial open-loop actions υ to align with the soft minimum of the Q -function. The updated sequence υ+ , along with transition (s,a,ℓ,s′) , is stored in the replay buffer to warm-start the planner’s value targets VMPPI . After a Q -iteration step, the planner is updated with the new Q , and the further-refined solution υ+ is stored for later reuse.
Fig. 2: Warm- vs. cold-starting model predictive path integral control targets with a single dynamics model or an ensemble. Lines represent the median of 10 seeds, shading the interquartile range. The dashed line marks soft actor-critic’s median after 106 time steps.
MPPI target computation iterations (SPS)
Warm
Cold
Ntarget
1 iter.
1 iter.
3 iters.
5 iters.
20
184 ± 2.80
181 ± 4.69
110 ± 1.71
78 ± 0.95
60
124 ± 2.92
125 ± 3.55
61 ± 0.66
40 ± 0.39
100
95 ± 1.92
95 ± 1.25
42 ± 0.76
27 ± 0.21
TABLE I: Mean and standard deviation of environment steps per second (SPS) during online learning for varying model predictive path integral control samples and iterations for computing targets; higher is better.
Fig. 3: Performance of soft MPCritic with a single model or an ensemble of dynamics models, compared with model predictive path integral control using oracle dynamics and soft actor-critic. Lines show the median over 10 seeds with the interquartile range shaded; model predictive path integral control is shown as the median over 105 time steps in closed-loop. The purple dashed line marks soft actor-critic’s median after 2×106 time steps.
Fig. 4: Closed-loop soft MPCritic trajectories from challenging initial conditions, colored by Qϕ(s,a) (lower is better). Insets: trajectory predictions for action sequences randomly sampled via model predictive path integral control from each end state, colored by their importance-weighted cost.
Fig. 5: Sensitivity of soft MPCritic to the horizon (left, 1000 model predictive path integral control samples) and model predictive path integral control samples (right, horizon of 20 ). Lines show the median over 10 seeds, with the interquartile range shaded; the dashed line marks the median for a horizon of 5 after 2×106 time steps.