Planning in entropy-regularized Markov decision processes and games

Jean-Bastien Grill; Omar Darwiche Domingues; Pierre Ménard; Rémi Munos; Michal Valko

cs.LGApr 21, 2026

Planning in entropy-regularized Markov decision processes and games

Authors: Jean-Bastien Grill, Omar Darwiche Domingues, Pierre Ménard, Rémi Munos, Michal Valko

Organizations: DeepMind Paris · SequeL team, Inria Lille

Abstract

We propose SmoothCruiser, a new planning algorithm for estimating the value function in entropy-regularized Markov decision processes and two-player games, given a generative model of the environment. SmoothCruiser makes use of the smoothness of the Bellman operator promoted by the regularization to achieve problem-independent sample complexity of order O~(1/epsilon^4) for a desired accuracy epsilon, whereas for non-regularized settings there are no known algorithms with guaranteed polynomial sample complexity in the worst case.

Explore similar work

Refined Analysis of Entropy-Regularized Actor-Critic
May 23, 2026Safwan Labbi, Paul Mangold, Daniil Tiapkin +1Soft Actor-Critic Entropy-Regularized Equilibrium Selection
Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning
Apr 16, 2026Jean-Bastien Grill, Michal Valko, Rémi MunosMarkov Decision Processes Classical Planning
Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
Jul 3, 2026Jialun Cao, Fernando Acero, David Šiška +1Entropy Regularized Reinforcement Learning Kullback-Leibler Regularization

May 23, 2026 · cs.LGJ/K move · Enter open · S save

Refined Analysis of Entropy-Regularized Actor-Critic

Safwan Labbi, Paul Mangold, Daniil Tiapkin, Eric Moulines

In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an

ε

-optimal regularized value with

\tilde{O}(\log(1/ε))

samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.

Soft Actor-Critic Entropy-Regularized Equilibrium Selection Critic Learning Variance Reduction Optimal Actions Refinement

Open paper Ask about it PDF

Abstract

Explore similar work

Refined Analysis of Entropy-Regularized Actor-Critic

Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning

Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning