Maximum Entropy RL

Maximum entropy reinforcement learning adds a preference for stochastic, high-entropy actions to reward maximization, yielding soft value functions and Boltzmann-like policies that can improve exploration and retain multiple useful behaviors. Common approaches include soft actor-critic, entropy-regularized policy gradients and Q-learning, plus temperature schedules and state-, task-, or token-level entropy control. Current work extends these methods to expressive diffusion, flow, and mixture policies, offline and multi-agent learning, and continuous-time control, while theory examines convergence and robustness. A central tension is preserving enough entropy to discover diverse high-value solutions without weakening reward optimization, destabilizing critics, or making policy densities and training too costly.