Maximum Entropy RL

RL: Reinforcement Learning

Maximum entropy reinforcement learning adds a preference for stochastic, high-entropy actions to reward maximization, yielding soft value functions and Boltzmann-like policies that can improve exploration and retain multiple useful behaviors. Common approaches include soft actor-critic, entropy-regularized policy gradients and Q-learning, plus temperature schedules and state-, task-, or token-level entropy control. Current work extends these methods to expressive diffusion, flow, and mixture policies, offline and multi-agent learning, and continuous-time control, while theory examines convergence and robustness. A central tension is preserving enough entropy to discover diverse high-value solutions without weakening reward optimization, destabilizing critics, or making policy densities and training too costly.

Latest papers

69 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning
Oct 23, 2026Hao Cheng, Yuling Jiao, Lican Kang +2Offline RL Maximum Entropy RL
School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. +1
DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning
Oct 12, 2026Ruslan Ageev, Huaze Tang, Runqing Miao +2Flow-Based Policy Learning Reinforcement Learning
Tsinghua University · Peng Cheng Laboratory
Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning
Oct 11, 2026Chunyu Luo, Hongyu Zhao, Guangyi Ni +2Reinforcement Learning Maximum Entropy RL
The University of Hong Kong · Beihang University · Harbin Institute of Technology, Shenzhen +2
Reinforcement Learning via Fixed-point Flow Matching
Oct 9, 2026Denis Blessing, Onur Celik, Gerhard NeumannFlow-Based Policy Learning Reinforcement Learning
Karlsruhe Institute of Technology (KIT) Autonomous Learning Robots Lab (ALR)
LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL
Oct 6, 2026Songyuan Zhang, Oswin So, Eric Yang Yu +3Reinforcement Learning Offline RL
MIT · MIT Lincoln Laboratory
Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems
Oct 3, 2026Qi Feng, Gu WangStochastic Optimal Control Policy Iteration
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
FERPO: Forward Entropy-Regularized Policy Optimization
Oct 1, 2026Sebastian Sanokowski, Alireza Sarmadi, Majid KhadivKL-Regularized RL Policy Optimization
Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich
An analysis of Mirror-Descent Soft Actor-Critic
Sep 28, 2026Denis Zorba, Michal ValkoReinforcement Learning Online Mirror Descent
School of Mathematics, University of Edinburgh, United Kingdom · INRIA, Paris, France
Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
Sep 27, 2026Woongyeong Yeo, Minki Kang, Chanuk Lee +3RL for Language Model Reasoning Credit Assignment in RL
KAIST
RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Sep 23, 2026Noa Rubin, Zohar RingelReinforcement Learning Maximum Entropy RL
Racah Institute of Physics The Hebrew University of Jerusalem
Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
Sep 14, 2026Jin Ma, Gaozhan Wang, Jianfeng Zhang +1Reinforcement Learning Stochastic Optimal Control
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027
SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
Sep 11, 2026Xiefeng Wu, Shu Zhang, Zhaojie Chu +1Offline-To-Online RL Conservative RL
School of Computer Science, Wuhan University, Wuhan, China · School of Electronic Information, Wuhan University, Wuhan, China · School of Internet, Anhui University, Hefei, China
Marginal Coverage Credit Reduces Redundant Exploration in Parallel State-Entropy Optimization
Aug 27, 2026Junhao Cao, Hongyi Xia, Jianian Wu +3RL Exploration Credit Assignment in RL
College of Information Engineering, Hunan Applied Technology University, Changde, 415100, Hunan, China
Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO
Aug 25, 2026Yiyang He, Zhichun Zhou, Ziwei Wang +2Policy Optimization Proximal Policy Optimization
Department of Engineering, Beihang University, Beijing, China · Department of Automation, Tsinghua University, Beijing, China
Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control
Aug 7, 2026Zhaoyu Zhu, Rui Gao, Shuang LiMaximum Entropy RL Policy Gradient Methods
University of Illinois Urbana-Champaign · The University of Texas at Austin · The Chinese University of Hong Kong, Shenzhen
Distributional Soft Bellman Operator under the Cramér Geometry
Jul 20, 2026Keru Wang, Yixin Deng, Yao Lyu +2Distributional RL Maximum Entropy RL
School of Electrical and Electronic Engineering, University College Dublin, Dublin, Ireland · School of Vehicle and Mobility, Tsinghua University, Beijing, China · College of Artificial Intelligence, Tsinghua University, Beijing, China
Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies
Jul 19, 2026Silviu PitisMarkov Decision Processes Maximum Entropy RL
University of Michigan
Group Entropy-Controlled Policy Optimization
Jul 18, 2026Guangran Cheng, Chengqi Lyu, Songyang Gao +2RL for Language Models Group Relative Policy Optimization
Shanghai AI Laboratory
Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
Jul 8, 2026Zetian Hu, Shunyu Liu, Junjie Zhang +4Agentic RL Group Relative Policy Optimization
Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798 · Tongyi Lab, Alibaba Group
Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
Jul 3, 2026Jialun Cao, Fernando Acero, David Šiška +1Robust Markov Decision Processes Maximum Entropy RL
School of Mathematics, University of Edinburgh, United Kingdom. · J.P. Morgan AI Research, United Kingdom. · Department of Mathematics, Imperial College London, United Kingdom.
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models
Jun 27, 2026Congde Hu, Danping Li, Lin Xu +1Stackelberg Games Maximum Entropy RL
School of Mathematics and Statistics, Anhui Normal University, Wuhu, Anhui 241002, China · School of Statistics, Key Laboratory of Advanced Theory and Application in Statistics and Data Science-MOE, Faculty of Economics and Management, East China Normal University, Shanghai, China · School of Mathematics, Southeast University, Nanjing, Jiangsu 211189, China
Entropy Regularized Reinforcement Learning for Zero-Sum Stochastic Differential Games in a Regime-Switching Jump-Diffusion Process
Jun 27, 2026Congde Hu, Zhuo Jin, Danping Li +1Zero-Sum Games Maximum Entropy RL
School of Mathematics and Statistics, Anhui Normal University, Wuhu, 241002, Anhui, China. · Department of Actuarial Studies and Business Analytics, Macquarie University, Sydney, 2109, NSW, Australia. · School of Statistics, KLATASDS-MOE, China Inclusive Ageing Finance Research Center, East China Normal University, Shanghai, 200062, Shanghai, China.
Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
Jun 21, 2026Serge Thilges, Onur Celik, Denis Blessing +2Diffusion Policy Maximum Entropy RL
Autonomous Learning Robots, Karlsruhe Institute of Technology
STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
Jun 17, 2026Haipeng Luo, Qingfeng Sun, Songli Wu +4RL for Language Model Reasoning Maximum Entropy RL
Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan
Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward
Jun 15, 2026Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı +1Reinforcement Learning Average-Reward RL
Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey
Mitigating Bias in Low-SNR Financial Reinforcement Learning via Quantum Representations
Jun 9, 2026Zeyu Liu, Xuanzhi Feng, Sing Kwong Lai +6Quantitative Finance Quantum Machine Learning
The Hong Kong University of Science and Technology
PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
Jun 7, 2026Shumeng Yang, Yisu Liu, Jiayi Zheng +2RL for Language Model Reasoning Maximum Entropy RL
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Artificial Intelligence, Beijing University of Posts and Telecommunications +2
Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
Jun 3, 2026Chirag Chawla, Rohan Charudatt Salvi, Madhav S. BaidyaReinforcement Learning RL for Language Model Reasoning
Indian Institute of Technology (BHU), Varanasi, India · Department of Computer Science, University of Illinois Chicago, IL 60607, USA
FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance
May 29, 2026Sungha Kim, Gawon Lee, Jusuk Lee +3RL Control Maximum Entropy RL
Seoul National University · Georgia Institute of Technology

Oct 23, 2026J/K move · Enter open · S save

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Hao Cheng, Yuling Jiao, Lican Kang, Jerry Zhijian Yang+1

School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China. · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China.+1

Offline RL Maximum Entropy RL

Open paper AskPDF arXiv

Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function

Q^*

from a fixed dataset without further interaction with the environment. Although

Q^*

is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating

Q^*

requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate

Q^*

. We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that

Q^*

inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index

β

, the value learning stage achieves an excess Bellman residual risk bound of order

\widetilde{\mathcal O}\big(n^{-\frac{2β}{d+2β}}\big),

where

d

is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.