Maximum-Entropy Exploration with Future State-Action Visitation Measures

Adrien Bolland; Gaspard Lambrechts; Damien Ernst

cs.LGMar 19, 2026

Maximum-Entropy Exploration with Future State-Action Visitation Measures

Authors: Adrien Bolland, Gaspard Lambrechts, Damien Ernst

Organizations: University of Liège

Abstract

Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by providing additional intrinsic rewards proportional to that entropy function. In this paper, we study intrinsic rewards proportional to the entropy of the discounted distribution of state-action features visited during future time steps. This approach is motivated by two results. First, we show that the expected sum of these intrinsic rewards is a lower bound on the entropy of the discounted distribution of state-action features visited in trajectories starting from the initial states, which we relate to an alternative maximum entropy objective. Second, we show that the distribution used in the intrinsic reward definition is the fixed point of a contraction operator and can therefore be estimated off-policy. Experiments highlight that the new objective often leads to improved visitation of features within individual trajectories, in exchange for slightly reduced visitation of features in expectation over different trajectories, as suggested by the lower bound. It also leads to improved convergence speed for learning exploration-only agents. Control performance remains similar across most methods on the considered benchmarks.

Figures & tables

Figure 1: Learning results for two representative environments, selected from Appendix E . The first two columns represent the evolution of the two exploration metrics defined in Section 5 , when learning exploration policies. The third column represents the evolution of the expected return when learning MaxEntRL control policies.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Convolutional layer	$2$
Neurons for each convolutional layer	$32$
State-feature size	$256$
Forward layers policy	$2$
Forward layers critic	$2$
Neurons forward layers	$256$

Appendix

Table 1: Hyperparameters

Figure 2: Evolution of the entropy of the discounted visitation probability measure of the position of the agent on the grid when computing exploration policies (i.e., when neglecting the rewards of the MDP). The entropy is computed empirically with Monte Carlo simulations. For each iteration, the interquartile mean over 5 runs is reported, along with its $95\%$ confidence interval.

Figure 3: Evolution of the entropy of the discounted visitation probability measure of the position of the agent on the grid within single trajectories, averaged over trajectories, when computing exploration policies (i.e., when neglecting the rewards of the MDP). The entropy is computed empirically with Monte Carlo simulations. For each iteration, the interquartile mean over 5 runs is reported, along with its $95\%$ confidence interval.

Figure 4: Expected return during the policy optimization. The expectation is computed empirically with Monte Carlo simulations. For each iteration, the interquartile mean over 5 runs is reported, along with its $95\%$ confidence interval.

Explore similar work

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
Dec 5, 2024Hongming Li, Zhao Yang, Xiaoxuan Liang +2Intrinsic Reward Methods Reinforcement Learning
Zhejiang Lab, Huangzhou, China · Vrije Universiteit Amsterdam, Amsterdam, the Netherlands · Computational NeuroEngineering Laboratory at the University of Florida, Gainesville, USA
Maximum Entropy Exploration Without the Rollouts
Mar 12, 2026Jacob Adamczyk, Adam Kamoski, Rahul V. KulkarniRL Exploration Maximum Entropy RL
Department of Physics, University of Massachusetts Boston · NSF Institute for Artificial Intelligence and Fundamental Interactions
When Do Intrinsic Rewards Lead to Exploration?
Oct 1, 2026Scott W. Viteri, Laura Gomezjurado Gonzalez, Clark BarrettIntrinsic Reward Methods RL Exploration
Stanford University

Dec 5, 2024J/K move · Enter open · S save

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

Hongming Li, Zhao Yang, Xiaoxuan Liang, Shujian Yu+1

Zhejiang Lab, Huangzhou, China · Vrije Universiteit Amsterdam, Amsterdam, the Netherlands · Computational NeuroEngineering Laboratory at the University of Florida, Gainesville, USA

Intrinsic Reward Methods Reinforcement Learning Intrinsic Motivation RL Exploration Maximum Entropy RL Reward-Free RL

Open paper AskPDF arXiv

Reinforcement learning agents depend on reward signals whose density is rarely under the designer's control, and when such signals are absent, an agent must generate its own drive to explore. State entropy maximization offers a principled objective for this, but existing methods break down at scale in two ways: the intrinsic reward vanishes once a state has been visited, discouraging revisits to the very gateways that lead onward, and estimating entropy over millions of accumulated observations becomes computationally prohibitive. We address both with Episodic and Lifelong Exploration via Maximum Entropy (ELEMENT), a multiscale intrinsically motivated framework for reward-free exploration that transfers to downstream tasks. ELEMENT couples lifelong entropy maximization with a complementary episodic term acting on a faster timescale. For the episodic term, we derive average episodic state entropy, an intrinsic reward that is the exact minimizer of a tractable upper bound on the reward-decomposition objective; for the lifelong term, we propose a

k

NN graph-based estimator that keeps entropy tractable without forgetting. ELEMENT consistently outperforms state-of-the-art intrinsic reward baselines on state coverage and unsupervised pre-training. Videos, code, and supplementary material: https://sites.google.com/view/element-rl.

Abstract

Explore similar work

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

Maximum Entropy Exploration Without the Rollouts

When Do Intrinsic Rewards Lead to Exploration?