cs.AIFeb 21, 2026

Task-Aware Exploration via a Predictive Bisimulation Metric

Authors: Dayang Liang, Ruihan Liu, Lipeng Wan, Yunlong Liu, Bo An

Organizations: Department of Automation, Xiamen University, Xiamen, China · Department of Computer Science, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · School of Artificial Intelligence, Xian Jiaotong University, Xian, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accelerating exploration in visual reinforcement learning under sparse rewards remains challenging due to the substantial task-irrelevant variations. Despite advances in intrinsic exploration, many methods either assume access to low-dimensional states or lack task-aware exploration strategies, thereby rendering them fragile in visual domains. To bridge this gap, we present TEB, a Task-aware Exploration approach that tightly couples task-relevant representations with exploration through a predictive Bisimulation metric. Specifically, TEB leverages the metric not only to learn behaviorally grounded task representations but also to measure behaviorally intrinsic novelty over the learned latent space. To realize this, we first theoretically mitigate the representation collapse of degenerate bisimulation metrics under sparse rewards by internally introducing a simple but effective predicted reward differential. Building on this robust metric, we design potential-based exploration bonuses, which measure the relative novelty of adjacent observations over the latent space. Extensive experiments on MetaWorld and Maze2D show that TEB achieves superior exploration ability and outperforms recent baselines.

Figures & tables

Figure 1: An illustration of existing explorations in noise space and our task-aware exploration. In complex tasks, existing explorations remain limited by task-irrelevant elements, resulting in risky transitions like the “yellow line”. In contrast, the task-relevant space and exploration bonus built on the bisimulation metric can help visual RL complete tasks faster. The right figure specifically illustrates how to construct an exploration bonus that measures task-relevant novelty between a state relative to a global anchor state using the unified metric in a rigorous task space.
Figure 2: Success rates of TEB and baselines in the MetaWorld environment. Each experiment runs with three random seeds, with shaded region representing the standard deviation across seeds.
Figure 3: Visualization of state coverage in representative Maze2D tasks.
DomainsSquare-aSquare-bSquare-cSquare-dCorridor2Square-treeSquare-bottleneck
Disagreement0.38 ±\pm 0.080.38 ±\pm 0.200.39 ±\pm 0.190.43 ±\pm 0.140.48 ±\pm 0.100.32 ±\pm 0.100.34 ±\pm 0.07
ICM0.54 ±\pm 0.080.57 ±\pm 0.140.46 ±\pm 0.060.59 ±\pm 0.050.75 ±\pm 0.070.41 ±\pm 0.040.33 ±\pm 0.06
LBS0.30 ±\pm 0.040.27 ±\pm 0.050.25 ±\pm 0.020.33 ±\pm 0.030.44 ±\pm 0.070.23 ±\pm 0.040.21 ±\pm 0.04
Proto0.40 ±\pm 0.040.40 ±\pm 0.060.38 ±\pm 0.090.48 ±\pm 0.040.71 ±\pm 0.040.24 ±\pm 0.020.23 ±\pm 0.01
RND0.42 ±\pm 0.100.60 ±\pm 0.130.39 ±\pm 0.120.37 ±\pm 0.040.63 ±\pm 0.100.28 ±\pm 0.090.32 ±\pm 0.09
BeCL0.52 ±\pm 0.050.48 ±\pm 0.120.43 ±\pm 0.090.47 ±\pm 0.050.67 ±\pm 0.130.37 ±\pm 0.070.30 ±\pm 0.05
Table 1: State coverage ratios in the Maze environment. We report the mean and std of 10 seeds for each algorithm.
Figure 4: Ablation Studies. Left : the three figures show the ablation curves of TEB components across three tasks; Right : the fourth figure illustrates the ablation results regarding anchor state selection for intrinsic rewards across the three tasks. Each task is run for 1M steps with three random seeds. Note that for better performance comparison, the anchor ablation experiment (right) exhibits the episode success rate at 50% of the training steps.
Figure 5: Performance analysis of the bisimulation metric with different reward signals.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example vision tasks in the MetaWorld environment.
MethodsPick-out-of-holePick-placePush-backBox-closeStick-pullCoffee-push
ICM0.761 ±\pm 0.0450.955 ±\pm 0.0140.000 ±\pm 0.0000.973 ±\pm 0.0130.715 ±\pm 0.0160.870 ±\pm 0.019
DrM0.599 ±\pm 0.0650.811 ±\pm 0.0300.611 ±\pm 0.0240.900 ±\pm 0.0280.334 ±\pm 0.0250.606 ±\pm 0.046
CeSD0.000 ±\pm 0.0000.267 ±\pm 0.0150.319 ±\pm 0.0100.000 ±\pm 0.0000.000 ±\pm 0.0000.793 ±\pm 0.027
LBS0.576 ±\pm 0.0220.873 ±\pm 0.0370.053 ±\pm 0.0270.937 ±\pm 0.0270.517 ±\pm 0.0250.344 ±\pm 0.022
RAP0.502 ±\pm 0.0410.776 ±\pm 0.0310.650 ±\pm 0.0140.970 ±\pm 0.0140.627 ±\pm 0.0130.791 ±\pm 0.024
CTRL-SR0.000 ±\pm 0.0000.470 ±\pm 0.0390.361 ±\pm 0.0700.919 ±\pm 0.0380.313 ±\pm 0.0100.838 ±\pm 0.033
Appendix
Table 2: We report the best episode success rate of TEB and baselines with mean ±\pm std on MetaWorld tasks.
Figure 7: Comparison of the learning curves of TEB and baselines in Maze environments with 100k training steps. Each experiment runs with 10 random seeds, with shaded region representing the standard deviation across seeds.
Figure 8: Visualization of state coverage in Maze2D tasks.
Shared HyperparameterValue in MetaWorldValue in Maze2D
Training steps1∼31\sim 3 M100K
Seed frames40004000
Exploration steps20002000
Evaluation episodes10-
Replay buffer capacity2×1052\times 10^{5}2×1052\times 10^{5}
Episode length100050
Appendix
Table 3: Detailed hyperparameters.
More in this topic

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 220 papers

Explore similar work

CardsList

  1. Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards

    Mar 2, 2026Faisal Mohamed, Catherine Ji, Benjamin Eysenbach +1RL ExplorationRobotic RL

    Mila-Quebec AI Institute · Université de Montréal · Princeton University

  2. Explore Beyond the Boundary Using Entropic Information

    Jul 31, 2026Bumgeun Park, Donghwan LeeReinforcement LearningRL Exploration

    Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea

  3. Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

    Jun 19, 2026Marco Pratticò, Pietro Novelli, Massimiliano Pontil +1Reinforcement LearningExploration-Exploitation Tradeoff

    Computational Statistics and Machine Learning - Istituto Italiano di Tecnologia, 16100 Genova, Italy · 2AI Centre, Computer Science Department, University College London, London, UK.