Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards

Faisal Mohamed; Catherine Ji; Benjamin Eysenbach; Glen Berseth

cs.LGMar 2, 2026

Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards

Authors: Faisal Mohamed, Catherine Ji, Benjamin Eysenbach, Glen Berseth

Organizations: Mila-Quebec AI Institute · Université de Montréal · Princeton University

Abstract

Effective exploration in reinforcement learning requires not only tracking where an agent has been, but also understanding how the agent perceives and represents the world. To learn powerful representations, an agent should actively explore states that contribute to its knowledge of the environment. Temporal representations can capture the information necessary to solve a wide range of potential tasks while avoiding the computational cost associated with full state reconstruction. In this paper, we propose an exploration method that leverages temporal contrastive representations to guide exploration, prioritizing states with unpredictable future outcomes. We demonstrate that such representations can enable the learning of complex exploratory x in locomotion, manipulation, and embodied-AI tasks, revealing capabilities and behaviors that traditionally require extrinsic rewards. Unlike approaches that rely on explicit distance learning or episodic memory mechanisms (e.g., quasimetric-based methods), our method builds directly on temporal similarities, yielding a simpler yet effective strategy for exploration.

Figures & tables

Figure 1

Panel of Figure 2 — Algorithm 1 Curiosity-Driven Exploration via Temporal Contrastive Learning

Figure 4: Environments . Maze coverage, robotic manipulation, and the survival game Craftax.

Figure 5: C-TeC Performance compared to ETD ( Jiang et al., 2025 ) C-TeC is competitive to ETD in terms of state coverage in continuous control environments, and outperform ETD in Crafter.

Figure 6: Evolution of the C-TeC reward during training . This figure shows how the intrinsic reward changes over the course of training based on future state visitation. The black circle in the lower-left corner represents the starting state. C-TeC reward captures the agent’s future state density and rewards the agent for visiting states faraway in the future.

Figure 7: C-TeC explores more states than prior methods. We compare the state coverage of C-TeC to APT ( Liu & Abbeel, 2021 ) , RND ( Burda et al., 2019b ) and ICM ( Pathak et al., 2017 ) . We include a uniform random policy as well.

Figure 8: C-TeC behavior in humanoid-u-maze . C-TeC agent learns to escape the u-maze by jumping over the wall. None of the alternative exploration methods discovered this kind of unexpected behavior 2 .

Figure 9: C-TeC achieves higher achievements in the Craftax survival game.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	500M steps	50M steps	30M steps
Ant-hardest-maze	$2500\pm 300$	$1916\pm 430$	$1119\pm 304$
Humanoid-u-maze	$230\pm 40$	$143\pm 34$	$102\pm 11$
Arm-binpick-hard	$135000\pm 10000$	$40000\pm 14000$	$31150\pm 3156$

Appendix

Table 1: Sample Efficiency of C-TeC

Figure 10: C-TeC State Coverage The Black circle in the lower left corner is the starting state. The figure on the left shows the state coverage when we run C-TeC with 50M environment steps in ant-hardest-maze . On the right, we show state coverage when we run C-TeC with 500M environment steps.

Figure 11: Representations Variance C-TeC’s contrastive model does not suffer from mode collapse.

Figure 12: State coverage when leveraging prior knowledge C-TeC outperforms prior methods APT, RND, ICM, and can explore effectively when leveraging prior knowledge. This shows the improved flexibility of C-TeC in incorporating prior knowledge by narrowing the exploration space compared to prior work.

Hyperparameter	Value
num_timesteps	500,000,000
max_replay_size	10,000
min_replay_size	1,000
episode_length	1,000
discounting	0.99
num_envs	1024 (256 for humanoid_u_maze )

Appendix

Table 2: Hyperparameters for all methods in robotics environments

Hyperparameter	Value
contrastive_lr	3e-4
contrastive_loss_function	InfoNCE
similarity_function	L1
logsumexp_penalty	0.1
hidden layers (for both encoders)	[1024,1024]
representation dimension	64

Appendix

Table 3: Hyperparameters for C-TeC in robotics environments

Hyperparameter	Value
rnd encoder lr	3e-4
rnd embedding dim	512
rnd encoder hidden layers	[256, 256]
icm encoder lr (forward and inverse models)	3e-4
icm embeddings_dim	512
icm encoders hidden layers	[1024, 1024]

Appendix

Table 4: Hyperparameters for baselines in robotics environments

Hyperparameter	Value
num_timesteps	1,000,000,000
num_steps	64
learning_rate	2e-4
anneal_learning_rate	True
update_epochs	4
discounting	0.99

Appendix

Table 5: Hyperparameters for all methods in robotics environments

Hyperparameter	Value
contrastive_lr	3e-4
contrastive_loss_function	InfoNCE
similarity_function	L2
Discounting	0.3
logsumexp_penalty	0.0
hidden layers (for both encoders)	[1024,1024,1024]

Appendix

Table 6: Hyperparameters for C-TeC in Craftax

Hyperparameter	Value
rnd encoder lr	3e-4
rnd embedding dim	512
rnd encoder hidden layers	[256, 256]
icm encoder lr (forward and inverse models)	3e-4
icm embeddings_dim	512
icm encoders hidden layers	[256, 256]

Appendix

Table 7: Hyperparameters for baselines in Craftax

Figure 13: Normalizing the contrastive representations. Normalizing the representations is crucial for effective exploration—using unnormalized representations significantly degrades exploration performance.

Figure 14: Comparison of Different Contrastive Losses. Overall, C-TeC is robust to the choice of contrastive loss. A notable exception is the Binary NCE loss in Craftax, where it performs relatively poorly.

Figure 15: Comparison of Critic function. Overall,the

L_{1}

distance yields the best performance across the robotic environments, while

L_{2}

performs better in Craftax.

Figure 16: Critic parameterization Using a monolithic critic results in poor exploration performance, while using the separable architecture results in much better exploration. This shows the importance of the critic parameterization as a distance function between two representations.

Figure 17: Forward vs Reverse KL C-TeC with the reverse KL reward promotes mode-seeking behavior which encourages the agent to prioritize visiting unfamiliar states resulting in much better exploration.

Figure 18: Sensitivity to future state sampling strategy. We compare variants of C-TeC with different future state sampling strategies, the method is robust to the choice of the sampling strategy and all the variants outperform the baselines.

Figure 19: Sensitivity of C-TeC to the discount factor Different environments require different discount factors to obtain the best exploration performance.

Figure 20: Sensitivity of C-TeC to the temperature

\tau

in the contrastive loss ( Equation 2 ) C-TeC is sensitive to the temperature parameter, however, in general, a value of

\tau=1

yields a large state coverage across environments.

Figure 21: In-episode vs Across-episode negative sampling Sampling in-episode negatives results in additional performance boost in humanoid-u-maze and arm-binpick-hard .

Figure 22: C-TeC Coverage in noisy TV setting C-TeC can effectively explore in the presence of noisy states

Figure 23: Xland-Minigrid ( Nikulin et al., 2024 ) with noisy TV states indicated by the random colors.

Figure 24: From the root, trajectories go either left or right. The left branch has fast, deterministic dynamics: the agent moves one level per timestep to the leaf. The right branch has slow, sticky dynamics: the agent stays in place with some probability and progresses with the remaining probability. After choosing a branch, the dynamics are policy-independent, reducing the problem to a two-armed bandit.

Figure 25: State visitation of C-TeC and ETD, C-TeC distribution has a mode on the most left node, while ETD prefers to stay in any state in the right branch

Figure 26: Future state distribution in the toy MDP in Figure 24 . C-TeC prefers the states in the left branch with fast, deterministic dynamics while ETD prefers the right branch with slow, sticky dynamics.

Figure 27: Toy Tree MDP. At the root, transitions are near deterministic. At level 1, taking the left child or right child action works only 10% of the time, while action stay works 90% of the time.

Figure 28: forward-looking (C-TeC) vs backward-looking (ETD) rewards As indicated by the large state visitation C-TeC prefers to stay at node 1 since it can reach multiple distinct future states, while ETD prefers to stay at node 3, the deepest node in the tree.

Environment	C-TeC	MBRL ( Stadie et al., 2015 )
Ant-hardest-maze	$2500\pm 300$	$849\pm 63$
Humanoid-u-maze	$230\pm 40$	$31\pm 8$
Arm-binpick-hard	$135000\pm 10000$	$35000\pm 3170$

Appendix

Table 8: Sample Efficiency of C-TeC

Figure 29: Emergent Exploration Behavior in humanoid-u-maze . C-TeC exhibits interesting emergent behaviors; for example, in the humanoid-u-maze environment, the agent learns to jump over the maze walls to escape the maze. Each row represents an independent evaluation epsidoe.

Figure 30: Qualitative Comparison in humanoid-u-maze .

Figure 31: C-TeC Achievements. C-TeC unlocks interesting achievements in Craftax-Classic; the plot shows a subset of the unlocked achievements during an evaluation episode.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 220 papers

Efficient Exploration Is Enough
Sep 7, 2026Mikel Malagón, Jon Vadillo, Josu Ceberio +2Intrinsic Reward Methods RL Exploration
University of the Basque Country (UPV/EHU) · University of Alberta · Alberta Machine Intelligence Institute +2
Task-Aware Exploration via a Predictive Bisimulation Metric
Feb 21, 2026Dayang Liang, Ruihan Liu, Lipeng Wan +2RL Exploration Visual RL
Department of Automation, Xiamen University, Xiamen, China · Department of Computer Science, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · School of Artificial Intelligence, Xian Jiaotong University, Xian, China +1
Look Before You Leap: Autonomous Exploration for LLM Agents
May 15, 2026Ziang Ye, Wentao Shi, Yuxin Liu +6RL Exploration LLM Agents
University of Science and Technology of China · Meituan

Sep 7, 2026J/K move · Enter open · S save

Efficient Exploration Is Enough

Mikel Malagón, Jon Vadillo, Josu Ceberio, Michael Bowling+1

University of the Basque Country (UPV/EHU) · University of Alberta · Alberta Machine Intelligence Institute+2

Intrinsic Reward Methods RL Exploration Generalization in RL

Open paper AskPDF arXiv

This work introduces an alternative view of efficient exploration and studies its theoretical and empirical implications in the absence of extrinsic rewards. Specifically, we define efficient explorers as agents that prioritize generating generalizable experience, i.e., data that supports learning models capable of predicting and adapting across the environment. This allows us to analyze efficient exploration through the lens of prediction and generalization. Theoretically, we demonstrate that optimally efficient explorers naturally schedule their trajectories to visit the most informative and learnable regions first. Empirically, we show that optimizing for these agents gives rise to an automatic curriculum of progressively more complex behaviors, even in relatively simple environments. These results indicate that pursuing this purely intrinsic objective alone is enough to drive the emergence of highly sophisticated behaviors. We believe that this new framework provides a principled mechanism by which agent-environment systems may sustain an open-ended process of increasingly complex behavior without external rewards, tasks, or objectives.

Abstract

RL Exploration

Explore similar work

Efficient Exploration Is Enough

Task-Aware Exploration via a Predictive Bisimulation Metric

Look Before You Leap: Autonomous Exploration for LLM Agents