Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

Armando Vieira

cs.LGSep 4, 2026

Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

Authors: Armando Vieira

Organizations: University of Tartu

Abstract

We propose a reinforcement learning framework in which exploration is driven by intrinsic curiosity, designed for scenarios where environments are non-stationary and rewards are sparse, delayed, uninformative, or absent. In our model, action selection is guided by a combination of external rewards and an epistemic motivation mechanism that biases the agent toward structured exploratory directions. The central hypothesis is that effective exploration emerges at intermediate levels of incoherence, while performance degrades under both overly rigid and overly disordered dynamics. To test this idea, we implement the framework on top of a Liquid State Machine (LSM) substrate and evaluate it on two standard benchmarks: the discrete-action LunarLanderv2 and the continuous-control BipedalWalkerv3. The proposed method achieves competitive performance on both tasks relative to established deep RL algorithms, including Proximal Policy Optimization (PPO) and Intrinsic Curiosity Module (ICM). We further show that the curiosity window is not recovered in Active Inference agents under the same analysis, suggesting that the proposed dynamics capture a distinct exploration regime

Figures & tables

Figure 1: Phase diagram of exploration dynamics across noise regimes. Each panel shows a different property of the dynamics as a function of the reach-noise parameter $\sigma_{\pi}$ : mean incoherence $I(t)$ , global overlap $G(t)$ , basin coherence, exploration entropy, dwell time, transition rate, and composite curiosity metrics. The curiosity window (bottom centre) is visible as a distinct intermediate regime.

Figure 2: Representative time series for three noise regimes. Top : Low noise ( $\sigma_{\pi}=0.05$ )—trapped in a single basin. Middle : Intermediate noise ( $\sigma_{\pi}=0.25$ )—coherent transitions across multiple basins. Bottom : High noise ( $\sigma_{\pi}=0.8$ )—rapid, incoherent switching with loss of basin structure.

Figure 3: Curiosity metrics as a function of noise $\sigma_{\pi}$ at coupling strength $\eta_{\pi}=0.12$ . The $C_{2}$ metric shows a clear peak at intermediate noise, confirming the Curiosity Window conjecture. The $>1.5\times$ drop-off on both sides of the peak demonstrates that the window is sharply defined.

Figure 4: Sedimentation learning across 50 episodes. The memory agent (red) achieves lower incoherence and higher global overlap than the memoryless control (blue), demonstrating that sedimentation functions as a viable learning mechanism. Error bands show $\pm 1$ standard deviation across runs.

Metric	Memory agent	Control (no memory)
Final incoherence improvement	$14.3\%$	$8.3\%$
Learning advantage	$+5.9$ pp	—
Final coherence $G$	$0.94$	$0.82$

Table 1: Sedimentation learning advantage over 60 episodes. The memory agent consistently outperforms the memoryless control on all coherence-related metrics.

Figure 5: Architecture of the ECF–PPO hybrid agent. The LSM substrate produces a latent encoding $z_{t}$ , from which the ECF module computes reach ( $\pi_{t}$ ), yield ( $y_{t}$ ), and memory ( $m_{t}$ ) fields. Incoherence and novelty signals modulate an intrinsic reward $p_{\mathrm{ECF},t}$ that augments the external reward before PPO updates the policy.

Agent	P1 (Train)	P2 (Retain)	P3 (Wind)	P4 (Recover)
ECF-RichMem	$\mathbf{16.3\pm 13.4}$	$\mathbf{40.0\pm 10.3}$	$\mathbf{-16.7\pm 12.7}$	$\mathbf{5.7\pm 16.2}$
ECF	$-6.4\pm 7.5$	$-34.2\pm 10.1$	$-58.3\pm 20.7$	$-45.3\pm 35.2$
ICM	$-18.3\pm 5.3$	$-18.6\pm 35.7$	$-90.3\pm 32.4$	$-72.3\pm 27.9$

Table 2: Mean performance across phases in LunarLander-v2. Values are mean

\pm

standard deviation across seeds. Bold indicates the best-performing agent in each phase.

Figure 6: BipedalWalker-v3 results across the four experimental phases. Phase 1: standard training under nominal dynamics. Phase 2: retention under nominal conditions without reward. Phase 3: disturbance via altered gravity. Phase 4: recovery after restoring nominal dynamics. PPO+ECF shows substantially better robustness in Phase 3.

Agent	P1 (Train)	P2 (Retain)	P3 ( $\Delta$ Gravity)	P4 (Recover)
PPO	$\mathbf{31.1\pm 45.9}$	$\mathbf{42.6\pm 60.3}$	$-20.3\pm 36.6$	$3.1\pm 58.2$
PPO+ECF	$29.0\pm 75.9$	$37.4\pm 80.2$	$\mathbf{45.3\pm 80.3}$	$\mathbf{5.4\pm 85.3}$

Table 3: BipedalWalker-v3 results across four phases. Values are mean

\pm

standard deviation. Bold indicates the best-performing agent in each phase.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 220 papers

Can In-Context Learning Support Intrinsic Curiosity?
Jun 17, 2026Eric Elmoznino, Sangnie Bhardwaj, Johannes von Oswald +5Intrinsic Reward Methods Intrinsic Motivation
1Google – Paradigms of Intelligence Team, 2Google DeepMind
Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration
May 21, 2026Lily Goli, Justin Kerr, Daniele Reda +3Reinforcement Learning Embodied Navigation
University of Toronto · Wayve · Vector Institute +2
Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators
Jul 18, 2026Alireza Furutanpey, Schahram DustdarFree Energy Calculation Intrinsic Reward Methods
Coovally, Barcelona, Spain · Distributed Systems Group, TU Vienna, Austria · ICREA Barcelona, Spain

Jun 17, 2026J/K move · Enter open · S save

Can In-Context Learning Support Intrinsic Curiosity?

Eric Elmoznino, Sangnie Bhardwaj, Johannes von Oswald, Rajai Nasser+4

1Google – Paradigms of Intelligence Team, 2Google DeepMind

Intrinsic Reward Methods Intrinsic Motivation Active Learning RL Exploration In-Context Learning

Open paper AskPDF arXiv

Effective machine learning depends not only on how we model data, but also on what data we choose to collect. While large sequence models have revolutionized data modeling, the problem of automated data selection, or "intrinsic curiosity", remains a significant challenge. Classic approaches incentivize exploration by rewarding an agent based on its "learning progress", which measures how much a newly acquired observation improves a world model's predictive ability. However, evaluating these rewards traditionally requires expensive inner loops of gradient descent updates within each trajectory, rendering them computationally impractical at scale. In this work, we investigate whether the emergent in-context learning (ICL) capabilities of sequence models can eliminate this bottleneck by serving as immediate, update-free world models. Specifically, we evaluate whether an exploration policy can be trained to maximize learning progress, using solely the prediction errors and counterfactual context manipulations of an in-context learner. We first prove that in general Markov decision processes, this is in fact impossible in an unbiased way: the resulting intrinsic rewards either suffer from nuisance terms that bias their estimation of true learning progress, or they cannot be implemented using an in-context learner's prediction errors. Conversely, we prove a positive result for a broad subclass of non-temporal settings, encompassing active learning and Bayesian Experimental Design: here, ICL-derived rewards successfully bound and asymptotically converge to the true learning progress. We corroborate our theory with controlled experiments across continuous and symbolic environments, demonstrating that our ICL-driven framework successfully trains curious data-collection policies that explore optimally.

Abstract

RL Exploration

Explore similar work

Can In-Context Learning Support Intrinsic Curiosity?

Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators