cs.LGSep 28, 2026
SaveDeep Epistemic Value Functions for Optimistic Exploration
Organizations: ETH Zürich, Switzerland · Max Planck Institute for Intelligent Systems, Germany
Abstract
Principled exploration in reinforcement learning requires an agent to quantify its epistemic uncertainty and act to resolve it. Uncertainty over the value function provides a natural signal for exploration, yet existing deep approximations remain brittle and perform inconsistently. The central challenge is therefore to scale these ideas robustly. We conduct a systematic empirical study of how epistemic uncertainty is represented, propagated, and optimized in deep epistemic value functions, and uncover distinct failure modes along each of these axes. These findings motivate DEVOTE, a model-free reinforcement learning algorithm that controls how uncertainty generalizes beyond observed data, stabilizes its temporal propagation, and preserves adaptation to the resulting non-stationary exploration objective. Across reward-free exploration and challenging continuous-control tasks, DEVOTE reaches novel states more effectively and achieves higher task return than strong model-free and model-based exploration baselines. These results provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.
Figures & tables
Figure 1: DEVOTE provides stable and scalable estimates of a deep epistemic value function, guides the exploratory policy toward novel and informative states, and outperforms strong RL exploration methods across a range of environments. We plot the average coverage and performance across environments outlined in Section 5 .
Figure 2: Comparison of uncertainty estimates from an Ensemble, an ENN with MLP prior (ENN MLP), and an ENN with RFN prior (ENN RFN) on a synthetic regression task. Dashed curves show the true function, dots indicate training observations, solid curves show predictive means, and shaded bands represent . ENNs with RFN function-space priors yield better calibrated uncertainty.
Figure 3: Epistemic value estimation on DMC Cartpole Swingup across prior complexities . Left: OOD-detection AUROC of the predicted uncertainty . Middle: Pearson correlation between predicted uncertainty and value-estimation error. Right: Predicted initial-state value over training, compared with the Monte Carlo reference. TUD improves both uncertainty metrics across length scales and tracks the Monte Carlo value more closely.
Figure 4: State coverage and the fraction of dormant neurons ( Sokar et al., 2023 ) in the U-shaped PointMaze environment across soft-reset strengths. Increasing the strength of soft resets increases exploration performance and avoids loss of plasticity.
Figure 5: State coverage over training for DEVOTE and the baselines in the pure-exploration environments. DEVOTE consistently drives exploration toward novel states, matching or outperforming all baselines across tasks.
Figure 6: State coverage over training when ablating the DEVOTE components introduced in Section 3 . All components contribute in the continuous exploration tasks, with their relative importance depending on the structure of the environment.
Figure 7: Normalized return over training in the high-dimensional control environments. DEVOTE achieves the strongest task performance across both environments, indicating that its exploration signal can be combined effectively with extrinsic reward.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Length scale |
|---|---|
| DeepSea | |
| PointMaze | |
| Cheetah | |
| AntMaze | |
| PickAndPlace |
Table 2: Environment-specific RFN length scales. For each environment, we select one value from the fixed candidate set and share it across all DEVOTE variants.
| Cartpole anchor | Cart position | Pole angle | Cart velocity |
|---|---|---|---|
| Upright, balanced | 0 | 0 | 0 |
| Near top, offset | 0 | 0.6 | 0 |
| Mid-swing, climbing | 0.4 | 1.5 | 0.5 |
| Rail-pinned, hanging | |||
| Racing upright | 0 | 0 | 6.0 |
| Rail-charging, hanging | 1.5 | 4.0 |
Table 3: Cartpole anchor configurations. Angles are in radians; anchor pole angular velocity and held action are zero. Pole angular velocity is swept over .
Figure 8: Environments used for evaluation. Top: Pure-exploration environments, with task rewards removed: Behaviour Suite DeepSea ( ) ( Osband et al., 2020 ) , Spiral PointMaze, and DMC Cheetah ( Tassa et al., 2018 ) . Together, these environments span discrete exploration, navigation, and high-dimensional continuous control. Bottom: Complex-control environments: AntMaze and PickAndPlace ( Zakka et al., 2025 ) . AntMaze requires navigating around obstacles to reach the goal, while PickAndPlace requires discovering coordinated manipulation behavior.
Figure 9: Samples from a GP prior, a randomly initialized ReLU MLP prior, and an RFN prior on the synthetic one-dimensional domain. The sampled MLP functions are approximately linear over this interval, whereas the RFN distributes variability throughout the domain. The GP and RFN use an RBF length scale of one.
Figure 10: Fixed-policy cartpole evaluation with RFN length scale . Columns show upright, near-top, and mid-swing anchors while pole angular velocity is varied and the remaining state coordinates and anchor action are held fixed. Top: Monte Carlo returns and TD/TUD predictions, with shaded bands showing one predicted standard deviation . Bottom: estimated log visitation density. TUD reduces uncertainty in frequently visited regions while retaining uncertainty outside them; TD produces comparatively uniform uncertainty across the slices.
Figure 11: Fixed-policy cheetah evaluation across RFN prior length scales. Left: OOD-detection AUROC. Middle: error–uncertainty correlation. Right: predicted initial-state value over training compared with Monte Carlo returns. TUD improves both calibration metrics for , while standard TD learning exhibits pronounced overestimation at small length scales. Results use three seeds.
Figure 12: U-shaped PointMaze diagnostics without resets ( ) and with resets ( ). Maps show visitation density , mean absolute residual , and mean residual bootstrap . Each snapshot depicts a single run. Color scales are specific to each map, so the comparison concerns spatial structure rather than absolute color intensity across maps.
Figure 13: Pure-exploration performance of DEVOTE under varying RFN length scales , measured by state coverage. Prior complexity has little effect in the discrete DeepSea environment, while PointMaze favors the smaller length scale and Cheetah favors the larger length scale .
Figure 14: State coverage in AntMaze and PickAndPlace, complementing the normalized returns in the main text. Coverage measures the fraction of visited bins in ant – position and grasped-cube – – position, respectively. DEVOTE expands coverage fastest in both environments. RND also reaches full coverage in AntMaze, but achieves lower coverage than SAC in PickAndPlace.