Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning

Hoda Yamani; Henry Williams; Bruce A. MacDonald

cs.LGAug 18, 2026

Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning

Authors: Hoda Yamani, Henry Williams, Bruce A. MacDonald

Organizations: Robot Learning Team and CARES Robotics Lab, Department of Electrical, Computer, and Software Engineering, University of Auckland, Auckland, New Zealand.

Abstract

Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.

Figures & tables

Figure 1: Schematic overview of Novelty and Surprise Prioritized Experience Replay (NSPER).

p(i)=\frac{\sigma_{i}^{\alpha}}{\sum_{k}\sigma_{k}^{\alpha}}

Algorithm 1 NSPER and NSPER+R with NSS (Novelty–Surprise Signal)

Panel of Figure 2 — Figure 2: Five continuous control tasks from the DeepMind Control Suite used in our evaluation.

Task	Action Space	Objective	Primary Challenge	Type	Difficulty
Cartpole-Balance	$[-1,1]^{1}$	Stabilize the pole by moving the cart horizontally.	Underactuated and highly unstable dynamics where small errors rapidly lead to failure.	Classic Control	Low
Finger-Spin	$[-1,1]^{2}$	Maintain continuous rotation of a free-floating object.	Requires fine torque modulation to sustain spin without drift or loss of contact.	Manipulation	High
Ball-in-Cup	$[-1,1]^{2}$	Swing the ball and capture it inside the cup.	Oscillatory motion requires precise timing and well-coordinated control.	Manipulation	Medium–High
Walker-Walk	$[-1,1]^{6}$	Achieve stable forward locomotion.	Balancing and coordinating a bipedal body in an inherently unstable locomotion regime.	Locomotion	Medium
Cheetah-Run	$[-1,1]^{6}$	Run forward at high speed.	High velocity amplifies instability, requiring strong, consistent, and fast control.	Locomotion	Medium

Table 1: Summary of DeepMind Control Suite tasks used in our experiments.

Environments
Parameter	Value (Description)
Observation	$84\times 84\times 3$ RGB images, with 3 consecutive frames stacked
Reward Scale	Maximum episodic return of 1000 per task
Evaluation Metric	Average cumulative reward per step (computed using extrinsic rewards)
Implementation Details
Replay Buffer Capacity	$1{,}000{,}000$ transitions

Table 2: Experimental setup for NSPER across DeepMind Control Suite environments.

Component	Input	Layers / Units	Activations	Output
Encoder	$84\times 84\times 3$ image	4 conv (32, $3\times 3$ ) + FC(200)	ReLU (conv), $\tanh$ (FC)	Latent $z\in\mathbb{R}^{200}$
Decoder	Latent $z$	Deconv mirror of encoder	ReLU, Sigmoid (final)	Image reconstruction
Actor (TD3)	$z$ (opt. + vector obs)	FC(1024) → FC(1024)	ReLU, $\tanh$ (out)	Action $\in[-1,1]^{d}$
Critics (TD3, $\times 2$ )	$z\\|a$ (opt. + vector obs)	FC(1024) → FC(1024)	ReLU	Scalar $Q$ -value
Predictive Ensemble	$z_{t}\\|a_{t}$	FC(512) → FC(512)	ReLU	Pred. latent $\hat{z}_{t+1}$

Table 3: Network architectures used in NSPER and NSPER+R.

Panel of Figure 3 — Figure 3: Learning curves for continuous control tasks using PixelTD3. Solid lines represent the algorithm’s performance with intrinsic rewards (+R), while dashed lines indicate performance without intrinsic rewards. Shaded regions denote the 95% confidence interval across 5 runs.

Panel of Figure 4 — Figure 4: Ablation results comparing experience prioritization based on novelty, surprise, and their combination, with and without intrinsic rewards. Shaded regions represent the 95% confidence interval over five runs.

Prioritized Experience Replay

See all 10 papers

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
Oct 27, 2024Kaiyan Zhao, Yiming Wang, Yuyang Chen +3Experience Replay Determinantal Point Process
School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China
Match or Replay: Self Imitating Proximal Policy Optimization
Mar 29, 2026Gaurav Chaudhary, Laxmidhar Behera, Washim Uddin MondalRL Exploration Sample-Efficient RL
Department of Electrical Engineering, Indian Institute of Technology Kanpur
Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
Apr 18, 2026Weiyu Ma, Yongcheng Zeng, Yan Song +4Reinforcement Learning RL Post-Training
King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London +1

Oct 27, 2024J/K move · Enter open · S save

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Kaiyan Zhao, Yiming Wang, Yuyang Chen, Yan Li+2

School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China

Experience Replay Determinantal Point Process Robotic RL Sample-Efficient RL

Open paper AskPDF arXiv

Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.

Abstract

Prioritized Experience Replay

Explore similar work

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Match or Replay: Self Imitating Proximal Policy Optimization

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning