Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning

Hoda Yamani; Yuning Xing; Koen van Rijnsoever; Bruce A. MacDonald; Henry Williams

cs.LGAug 18, 2026

Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning

Authors: Hoda Yamani, Yuning Xing, Koen van Rijnsoever, Bruce A. MacDonald, Henry Williams

Organizations: Robot Learning Team, Center for Automation and Robotic Engineering Science (CARES), Department of Electrical, Computer and Software Engineering, University of Auckland, New Zealand

Abstract

Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance. Motivated by this biological principle, we introduce Instant Episode Repetition (IER), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction. Unlike conventional approaches such as Experience Replay and Self-Imitation Learning (SIL), which passively reuse past experience during training updates, IER directly influences the data collection process. Upon identifying a high-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment. We integrate IER into state-of-the-art SAC and TD3 algorithms and evaluate its effectiveness on continuous-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real-world dynamic object translation task with a robotic manipulator. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self-imitation-based baselines.

Figures & tables

Figure 1: Illustration of the Instant Episode Repetition Algorithm: At the end of each episode (when done is true), if the agent identifies an episode with a high reward , it immediately initiates repetition by executing the actions from that episode sequentially, from start to end, for $RN$ repetitions within the environment.

Figure 2: Real-world setup for dynamic object translation. The cube must be moved laterally while suspended in the air by a custom-built gripper.

Panel of Figure 3 — Figure 3: Performance comparison of Baseline, SIL, and the proposed IER method across eight continuous-control tasks under SAC (top) and TD3 (bottom). Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds.

Panel of Figure 4 — Figure 4: Performance comparison of standard, SIL-enhanced, and IER-enhanced SAC and TD3 variants in the real-world gripper manipulation task. Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds.

Figure 5: Mean $\Delta$ AUC% across all eight tasks as a function of the RN, jointly averaged over SAC and TD3.

Figure 6: Per-task $\Delta$ AUC% as a function of the Repetition Number (RN), from RN0 (no repetition) to RN7, for SAC and TD3.

Panel of Figure 7 — Figure 7: Impact of repetition number RN on IER-SAC performance across all tasks, showing how repeated execution of the highest-reward episode affects learning. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity.

Panel of Figure 8 — Figure 8: Impact of repetition number RN on IER-TD3 performance across all tasks, showing how repeated execution of the highest-reward episode affects learning. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity.

Panel of Figure 9 — Figure 9: tasks used in our experiments.

Panel of Figure 10 — Figure 10: Mean $\Delta$ AUC% across environment groups. Left: MuJoCo tasks. Right: DeepMind Control tasks.

Task	Agreement RN	Selection Type
Ant-v4	RN1	Intersection
Cheetah-Run	RN3	Intersection
Walker-Walk	RN5	Intersection
Cartpole-Swingup	RN2	Intersection
HalfCheetah-v4	RN4	Rank-based
Hopper-v4	RN2	Rank-based

Table 1: Per-task RN agreement based on 95% near-best criterion.

Figure 11: Frequency of RN values selected as robust joint candidates across tasks. RN2, RN3, and RN5 appear most frequently.

Environment	Suite	Task Type	Description	Approx. Difficulty
Ant-v4	MuJoCo	Locomotion	Quadruped; coordination-heavy with complex contact dynamics.	High
HalfCheetah-v4	MuJoCo	Locomotion	Planar biped; dense unbounded rewards support long-term learning.	Medium
Humanoid-v4	MuJoCo	Locomotion	Biped with high DOF; unstable dynamics and sensitive to control noise.	High
Hopper-v4	MuJoCo	Locomotion	Single-legged hopping; requires balance and precise control, with early termination on failure.	Medium
Walker-Walk	DMC	Locomotion	Bipedal walking; fewer DOF than Humanoid, but still unstable.	Medium
Cheetah-Run	DMC	Locomotion	Fast planar cheetah; velocity-based control under bounded rewards.	Low

Table 2: Summary of environments used in evaluation.

MuJoCo (vector-based)
Environment	Observation Space	Action Space	Horizon
Ant-v4	$\mathbb{R}^{111}$	$[-1,1]^{8}$	1000 (may terminate early due to instability)
HalfCheetah-v4	$\mathbb{R}^{17}$	$[-1,1]^{6}$	1000
Humanoid-v4	$\mathbb{R}^{376}$	$[-1,1]^{17}$	1000 (may terminate early due to instability)
Hopper-v4	$\mathbb{R}^{11}$	$[-1,1]^{3}$	1000
DeepMind Control Suite (vector-based)

Table 3: Observation and action space dimensions and episode horizon for each environment.

\mathbb{R}^{d}

denotes a

d

-dimensional real vector;

[-1,1]^{d}

denotes a bounded action space.

Feature	Description
Task Overview
Task Type	Dynamic cube translation with robotic gripper
Object	Cube tracked via ArUco markers
Goal	Translate cube as far as possible within camera view
Reward Components	Force-closure grasp + lateral displacement
Hardware Setup

Table 4: Real-World Robotic Gripper Task Characteristics

Figure 12: Workspace boundaries and cube tracking for reward evaluation. Green and red markers indicate current and previous cube positions used to compute displacement-based rewards.

Shared Parameters
Parameter	Value (Description)
batch_size	256 (Mini-batch size)
buffer_size	$10^{6}$ (Replay buffer size)
max_steps_exploration	$10^{3}$ (Exploration steps before training)
max_steps_training	$10^{6}$ (Total training steps)
number_steps_per_evaluation	$10^{4}$ (Steps between evaluations)

Table 5: Hyperparameters used in training. Shared parameters apply to both SAC and TD3; SAC, TD3, and SIL sections list algorithm-specific settings.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
Oct 27, 2024Kaiyan Zhao, Yiming Wang, Yuyang Chen +3Experience Replay Determinantal Point Process
School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China
Temporal Self-Imitation Learning
Jun 18, 2026Yinsen Jia, Boyuan ChenRL for Robotics Long-Horizon Robotic Manipulation
Duke University
Match or Replay: Self Imitating Proximal Policy Optimization
Mar 29, 2026Gaurav Chaudhary, Laxmidhar Behera, Washim Uddin MondalRL Exploration Sample-Efficient RL
Department of Electrical Engineering, Indian Institute of Technology Kanpur

Oct 27, 2024J/K move · Enter open · S save

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Kaiyan Zhao, Yiming Wang, Yuyang Chen, Yan Li+2

School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China

Experience Replay Determinantal Point Process Robotic RL Sample-Efficient RL

Open paper AskPDF arXiv

Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.

Abstract

Reinforcement Learning

Explore similar work

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Temporal Self-Imitation Learning

Match or Replay: Self Imitating Proximal Policy Optimization