Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning
Authors: Hoda Yamani, Yuning Xing, Koen van Rijnsoever, Bruce A. MacDonald, Henry Williams
Organizations: Robot Learning Team, Center for Automation and Robotic Engineering Science (CARES), Department of Electrical, Computer and Software Engineering, University of Auckland, New Zealand
Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance. Motivated by this biological principle, we introduce Instant Episode Repetition (IER), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction. Unlike conventional approaches such as Experience Replay and Self-Imitation Learning (SIL), which passively reuse past experience during training updates, IER directly influences the data collection process. Upon identifying a high-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment. We integrate IER into state-of-the-art SAC and TD3 algorithms and evaluate its effectiveness on continuous-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real-world dynamic object translation task with a robotic manipulator. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self-imitation-based baselines.
Figures & tables
Figure 1: Illustration of the Instant Episode Repetition Algorithm: At the end of each episode (when done is true), if the agent identifies an episode with a high reward , it immediately initiates repetition by executing the actions from that episode sequentially, from start to end, for RN repetitions within the environment.
Figure 2: Real-world setup for dynamic object translation. The cube must be moved laterally while suspended in the air by a custom-built gripper.
Figure 3: Performance comparison of Baseline, SIL, and the proposed IER method across eight continuous-control tasks under SAC (top) and TD3 (bottom). Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds.
Figure 4: Performance comparison of standard, SIL-enhanced, and IER-enhanced SAC and TD3 variants in the real-world gripper manipulation task. Curves show the mean episodic return over five independent seeds, and shaded regions represent the standard deviation across five random seeds.
Figure 5: Mean Δ AUC% across all eight tasks as a function of the RN, jointly averaged over SAC and TD3.
Figure 6: Per-task Δ AUC% as a function of the Repetition Number (RN), from RN0 (no repetition) to RN7, for SAC and TD3.
Figure 7: Impact of repetition number RN on IER-SAC performance across all tasks, showing how repeated execution of the highest-reward episode affects learning. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity.
Figure 8: Impact of repetition number RN on IER-TD3 performance across all tasks, showing how repeated execution of the highest-reward episode affects learning. Curves show the mean episodic return over ten independent seeds and are smoothed using a sliding window of size 5 for visual clarity.
Figure 9: tasks used in our experiments.
Figure 10: Mean Δ AUC% across environment groups. Left: MuJoCo tasks. Right: DeepMind Control tasks.
Task
Agreement RN
Selection Type
Ant-v4
RN1
Intersection
Cheetah-Run
RN3
Intersection
Walker-Walk
RN5
Intersection
Cartpole-Swingup
RN2
Intersection
HalfCheetah-v4
RN4
Rank-based
Hopper-v4
RN2
Rank-based
Table 1: Per-task RN agreement based on 95% near-best criterion.
Figure 11: Frequency of RN values selected as robust joint candidates across tasks. RN2, RN3, and RN5 appear most frequently.
Environment
Suite
Task Type
Description
Approx. Difficulty
Ant-v4
MuJoCo
Locomotion
Quadruped; coordination-heavy with complex contact dynamics.
High
HalfCheetah-v4
MuJoCo
Locomotion
Planar biped; dense unbounded rewards support long-term learning.
Medium
Humanoid-v4
MuJoCo
Locomotion
Biped with high DOF; unstable dynamics and sensitive to control noise.
High
Hopper-v4
MuJoCo
Locomotion
Single-legged hopping; requires balance and precise control, with early termination on failure.
Medium
Walker-Walk
DMC
Locomotion
Bipedal walking; fewer DOF than Humanoid, but still unstable.
Medium
Cheetah-Run
DMC
Locomotion
Fast planar cheetah; velocity-based control under bounded rewards.
Low
Table 2: Summary of environments used in evaluation.
Environment
Observation Space
Action Space
Horizon
MuJoCo (vector-based)
Ant-v4
R111
[−1,1]8
1000 (may terminate early due to instability)
HalfCheetah-v4
R17
[−1,1]6
1000
Humanoid-v4
R376
[−1,1]17
1000 (may terminate early due to instability)
Hopper-v4
R11
[−1,1]3
1000
DeepMind Control Suite (vector-based)
Table 3: Observation and action space dimensions and episode horizon for each environment. Rd denotes a d -dimensional real vector; [−1,1]d denotes a bounded action space.
Feature
Description
Task Overview
Task Type
Dynamic cube translation with robotic gripper
Object
Cube tracked via ArUco markers
Goal
Translate cube as far as possible within camera view
Figure 12: Workspace boundaries and cube tracking for reward evaluation. Green and red markers indicate current and previous cube positions used to compute displacement-based rewards.
Parameter
Value (Description)
Shared Parameters
batch_size
256 (Mini-batch size)
buffer_size
106 (Replay buffer size)
max_steps_exploration
103 (Exploration steps before training)
max_steps_training
106 (Total training steps)
number_steps_per_evaluation
104 (Steps between evaluations)
Table 5: Hyperparameters used in training. Shared parameters apply to both SAC and TD3; SAC, TD3, and SIL sections list algorithm-specific settings.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.
Kaiyan Zhao, Yiming Wang, Yuyang Chen +3
School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China
Long-horizon policies trained with reinforcement learning can still achieve high return through inefficient interactions, while rare efficient behaviors discovered during training may be forgotten. We argue that temporal efficiency itself provides a source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 30 long-horizon tasks spanning robot manipulation and interactive navigation, TSIL consistently improves task success rates, learning efficiency, behavioral efficiency, and robustness to unstable training conditions.
Reinforcement Learning (RL) agents often struggle with inefficient exploration, particularly in environments with sparse rewards. Traditional exploration strategies can lead to slow learning and suboptimal performance because agents fail to systematically build on previously successful experiences, thereby reducing sample efficiency. To tackle this issue, we propose a self-imitating on-policy algorithm that enhances exploration and sample efficiency by leveraging past high-reward state-action pairs to guide policy updates. Our method incorporates self-imitation by using optimal transport distance in dense reward environments to prioritize state visitation distributions that match the most rewarding trajectory. In sparse-reward environments, we uniformly replay successful self-encountered trajectories to facilitate structured exploration. Experimental results across diverse environments demonstrate substantial improvements in learning efficiency, including MuJoCo for dense rewards and the partially observable 3D Animal-AI Olympics and multi-goal PointMaze for sparse rewards. Our approach achieves faster convergence and significantly higher success rates compared to state-of-the-art self-imitating RL baselines. These findings underscore the potential of self-imitation as a robust strategy for enhancing exploration in RL, with applicability to more complex tasks.