Diversity-Aware Experience Replay via Statistical Leverage Scores

Shaifalee Saxena; Alexander Scheinker

cs.RONov 4, 2026

Diversity-Aware Experience Replay via Statistical Leverage Scores

Authors: Shaifalee Saxena, Alexander Scheinker

Organizations: Instrumentation & Controls Group at Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA. · Shaifalee Saxena is a PhD student at the University of New Mexico, Albuquerque, New Mexico, 87131, USA.

Abstract

Off-policy deep reinforcement learning trains on random samples from a replay buffer of past experience. In physical robot settings, that buffer is dominated by routine: thousands of near-identical transitions from normal operation, while the rare events that define the edges of the task are rarely sampled. We propose scoring every stored transition by its statistical leverage, which measures how unusual it is relative to the buffer's own data, computed from the stored numbers alone, with no labels, rewards, or value network. We draw each mini-batch from a mixture of this score and uniform sampling. First, the natural way to build the score along the directions where the data varies most turns out to be blind when actions are stored as one-hot indicators, because indicator columns always win the variance ranking; we instead select the feature columns with the heaviest tails (largest kurtosis) where small far-away populations are revealed. Second, the importance weights that prioritized replay uses to keep gradients unbiased cancel the method exactly: with full correction, training matches uniform sampling, so we drop the correction and deliberately optimize the rebalanced objective. Third, when the rare situations of a log are relational, visible only in how a stretch of behavior relates to its goal, not in any single stored coordinate-per-transition scores are blind; applying the same tail rule to short trajectory windows described relative to their goal restores the signal, and on a three-joint arm that must push an object around an obstacle it raises the rare-task success from 0.49 to 0.69 against a labeled-oracle ceiling of 0.74.

Figures & tables

Fig. 1: Overview of the proposed kurtosis-based leverage-mixture replay framework.

Fig. 2: Two-room gridworld. (a) The logged buffer at $\varepsilon=0.05$ : routine operation covers the main room ( $\approx 275$ samples/cell) while the corridor and annex region B, dashed, receive $\approx 12$ . (b) Ground-truth $Q^{*}$ with the optimal escape policy: the annex’s value structure follows its own geometry and cannot be inferred from region-A data.

Fig. 3: Sampling-share laws on two-cluster Gaussian buffers ( $N=2\times 10^{5}$ , $d=20$ , $s_{A}=s_{B}=s=1$ ): theory (lines) vs. simulation (markers). (a) Rare share under pure-leverage sampling; dashed: uniform sampling $(\epsilon)$ , dotted: full-inversion limit $(1-\epsilon)$ . (b) Rare batch fraction versus mixing weight $\beta$ at $\epsilon=0.02$ , for mixture draws (squares) and stratified batches (triangles). (c) Amplification over uniform sampling at $\beta=1/2$ ; dotted: theoretical ceiling $1+\Delta^{2}/(2s^{2})$ , dashed: naive $1/(2\epsilon)$ .

Fig. 4: Effect of rare cluster width ( $s_{A}{=}1$ ). (a) Rare share of pure-leverage draws at $\Delta{=}5$ for $s_{B}/s_{A}$ from $0.1$ to $3$ , divided by the equal-width value: lines ( 7 ). (b) Fraction of rare points whose score falls below the $99$ th percentile of the common cluster’s scores ( $\varepsilon{=}0.01$ ).

Fig. 5: Leverage behavior in higher-dimensional buffers. (a) Multi-cluster leverage mass $M_{c}$ versus the law $1-\pi_{c}$ in (9). (b) Rare share under top- $k$ PCA leverage when the separating direction has variance rank $r=8$ ; lines show (10) and leverage along the separating direction alone. (c) Rare-cluster amplification under top-4 PCA leverage over $(\epsilon,\Delta)$ ; white curve shows the predicted boundary (11)

		Q-RMSE
Condition	Final succ. B	B	A
Uniform	$0.882\pm 0.018$	3.33	0.31
Leverage ( $\beta=0.5$ )	$0.941\pm 0.010$	2.42	0.36
Leverage ( $\beta=0.75$ )	$0.928\pm 0.021$	1.43	0.38
Tail-flat ( $\beta=0.5$ )	$0.978\pm 0.007$	1.63	0.57
Tail-flat ( $\beta=0.75$ )	$\textbf{0.989}\pm\textbf{0.006}$	1.58	0.87

TABLE I: Success rate on task B and Q-value RMSE on tasks B and A under each sampling condition (mean

\pm

std.) over 20 seeds.

Fig. 6: Success rate for each method for regions A and B as a function of training steps of

Q_{\theta}(\mathbf{s},a)

are shown for the various methods. All methods quickly succeed in the common region A (opaque curves), and their performance separates in the rare/difficult region B with our proposed methods all outperforming PER and uniform sampling.

	Orbit		Batch
Condition	(last-3)	Straight	orbit share
Uniform	0.49	1.00	0.069
Window tail-flat ( $b=0.5$ )	0.69	1.00	0.220
Leverage ( $\beta=0.5$ )	0.51	1.00	0.110
Leverage ( $\beta=0.75$ )	0.44	1.00	0.131
Tail-flat ( $\beta=0.5$ )	0.45	1.00	0.098

TABLE II: TD3+BC for 20 seeds & 10k gradient steps. Orbit performance is averaged over the last three evaluations.

Fig. 7: The dashed circles show the annulus from which the object start

o_{0}

and goal

g

are drawn. (a) For straight tasks the line of sight

o_{0}\!\to\!g

clears the obstacle. (b) For orbit tasks the goal lies behind the obstacle and the object must be pushed around it. Solid curves are logged object paths (scripted controller with exploration noise), dotted curves the tool-tip path.

Fig. 8: Comparison of the various approaches for the 3-joint robot arm.

Off-Policy RL

Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.

See all 132 papers

Jul 27, 2026

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected

n

-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

Parham Mohammad Panahi, Armin Ashrafi, Haoyu Du +3

Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Deep Q-Learning Experience Replay

Ask PDF

Oct 27, 2024

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Experience replay is widely used to improve learning efficiency in reinforcement learning by leveraging past experiences. However, existing experience replay methods, whether based on uniform or prioritized sampling, often suffer from low efficiency, particularly in real-world scenarios with high-dimensional state spaces. To address this limitation, we propose a novel approach, Efficient Diversity-based Experience Replay (EDER). EDER employs a determinantal point process to model the diversity between samples and prioritizes replay based on the diversity between samples. To further enhance learning efficiency, we incorporate Cholesky decomposition for handling large state spaces in realistic environments. Additionally, rejection sampling is applied to select samples with higher diversity, thereby improving overall learning efficacy. Extensive experiments are conducted on robotic manipulation tasks in MuJoCo, Atari games, and realistic indoor environments in Habitat. The results demonstrate that our approach not only significantly improves learning efficiency but also achieves superior performance in high-dimensional, realistic environments.

Kaiyan Zhao, Yiming Wang, Yuyang Chen +3

School of Computer Science, Wuhan University, Wuhan, China · State Key Laboratory of Internet of Things for Smart City, University of Macau, Macao, China · School of Artificial Intelligence Shenzhen Polytechnic University, China

Experience Replay Determinantal Point Process Robotic RL Sample-Efficient RL

Ask PDF

May 11, 2026

When Does Non-Uniform Replay Matter in Reinforcement Learning?

Modern off-policy reinforcement learning algorithms often rely on simple uniform replay sampling and it remains unclear when and why non-uniform replay improves over this strong baseline. Across diverse RL settings, we show that the effectiveness of non-uniform replay is governed by three factors: replay volume, the number of replayed transitions per environment step; expected recency, how recent sampled transitions are; and the entropy of the replay sampling distribution. Our main contribution is clarifying when non-uniform replay is beneficial and providing practical guidance for replay design in modern off-policy RL. Namely, we find that non-uniform replay is most beneficial when replay volume is low, and that high-entropy sampling is important even at comparable expected recency. Motivated by these findings, we adopt a simple Truncated Geometric replay that biases sampling toward recent experience while preserving high entropy and incurring negligible computational overhead. Across large-scale parallel simulation, single-task, and multi-task settings, including three modern algorithms evaluated on five RL benchmark suites, this replay sampling strategy improves sample efficiency in low-volume regimes while remaining competitive when replay volume is high.

Michal Korniak, Mikołaj Czarnecki, Yarden As +3

1ETH Zurich · University of Warsaw · 3UC Berkeley +1

Experience Replay Sample-Efficient RL Off-Policy RL

Ask PDF

Abstract

Off-Policy RL

Explore similar work

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

When Does Non-Uniform Replay Matter in Reinforcement Learning?