Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics

Bozhou Chen; Yongyi Wang; Hanyu Liu; Xionghui Yang; Wenxin Li

cs.LGAug 17, 2026

Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics

Authors: Bozhou Chen, Yongyi Wang, Hanyu Liu, Xionghui Yang, Wenxin Li

Organizations: School of Computer Science, Peking University, Beijing, China

Abstract

Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.

Figures & tables

Fig. 1 : Reward-triggered self-reinforcing bias in Atari Alien . The dashed blue curve (left axis) shows the recent 10-episode mean return during training, while the solid red curve (right axis) reports the minibatch statistic $\mathrm{Count}\!\left(a=\arg\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\right)$ , the number of sampled transitions whose greedy bootstrap action at the next state coincides with the original action. The action space contains 18 actions and the minibatch size is 32. The consistently elevated counts indicate a strong coupling between $a$ and the bootstrap action $a^{\prime*}$ , providing empirical evidence for the self-reinforcing trap discussed in this section.

Fig. 2 : Layer-wise spike ratio evolution during training under different replay ratios. Each row corresponds to a network layer, and color intensity indicates spike magnitude. Higher replay ratios lead to more pronounced spike growth, particularly in post-encoder MLP layers. Further details are provided in the supplementary materials.

Algorithm	#Human	#Best	Mean	Median	IQM
DER	2	0	0.350	0.189	0.183
DrQ	3	0	0.465	0.313	0.280
IRIS	9	0	1.046	0.289	0.501
REM	12	1	1.222	0.280	0.673
STORM	10	5	1.266	0.580	0.636
DreamerV3	9	1	1.120	0.466	0.490

TABLE I: Aggregate performance on Atari-100K across 26 games. All metrics are computed using human-normalized scores (HNS). Mean, Median, and Interquartile Mean (IQM) are reported following standard evaluation practice. #Human denotes the number of games surpassing human-level performance, and #Best indicates the number of per-game best results. Higher is better for all metrics.

	Test Mean	Test Median	Test IQM
PPO (Impala CNN)	0.33	0.38	0.27
Ours	0.40	0.40	0.39

TABLE II : Procgen (easy difficulty, 200 training levels). Scores are normalized using min-max normalization as defined in the original Procgen paper.

Fig. 3 : Training performance under different ensemble sizes $N_{e}$ on four Atari-100K environments. Curves show the recent 10-episode mean return; shaded regions denote 95% CIs.

Fig. 4 : Normalized final performance as a function of the ensemble size. Results are normalized within each environment.

Fig. 5 : Comparison of training dynamics with and without the Action-Decoupling Constraint. For each environment, the left axis shows the recent 10-episode mean return, and the right axis shows the average Q-value. Solid lines correspond to the constrained version, while dashed lines denote the unconstrained variant.

Fig. 6 : Effect of parameter resetting on training performance. Solid lines denote the version with parameter reset, while dashed lines indicate the variant without resetting. The right axis shows the frequency of reset events during training.

Fig. 7 : Training performance under different replay ratios on four Atari-100K environments. Curves show the recent 10-episode mean return; shaded regions denote 95% CIs.

Fig. 8 : Normalized final performance as a function of replay ratio. Results are normalized within each environment.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9 : Layer-wise spike ratio evolution during training under different replay ratios. Each row corresponds to a network layer, and color intensity indicates spike magnitude.

Layer Group	Layer Name	Input Shape	Weight Shape
Stem	stem.0	$4\times 84\times 84$	$(8s)\times 4\times 3\times 3$
Stage1	l1.0.conv1	$(8s)\times 84\times 84$	$(8s)\times(8s)\times 3\times 3$
	l1.0.conv2	$(8s)\times 84\times 84$	$(8s)\times(8s)\times 3\times 3$
	l1.1.conv1	$(8s)\times 84\times 84$	$(8s)\times(8s)\times 3\times 3$
	l1.1.conv2	$(8s)\times 84\times 84$	$(8s)\times(8s)\times 3\times 3$
Stage2	l2.0.conv1	$(8s)\times 84\times 84$	$(16s)\times(8s)\times 3\times 3$

Appendix

TABLE III : Network layers monitored for spike ratio statistics in Figure 9 . The spike ratio is computed for each layer every 1000 environment steps to analyze how parameter imbalance evolves during training. The encoder’s channel width is controlled by a scale factor

s

(with

s=4

in our experiments).

Training Configuration (Atari-100K)
Total environment steps	$100{,}000$
Discount factor $\gamma$	$0.99$
Learning rate	$1\times 10^{-4}$ (Adam)
Batch size	$32$
Replay buffer size	$100{,}000$
Start learning after steps	$2{,}000$

Appendix

TABLE IV : Hyperparameter configuration used for Atari-100K experiments.

Total environment steps	25M
Training levels	200
Evaluation levels	Unseen procedural levels
Parallel environments	128
Batch size	256
Discount factor $\gamma$	0.99
Optimizer	Adam

Appendix

TABLE VI : Hyperparameters for Procgen (generalization mode, 200 training levels).

Environment	PPO (Impala CNN)	Ours
BigFish	3.0	18.1
BossFight	8.3	8.1
CaveFlyer	5.6	7.0
Chaser	5.5	1.9
Climber	6.1	7.5
CoinRun	8.8	6.0

Appendix

TABLE VII : Per-environment raw test performance on Procgen (easy, 200 training levels).

Fig. 10 : Per-environment train performance during training on Procgen.

RL Stability

See all 29 papers

Revisiting TD Target Aggregation under Uncertainty in Q-Learning
Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-Learning Value Function Estimation
Computer Science Department, Florida State University, Tallahassee, Florida, USA
SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
Sep 11, 2026Xiefeng Wu, Shu Zhang, Zhaojie Chu +1Offline RL Conservative RL
School of Computer Science, Wuhan University, Wuhan, China · School of Electronic Information, Wuhan University, Wuhan, China · School of Internet, Anhui University, Hefei, China
Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
May 7, 2026Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea FinnQ-Learning Temporal-Difference Learning
Stanford University

Aug 4, 2026J/K move · Enter open · S save

Revisiting TD Target Aggregation under Uncertainty in Q-Learning

Lipeng Zu, Xiaonan Zhang

Computer Science Department, Florida State University, Tallahassee, Florida, USA

Deep Q-Learning Value Function Estimation Q-Learning Temporal-Difference Learning

Open paper AskPDF arXiv

Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the Successor Rollout Aggregation Deep Q-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.

Abstract

RL Stability

Explore similar work

Revisiting TD Target Aggregation under Uncertainty in Q-Learning

SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities