Authors: Prabhat Nagarajan, Martha White, Marlos C. Machado
Organizations: Department of Computing Science, University of Alberta, Alberta Machine Intelligence Institute, 1 Canada CIFAR AI Chair, Edmonton, AB, Canada
Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection from action-evaluation when computing bootstrap targets. Double DQN adapts this decoupling to deep reinforcement learning (RL) but directly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through the Double Q-learning objective. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. We study key design choices of DDQL, including its network architecture, replay ratio, and minibatch sampling strategies.
Figures & tables
Figure 1: DDQL architectures . a) DQN/Double DQN, b) DDQL, c) DM-DDQL, and d) DN-DDQL.
Double Q-learning Defining Feature
Double DQN
DDQL & DM-DDQL
DN-DDQL
(1)
Target bootstrap decoupling
✓
✓
✓
(2)
Double estimation
-
✓
✓
(2a)
Exclusive parameters
-
✗
✓
(3)
Dataset partitioning
-
✓
✓
(3a)
Separate buffers
-
✗
✗
Table 1: Different adaptations of Double Q-learning to deep RL in terms of its defining features.
Figure 2: Per-game DDQL overestimation. Final overestimations averaged across five seeds of Double DQN and DDQL (Video Pinball is omitted for visibility).
Figure 3: Aggregate learning curves and per-game performance improvements. ( left ) Interquartile mean of the HNS throughout training. The shaded region represents a 95% stratified bootstrap confidence interval [ 2 ] . ( right ) Per-game improvement in HNS of DDQL over Double DQN, calculated as the average area under the curve across five seeds. The y-axis is on a log-scale.
Figure 4: Per-game overestimation for the DDQL variants. Final overestimations averaged across five seeds per algorithm on each of the 57 games. DDQL generally overestimates the most, followed by DM-DDQL, then DN-DDQL. The latter two often underestimate.
Figure 5: Dataset partitioning experiments. ( a ) The IQM of the HNS on the Ablation-11 games of DDQL and its double-buffer counterparts. Shaded region: 95% stratified bootstrap confidence interval. ( b & c ) Final overestimation, averaged over five seeds of DDQL and DN-DDQL, respectively, and their double-buffer counterparts. ( d ) Overestimation throughout training for different algorithms averaged across five seeds on Space Invaders. Translucent lines depict individual seeds.
Figure 6: Replay Ratio Experiments. ( left ) IQM of the HNS across five seeds on the Ablation-11 games. Shaded region: 95% stratified bootstrap confidence interval. Increasing the replay ratio harms DDQL’s performance. ( right ) Compares the final overestimation of DDQL with a replay ratio of \nicefrac14 to Double DQN, averaged over five seeds, on the Ablation-11 games.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Detail
Setting
Description
sticky action probability
0.25
Probability by which the simulator ignores the agent’s selected action and repeats the action executed in the previous frame.
termination criterion
end-of-game
How episode termination is signaled during training (either end-of-game or loss-of-life).
action space
full
The full action space has 18 actions. The minimal action space uses the game-specific minimum required actions.
Appendix
Table 2: Environment details, following Machado et al. [21] .
Hyperparameter
Value
Description
minibatch size
32
Number of transitions used per update per Q-function.
replay memory size
1,000,000
Number of transitions stored in the replay buffer.
agent history length
4
Number of previous frames stacked in state representation.
target network update frequency
7,500
Frequency (in terms of parameter updates) of target network updates
discount factor
0.99
Value of γ used in the target computation.
action repeat
4
The number of simulator frames for which an action is repeated in a single timestep.
Appendix
Table 3: Hyperparameters and training details.
Detail
Value
Description
evaluation ϵ
0.001
The ϵ used for the ϵ -greedy policy used during evaluation.
maximum episode length
27,000
Number of timesteps in evaluation episodes. Corresponds to 30 minutes of gameplay. 3 3 3 (30 mins = 30 minutes * 60 seconds/minute * 60 frames/second / 4 frames per timestep = 27,000 timesteps)
evaluation phase length
125,000
Length of periodic evaluation phases in terms of number of timesteps.
evaluation frequency
250,000
Frequency (in terms of training timesteps) of evaluation phase.
Appendix
Table 4: Evaluation details.
Algorithm
Qeval
Qsel
Double DQN
θ−
θ
Inverse Double DQN
θ
θ−
Appendix
Table 5: Single network target bootstrap decoupling.
Algorithm
θeval
θsel
DDQLDQN
θ2−
θ1−
DDQLDouble DQN
θ2−
θ1
DDQLInverse
θ2
θ1−
DDQLNo target
θ2
θ1
Appendix
Table 6: Options for target bootstrap decoupling with reciprocal bootstrapping in DDQL.
Figure 7: A comparison of DN-DDQL with a short target network update interval to DN-DDQL with a short target network update interval and non-identical initialization. Identical initialization can help performance.
Figure 8: Performance of DDQL compared to DDQLDouble DQN . The algorithms perform at a similar level, with DDQL performing better in two environments.
Figure 9: Performance of DN-DDQL compared to DN-DDQLDouble DQN . DN-DDQL performs more stably.
Figure 10: Performance of DDQL compared to DDQL (double buffer). The algorithms largely perform at a similar level, with a few exceptions.
Figure 11: Performance of DN-DDQL compared to DN-DDQL (double buffer). DN-DDQL (double buffer) performs quite poorly.
Figure 12: Overestimation of DDQL compared to DDQL (double buffer). The double buffer strategy reduces overestimation.
Figure 13: Performance of DDQL compared to DDQL(RR=\nicefrac14) . They perform similarly in some environments, but DDQL(RR=\nicefrac14) is generally worse.
Figure 14: Performance of DN-DDQL compared to DN-DDQL(RR=\nicefrac14) . DN-DDQL(RR=\nicefrac14) is generally worse.
Figure 15: Overestimation of DDQL(RR=\nicefrac14) , DN-DDQL(RR=\nicefrac14) , and Double DQN. Overestimation is clipped at −8 due to divergence in BattleZone . The DDQL variants continue to reduce overestimation even with double the replay ratio.
Figure 16: Interquartile mean of the HNS throughout training on 57 Atari environments. The shaded region represents a 95% stratified bootstrap confidence interval. Double DQN(RR=\nicefrac18) is run for three seeds per game and the other algorithms are run for five seeds per game.
Figure 23
Figure 17: Overestimation over 50M timesteps for 57 Atari 2600 games. For readability, overestimation is bounded below at −20 and above at 25. Plots are smoothed with a moving average of five. The semitransparent curves are the individual seeds.
Figure 18: Per-game performance improvements of DN-DDQL. Per-game improvements in HNS of DN-DDQL over Double DQN in each of 57 games, calculated as the average area under the curve across 5 seeds. The y-axis is log-scale.
Figure 26
Figure 19: Scores across 50M timesteps across 57 Atari 2600 games. Plots are smoothed with a moving average of seven. The semitransparent curves are the individual seeds.
Environment
Double DQN
DDQL
DM-DDQL
DN-DDQL
Alien
2805.1
4314.7
3483.1
1811.0
Amidar
508.9
922.1
600.9
366.6
Assault
1888.3
1870.4
2290.9
2462.5
Asterix
13371.6
22896.3
26851.5
27554.4
Asteroids
1172.4
1221.9
1176.6
1071.7
Atlantis
788923.3
782803.3
827005.0
852533.3
Appendix
Table 7: The mean evaluation score across the last three evaluations during training for different algorithms. The highest scores for an environment are bolded.
Department of Computing Science, University of Alberta, Edmonton, AB, Canada. · Alberta Machine Intelligence Institute · Meta (independent work, not on Meta’s behalf). +1