cs.LGMar 31, 2026

Target-Aligned Reinforcement Learning

Authors: Leonard S. Pleiss, James Harrison, Maximilian Schiffer

Organizations: Google DeepMind

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many value-based deep reinforcement learning algorithms rely on target networks - lagged copies of the online network - to stabilize training. While effective, this mechanism introduces a fundamental stability-recency tradeoff: slower target updates improve stability but reduce the recency of learning signals, hindering convergence speed. We propose Target-Aligned Reinforcement Learning (TARL), a simple drop-in refinement for existing algorithms that emphasizes transitions for which the target and online network estimates are highly aligned. By focusing updates on well-aligned targets, TARL mitigates the adverse effects of stale target estimates while retaining the stabilizing benefits of target networks. We empirically demonstrate consistent improvements within discrete and continuous control algorithms across various benchmark environments without any hyperparameter tuning, including a 38.18% peak score gain on Atari-10, while incurring less than a 4% increase in wall-clock time.

Figures & tables

Figure 1: Stylized visualization of alignment scenarios. We distinguish between updates fully supported by the online network (scenario I & II) and updates that are only partially or not at all supported by the online network (scenario III & IV). We posit that update types I and II are safer than update types III and IV.
Algorithm 1 Target-Aligned Deep Q-Network (TA-DQN)
Figure 2: Comparison of DDQN, Target-aligned DDQN (TA-DDQN), and DDQN with doubled batch size across the Atari-10 benchmark. (Left) Per-game peak score improvement of TA-DDQN over DDQN (%). (Center-left) Median normalized cumulative maximum score aggregated over all games as a function of environment frames. (Center-right) Median normalized score over training, reflecting instantaneous rather than best-seen performance. (Right) Mean wall-clock runtime for DDQN and TA-DDQN. Score normalization per game is performed using the range between the random baseline and the best observed performance across all conditions. Score curves are smoothed over 10 evaluations.
Figure 3: Performance comparison between SAC and target-aligned SAC on six continuous control environments over five seeds. Curves represent median returns, smoothed over 10 evaluations, and shaded areas indicate interquartile ranges.
Figure 4: Performance comparison between DQN and target-aligned DQN under different experience replay strategies on four games of the MinAtar benchmark over five seeds. Curves represent median returns, smoothed over 10 evaluations.
Figure 5: Impact of the oversampling margin on performance and batch alignment. Results are aggregated across four MinAtar environments. The subplots compare standard DQN and a DQN with doubled batch size—denoted as DQN (2m)—against TA-DQN , evaluated at oversampling margins ( b∈{0.1m,0.5m,1.0m,3.0m}b\in\{0.1m,0.5m,1.0m,3.0m\} ). All curves are smoothed over 10 evaluations. (Left) Median normalized evaluation scores. Scores were min-max normalized per environment prior to aggregation. (Center) Average batch alignment post-discard. (Right) Minimum batch alignment post-discard.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Table 7
EnvironmentDDQNTA-DDQNGain (%)
Amidar0.3250.414+27.4
BattleZone0.0880.270+205.8
Bowling0.0350.193+446.5
DoubleDunk0.1080.296+175.0
Frostbite0.0770.338+337.0
KungFuMaster0.2390.298+24.7
Appendix
Table 4: Normalized Area Under the Curve for Atari-10, standard deviation indicated as ±\pm .
Figure 6: Per-game performance comparison between DDQN and TA-DDQN on the Atari-10 benchmark. Curves represent returns, smoothed over 10 evaluations.
Figure 7: Per-seed performance comparison between SAC and TA-SAC on six continuous control environments. Curves represent returns, smoothed over 10 evaluations.
Figure 8: Wall-clock training time for SAC and TA-SAC on six continuous control environments.
Figure 9: Per-seed performance comparison between DQN and target-aligned TA-DQN on four games of the MinAtar benchmark. Curves represent returns, smoothed over 10 evaluations.
Figure 10: Per-seed performance comparison between DQN with PER and TA-DQN with PER on four games of the MinAtar benchmark. Curves represent returns, smoothed over 10 evaluations.
Figure 11: Per-seed performance comparison between DQN with ReaPER and TA-DQN with ReaPER on four games of the MinAtar benchmark. Curves represent returns, smoothed over 10 evaluations.
Figure 12: Performance by oversampling margin ( bb = oversampling margin, mm = batch size). Curves represent returns, smoothed over 10 evaluations.
Figure 13: Mean batch alignment by oversampling margin ( bb = oversampling margin, mm = batch size).
Figure 14: Minimum batch alignment by oversampling margin ( bb = oversampling margin, mm = batch size).
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Explore similar work

CardsList

  1. Streaming Deep Reinforcement Learning Finally Works

    Oct 18, 2024Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1Reinforcement LearningActor-Critic Methods

    Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

  2. Score Centering Stabilizes Off-policy Reinforcement Learning

    Sep 17, 2026Martin Marek, Max RyabininRL for Language ModelsOff-Policy RL

    Together AI

  3. Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

    Sep 23, 2026Jiaxi Wu, Tiantian Zhang, Yuxing Wang +2Reinforcement LearningRisk-Sensitive RL

    Shenzhen International Graduate School, Tsinghua University Shenzhen, China