cs.LGMar 23, 2026

Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors

Authors: Juan Sebastian Rojas, Chi-Guhn Lee

Organizations: University of Toronto, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The temporal difference (TD) error was first formalized in Sutton (1988), where it was first characterized as the difference between temporally successive predictions, and later, in that same work, formulated as the difference between a bootstrapped target and a prediction. Since then, these two interpretations of the TD error have been used interchangeably in the literature, with the latter eventually being adopted as the standard critic loss in deep reinforcement learning (RL) architectures. In this work, we show that these two interpretations of the TD error are not always equivalent. In particular, we show that increasingly-nonlinear deep RL architectures can cause these interpretations of the TD error to yield increasingly different numerical values. Then, building on this insight, we show how choosing one interpretation of the TD error over the other can affect the performance of deep RL algorithms that utilize the TD error to compute other quantities, such as with deep differential (i.e., average-reward) RL methods. All in all, our results show that the default interpretation of the TD error as the difference between a bootstrapped target and a prediction does not always hold in deep RL settings.

Figures & tables

Figure 1: Rolling average of the TD error absolute difference (i.e., ∣δte−δti∣|\delta^{e}_{t}-\delta^{i}_{t}| ) as learning progresses when using a) a Q-learning algorithm in the inverted pendulum environment, and b) a DQN algorithm in the Atari Breakout environment. A solid line denotes the mean TD error absolute difference, and a shaded region denotes a 95% confidence interval over 4 runs.
Figure 2: Rolling average-reward estimates when using a deep differential Q-learning algorithm in the Breakout environment. Figures a) , b) , and c) utilize an explicit TD error average-reward update with an initial guess of -0.25, 0.0, and 0.25, respectively. Figures d) , e) , and f) utilize an implicit TD error update with an initial guess of -0.25, 0.0, and 0.25, respectively. A solid line denotes the mean average-reward estimate, and a shaded region denotes a 95% confidence interval over 4 runs.
Figure 3: Rolling total reward per episode when using: a) a differential Q-learning algorithm in the Breakout environment, b) a Q-learning algorithm with value-based reward centering in the Pong environment, and c) an A2C algorithm in the HalfCheetah environment. Each plot shows the performance of each algorithm when using the explicit vs. implicit TD error. A solid line denotes the mean total reward per episode, and a shaded region denotes a 95% confidence interval over 8 runs.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1 : An illustration of the a) inverted pendulum, and b) Atari Breakout environments.
LayerInput ShapeOutput ShapeCalculation FormulaParameters
conv14×84×844\times 84\times 8416×20×2016\times 20\times 20(16×4×8×8)+16(16\times 4\times 8\times 8)+164,112
conv216×20×2016\times 20\times 2032×9×932\times 9\times 9(32×16×4×4)+32(32\times 16\times 4\times 4)+328,224
conv332×9×932\times 9\times 932×7×732\times 7\times 7(32×32×3×3)+32(32\times 32\times 3\times 3)+329,248
fc11,568256(256×1,568)+256(256\times 1,568)+256401,664
fc22564(4×256)+4(4\times 256)+41,028
Total   424,276
Appendix
Table 1: Network Parameter Calculation for “Small” Network
LayerInput ShapeOutput ShapeCalculation FormulaParameters
conv14×84×844\times 84\times 8464×20×2064\times 20\times 20(64×4×8×8)+64(64\times 4\times 8\times 8)+6416,448
conv264×20×2064\times 20\times 20128×9×9128\times 9\times 9(128×64×4×4)+128(128\times 64\times 4\times 4)+128131,200
conv3128×9×9128\times 9\times 9128×7×7128\times 7\times 7(128×128×3×3)+128(128\times 128\times 3\times 3)+128147,584
fc16,2721,024(1,024×6,272)+1,024(1,024\times 6,272)+1,0246,423,552
fc21,0244(4×1,024)+4(4\times 1,024)+44,100
Total   6,722,884
Appendix
Table 2: Network Parameter Calculation for “Large” Network
Figure B.2 : Rolling total reward per episode when using a deep differential Q-learning algorithm in the Breakout environment with an initial average-reward guess of: a) -0.25, b) 0.0, and c) 0.25. Each plot shows the performance when using (batch-averaged) explicit vs. implicit TD error average-reward updates. A solid line denotes the mean total reward per episode, and a shaded region denotes a 95% confidence interval over 4 runs.
Figure B.3 : Illustration of the a) Atari Breakout, b) Atari Pong, and c) MuJoCo HalfCheetah tasks.
More in this topic

Temporal-Difference Learning

Temporal-difference learning estimates values by bootstrapping from later predictions, enabling online learning and control from sequential experience.

See all 80 papers

Explore similar work

CardsList

  1. Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error

    Apr 2, 2026Taisuke KobayashiTemporal-Difference LearningRobust RL

    Principles of Informatics Research Division, National Institute of Informatics (NII), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan · Informatics Program, Graduate Institute for Advanced Studies (SOKENDAI), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan

  2. A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

    Jun 16, 2026M. Forzo, E. Monzio Compagnoni, A. Russo +1Stochastic Differential EquationsTemporal-Difference Learning

    Technical University of Munich (TUM), Munich, Germany · University of Basel, Basel, Switzerland · Boston University, Boston, USA.

  3. Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning

    Jun 3, 2026Viktor Veselý, Aleksandar Todorov, Erwan Escudie +1Temporal Credit AssignmentTemporal-Difference Learning

    Department of AI, University of Groningen, The Netherlands