Temporal-Difference Learning

Follow this topic Ask about this topic

Temporal-difference (TD) learning studies how to estimate future rewards and improve decisions by updating predictions from the difference between a current estimate and a bootstrapped next-step target. Its methods range from tabular and linear TD with eligibility traces to deep Q-learning, distributional critics, and multi-step or off-policy updates, used in prediction, control, and policy evaluation. Current work develops sharper convergence and uncertainty guarantees for single-trajectory, Markovian data while extending TD to long-horizon, multi-agent, and partially observed settings. A central tension is that bootstrapping makes learning data-efficient, but function approximation, off-policy sampling, and moving targets can cause bias or instability; balancing stability, speed, and accurate credit assignment remains open.

Latest papers

82 papers13 papers in the last four weeks, up 86% on the four weeks before. 0.1% of all new papers.

Extending Differential Temporal Difference Methods for Episodic Problems
May 6, 2026Kris De Asis, Mohamed Elsayed, Jiamin HeReinforcement Learning Temporal-Difference Learning
1Openmind Research Institute · Department of Computing Science, University of Alberta, Canada · 3Alberta Machine Intelligence Institute (Amii)
Bridging the Gap Between Average and Discounted TD Learning
May 3, 2026Haoxing Tian, Zaiwei Chen, Ioannis Ch. Paschalidis +1Reinforcement Learning Average-Reward RL
Department of Electrical and Computer Engineering Boston University · Edwardson School of Industrial Engineering Purdue University
AdamO: A Collapse-Suppressed Optimizer for Offline RL
May 3, 2026Nan Qiao, Sheng Yue, Shuning Wang +1Reinforcement Learning Orthogonal Optimization
1Tsinghua University, China · 2Central South University, China · 3Sun Yat-sen University, China
Regularized Centered Emphatic Temporal Difference Learning
May 2, 2026Xingguo Chen, Chaohui Wu, Jinguo Ye +5Temporal-Difference Learning
Nanjing University of Posts and Telecommunications · Nanjing University · Microsoft +1
Bayesian policy gradient and actor-critic algorithms
Apr 30, 2026Mohammad Ghavamzadeh, Yaakov Engel, Michal ValkoBayesian RL Temporal-Difference Learning
Adobe Research & Inria · Rafael Advanced Defence System, Israel · Inria Lille — SequeL team, France
Intentional Updates for Streaming Reinforcement Learning
Apr 21, 2026Arsalan Sharifnassab, Mohamed Elsayed, Kris De Asis +2Reinforcement Learning Temporal-Difference Learning
Openmind Research Institute · Department of Computing Science, University of Alberta.
Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error
Apr 2, 2026Taisuke KobayashiTemporal-Difference Learning Robust RL
Principles of Informatics Research Division, National Institute of Informatics (NII), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan · Informatics Program, Graduate Institute for Advanced Studies (SOKENDAI), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan
Full-Gradient Successor Feature Representations
Apr 1, 2026Ritish Shrirao, Aditya Priyadarshi, Raghuram Bharadwaj DiddigiReinforcement Learning Successor Representations
Dept. of CSE, IIIT Bangalore · Dept. of DSAI, IIIT Bangalore
Stability and Sensitivity Analysis of Relative Temporal-Difference Learning: Extended Version
Mar 29, 2026Masoud S. Sakha, Rushikesh Kamalapurkar, Sean MeynTemporal-Difference Learning
Deep Reinforcement Learning and The Tale of Two Temporal Difference Errors
Mar 23, 2026Juan Sebastian Rojas, Chi-Guhn LeeTemporal-Difference Learning
University of Toronto, Canada
Chunk-Guided Q-Learning
Mar 14, 2026Gwanwoo Song, Kwanyoung Park, Youngwoon LeeReinforcement Learning Q-Learning
Department of Artificial Intelligence, Yonsei University · UC Berkeley
Gradient Iterated Temporal-Difference Learning
Mar 8, 2026Théo Vincent, Kevin Gerhardt, Yogesh Tripathi +5Temporal-Difference Learning
DFKI · TU Darmstadt · Alberta Machine Intelligence Institute (Amii) +4
What Does Flow Matching Bring To TD Learning?
Mar 4, 2026Bhavya Agrawalla, Michal Nauman, Aviral KumarReinforcement Learning Value Function Estimation
Carnegie Mellon University · University of Warsaw
Towards Parameter-Free Temporal Difference Learning
Mar 3, 2026Yunxiang Li, Mark Schmidt, Reza Babanezhad +1Temporal-Difference Learning
University of British Columbia · Samsung AI - Montreal · Simon Fraser University
Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation
Mar 2, 2026Leo Muxing Wang, Pengkun Yang, Lili SuTemporal-Difference Learning Multi-Agent Reinforcement Learning
Northeastern University · Tsinghua University
Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
Feb 18, 2026Ethan Blaser, Jiuqi Wang, Shangtong ZhangTemporal-Difference Learning Average-Reward RL
University of Virginia
Finite-Sample Wasserstein Error Bounds and Concentration Inequalities for Nonlinear Stochastic Approximation
Feb 2, 2026Seo Taek Kong, R. SrikantStochastic Approximation Stochastic Gradient Descent
University of Illinois Urbana-Champaign
Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning
Jun 6, 2025Adrian Ly, Richard Dazeley, Peter Vamplew +2Reinforcement Learning Deep Q-Learning
Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis
Mar 24, 2025Mohsen Amiri, Sindri MagnússonNon-Stationary RL Markov Decision Processes
Department of Computer and System Science, Stockholm University, SE-164 25 Stockholm, Sweden
Temporal difference learning adapts to incomplete coverage: an oracle inequality over subgraphs
Nov 14, 2024Wenlong Mou, Jian QianTemporal-Difference Learning Reinforcement Learning
Department of Statistical Sciences, University of Toronto · Department of EECS, Massachusetts Institute of Technology
Statistical Inference for Policy Evaluation with Temporal Difference Learning
Oct 21, 2024Weichen Wu, Gen Li, Yuting Wei +1Parameter Estimation Confidence Region Estimation
The Voleon Group, New York, NY, USA · Department of Statistics, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. +1
Deep End-to-End Survival Analysis with Temporal Consistency
Oct 9, 2024Mariana Vargas Vieyra, Pascal FrossardSurvival Analysis Temporal Consistency
EPFL

May 6, 2026J/K move · Enter open · S save

Extending Differential Temporal Difference Methods for Episodic Problems

Kris De Asis, Mohamed Elsayed, Jiamin He

1Openmind Research Institute · Department of Computing Science, University of Alberta, Canada · 3Alberta Machine Intelligence Institute (Amii)

Reinforcement Learning Temporal-Difference Learning

Open paper AskPDF arXiv

Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function's state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming reinforcement learning algorithms to their differential counterparts. Across a range of base algorithms and environments, we empirically validate that reward centering can improve sample efficiency in episodic problems.