Temporal-Difference Learning

Overview

Temporal-difference (TD) learning studies how to estimate future rewards and improve decisions by updating predictions from the difference between a current estimate and a bootstrapped next-step target. Its methods range from tabular and linear TD with eligibility traces to deep Q-learning, distributional critics, and multi-step or off-policy updates, used in prediction, control, and policy evaluation. Current work develops sharper convergence and uncertainty guarantees for single-trajectory, Markovian data while extending TD to long-horizon, multi-agent, and partially observed settings. A central tension is that bootstrapping makes learning data-efficient, but function approximation, off-policy sampling, and moving targets can cause bias or instability; balancing stability, speed, and accurate credit assignment remains open.

Latest papers

CardsList

93 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. Temporal difference learning adapts to incomplete coverage: an oracle inequality over subgraphs

    Nov 14, 2024Wenlong Mou, Jian QianTemporal-Difference LearningReinforcement Learning

  2. Statistical Inference for Policy Evaluation with Temporal Difference Learning

    Oct 21, 2024Weichen Wu, Gen Li, Yuting Wei +1Parameter EstimationConfidence Region Estimation