Temporal-Difference Learning

Overview

Temporal-difference (TD) learning studies how to estimate future rewards and improve decisions by updating predictions from the difference between a current estimate and a bootstrapped next-step target. Its methods range from tabular and linear TD with eligibility traces to deep Q-learning, distributional critics, and multi-step or off-policy updates, used in prediction, control, and policy evaluation. Current work develops sharper convergence and uncertainty guarantees for single-trajectory, Markovian data while extending TD to long-horizon, multi-agent, and partially observed settings. A central tension is that bootstrapping makes learning data-efficient, but function approximation, off-policy sampling, and moving targets can cause bias or instability; balancing stability, speed, and accurate credit assignment remains open.

Latest papers

CardsList

93 papers14 papers in the last four weeks, down 7% on the four weeks before. 0.1% of all new papers.

  1. MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

    Nov 18, 2026Seungyub Han, Taehyun Cho, Dohyeong Kim +2Temporal-Difference LearningOff-Policy RL

  2. Understanding the Asymptotic Transience of Q-Learning via Distributional Expansion

    Nov 11, 2026Shengbo WangQ-LearningTemporal-Difference Learning

  3. Off-policy Learning with Excursion Policies

    Nov 9, 2026Jiamin He, Mark Rowland, Zhaohan Daniel Guo +2Off-Policy RLOff-Policy Evaluation

  4. Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels

    Nov 9, 2026Sreejeet Maity, Aritra MitraPolicy EvaluationTemporal-Difference Learning

  5. Horizon-Scalable Q-Learning

    Nov 7, 2026Haoran Xu, Pragya Srivastava, Xinjie Liu +2Q-LearningTemporal-Difference Learning

  6. Finite-Time Convergence and Regret Guarantees for Neural Reinforcement Learning in POMDPs

    Nov 4, 2026Jingjuan Huang, Zhiyao Zhang, Alvaro Velasquez +2Partially Observable RLRegret Minimization in RL

  7. On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces

    Nov 4, 2026Jingyi Jin, Martha White, Hao WangTemporal-Difference LearningRL Stability

  8. Revisiting Critic Estimation in Soft Actor-Critic Algorithms

    Nov 3, 2026Jan Felix Kleuker, Koen Ponse, Aske Plaat +1Reinforcement LearningSoft Actor-Critic

  9. On the Feature Evolution of Temporal Difference Learning

    Nov 3, 2026Shangzhe Li, Dechen Zhang, Difan Zou +1Temporal-Difference LearningRepresentation Learning for RL

  10. Bridging Neural Tangent Kernel Analysis and Reinforcement Learning for Deep Function Approximation

    Nov 2, 2026William St-Arnaud, Margarida Carvalho, Golnoosh FarnadiNeural Tangent KernelNeural Network Training Dynamics

  11. Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy

    Oct 31, 2026Sunyoung An, Xiaoming HuoTemporal-Difference LearningPolicy Evaluation

  12. Temporal-Difference Calibration of Black-Box Simulators

    Oct 26, 2026Min Zeng, Xin Wen, Xi Chen +2Model CalibrationTemporal-Difference Learning

  13. A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory

    Oct 24, 2026Yanjie Dong, Zhou Su, Wei Wang +1Reinforcement LearningActor-Critic Methods

  14. Stochastic Tracking of Drifting Affine Fixed Points

    Oct 19, 2026Sajad KhodadadianStochastic ApproximationNon-Stationary RL

  15. Beyond Fixed Acquisition Functions: Temporal Difference Active Learning for Perturbation Discovery

    Oct 15, 2026Xudong Zhang, Jianzhe Li, Yitong Tang +1Active LearningAdaptive Sampling

  16. On the Convergence of Semi-Gradient TD: A Mean Field Perspective

    Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference LearningRL Stability

  17. Provable Transformer Context Compression for In-Context Reinforcement Learning

    Oct 14, 2026Yajie Zhao, Yingjun Mo, Zhiyong Wang +3In-Context RLIn-Context Learning

  18. Posterior-Averaged Bellman Flow Matching for Distributional Critics

    Oct 13, 2026Zeqiao Li, Yijing Wang, Haoyu Wang +2Distributional RLFlow Matching

  19. Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

    Oct 6, 2026Zijie Xu, Bingrui Guo, Yiding Sun +3Deep Q-LearningTemporal-Difference Learning

  20. Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning

    Oct 4, 2026Bingyun Liu, Yuheng JingHierarchical RLTemporal-Difference Learning

  21. Temporal-Difference Learning for Dragonchess

    Oct 1, 2026Jim O'Connor, Annika Hoag, Sarah Goyette +1Game-Playing AgentsTemporal-Difference Learning

  22. Fast Regularized Policy Mirror Descent with One-Step TD Updates

    Sep 30, 2026Qipei Chen, Wenye Li, Yule Sun +1Markov Decision ProcessesPolicy Optimization

  23. Near-Minimax-Optimal Rates for Asynchronous TD Learning with Markovian Data

    Sep 30, 2026Yang PengAsynchronous RLStochastic Approximation

  24. GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

    Sep 27, 2026Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman +1Reinforcement LearningTemporal-Difference Learning

  25. From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

    Sep 24, 2026Yichen Lin, Xuyuan Xiong, Xue Wang +3In-Context RLTransformer-Based RL

  26. A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning

    Sep 24, 2026Ids van der Werf, Sergio Rozada, Antonio G. MarquesStochastic ApproximationTemporal-Difference Learning

  27. Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning

    Sep 23, 2026James Wu, Chris R. SimsComputational Cognitive ModelingReinforcement Learning

  28. Limiting-Kernel Q(λλ): Bridging Short and Long Horizons

    Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function EstimationPolicy Evaluation