Temporal-Difference Learning

Follow this topic Ask about this topic

Temporal-difference (TD) learning studies how to estimate future rewards and improve decisions by updating predictions from the difference between a current estimate and a bootstrapped next-step target. Its methods range from tabular and linear TD with eligibility traces to deep Q-learning, distributional critics, and multi-step or off-policy updates, used in prediction, control, and policy evaluation. Current work develops sharper convergence and uncertainty guarantees for single-trajectory, Markovian data while extending TD to long-horizon, multi-agent, and partially observed settings. A central tension is that bootstrapping makes learning data-efficient, but function approximation, off-policy sampling, and moving targets can cause bias or instability; balancing stability, speed, and accurate credit assignment remains open.

Latest papers

82 papers13 papers in the last four weeks, up 86% on the four weeks before. 0.1% of all new papers.

Temporal-Difference Calibration of Black-Box Simulators
Oct 26, 2026Min Zeng, Xin Wen, Xi Chen +2Model Calibration Temporal-Difference Learning
College of Computing, City University of Hong Kong · Stern School of Business, New York University · Daniels School of Business, Purdue University +1
A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
Oct 24, 2026Yanjie Dong, Zhou Su, Wei Wang +1Reinforcement Learning Actor-Critic Methods
School of Cyber-Science and Engineering, Xi’an Jiaotong University · School of Information and Communications Engineering, Xi’an Jiaotong University · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology
Short-to-Long: Detached Features for Long-Horizon Value Learning
Oct 21, 2026Tong Che, Boris Ivanovic, Marco PavoneReinforcement Learning Long-Horizon RL
NVIDIA Research
Stochastic Tracking of Drifting Affine Fixed Points
Oct 19, 2026Sajad KhodadadianStochastic Approximation Non-Stationary RL
Grado Department of Industrial and Systems Engineering, Virginia Tech
Beyond Fixed Acquisition Functions: Temporal Difference Active Learning for Perturbation Discovery
Oct 15, 2026Xudong Zhang, Jianzhe Li, Yitong Tang +1Active Learning Adaptive Sampling
Center for Machine Learning Research, Peking University, Beijing, China · AI for Science Institute (AISI), Beijing, China
On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Oct 14, 2026Haruki Settai, Tadashi Kozuno, Shinji ItoTemporal-Difference Learning RL Stability
The University of Tokyo · Sun Intelligence · The University of Tokyo & RIKEN
Provable Transformer Context Compression for In-Context Reinforcement Learning
Oct 14, 2026Yajie Zhao, Yingjun Mo, Zhiyong Wang +3In-Context RL In-Context Learning
Southern University of Science and Technology · University of Electronic Science and Technology of China · Harbin Institute of Technology
Posterior-Averaged Bellman Flow Matching for Distributional Critics
Oct 13, 2026Zeqiao Li, Yijing Wang, Haoyu Wang +2Distributional RL Flow Matching
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China
Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks
Oct 6, 2026Zijie Xu, Bingrui Guo, Yiding Sun +3Deep Q-Learning Temporal-Difference Learning
Peking University Beijing, 100871, China
Fast Last-Iterate Convergence in Zero-Sum Markov Games with Bandit Feedback
Oct 5, 2026Yuheng ZhangLast-Iterate Convergence Reinforcement Learning
University of Illinois Urbana-Champaign
Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning
Oct 4, 2026Bingyun Liu, Yuheng JingHierarchical RL Temporal-Difference Learning
Institute of Automation Chinese Academy of Sciences
Temporal-Difference Learning for Dragonchess
Oct 1, 2026Jim O'Connor, Annika Hoag, Sarah Goyette +1Game-Playing Agents Temporal-Difference Learning
Connecticut College, New London CT 06320, USA
Fast Regularized Policy Mirror Descent with One-Step TD Updates
Sep 30, 2026Qipei Chen, Wenye Li, Yule Sun +1Markov Decision Processes Policy Optimization
School of Mathematical Sciences, Fudan University · School of Data Science, Fudan University
Near-Minimax-Optimal Rates for Asynchronous TD Learning with Markovian Data
Sep 30, 2026Yang PengAsynchronous RL Stochastic Approximation
Yau Mathematical Sciences Center, Tsinghua University
GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
Sep 27, 2026Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman +1Reinforcement Learning Temporal-Difference Learning
Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, Texas State University, USA · Department of Computer Science, University of Oxford, UK
From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
Sep 24, 2026Yichen Lin, Xuyuan Xiong, Xue Wang +3In-Context RL Transformer-Based RL
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group · School of Management, University at Buffalo, Buffalo, NY, USA
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Sep 24, 2026Ids van der Werf, Sergio Rozada, Antonio G. MarquesStochastic Approximation Temporal-Difference Learning
Delft University of Technology, Dep. Microelectronics, Delft, The Netherlands
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
Sep 23, 2026James Wu, Chris R. SimsComputational Cognitive Modeling Reinforcement Learning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, 12180 · Department of Cognitive Science Rensselaer Polytechnic Institute Troy, NY, 12180
Limiting-Kernel Q( $λ$ ): Bridging Short and Long Horizons
Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function Estimation Policy Evaluation
Delft University of Technology, The Netherlands · Alpha Brain Technologies, The Netherlands · University of Toronto, Canada
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Sep 17, 2026Xiang Zou, Shengzhu Shi, Junqi Gao +1Temporal-Difference Learning Actor-Critic Methods
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
Regularized Emphatic Temporal-Difference Learning: Stability under Constant Stepsizes
Sep 14, 2026Xingguo Chen, Zhaohui Wu, Jinguo Ye +5Temporal-Difference Learning Stochastic Approximation
Jiangsu Key Laboratory of Big Data Security & Intelligent Processing, Nanjing University of Posts & Telecommunications, Nanjing 210023, China · Nanjing University, 163 Xianlin Avenue, Nanjing 210023, China · Microsoft Corporation, One Microsoft Way, Redmond, WA 98052, USA +1
A Finite-Sample Analysis of Quantile Temporal-Difference Learning
Aug 27, 2026Zijie Cheng, Xiang Li, Yang Peng +1Distributional RL Stochastic Approximation
School of Mathematical Sciences, Peking University · Perelman School of Medicine, University of Pennsylvania · Yau Mathematical Sciences Center, Tsinghua University
PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search
Aug 15, 2026Alexander L. StrehlRL for Game AI Game-Playing Agents
Independent Researcher
Online Inference in Distributional Temporal-Difference Learning
Aug 14, 2026Yang Peng, Liangyu ZhangDistributional RL Temporal-Difference Learning
Yau Mathematical Sciences Center, Tsinghua University. · School of Statistics and Data Science, Shanghai University of Finance and Economics.
Online Inference for Quantile Distributional Reinforcement Learning
Aug 13, 2026Zijie Cheng, Yang Peng, Zhihua ZhangDistributional RL Reinforcement Learning
School of Mathematical Sciences, Peking University · Yau Mathematical Sciences Center, Tsinghua University
Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling
Aug 11, 2026Min Zeng, Yichen Zhang, Xiaofeng ShaoConfidence Region Estimation Stochastic Approximation
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China · Department of Quantitative Methods, Purdue University, West Lafayette, Indiana, USA · Department of Statistics and Data Science, Washington University in St. Louis, St. Louis, Missouri, USA +1
Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis
Aug 7, 2026Idil GözelRL Control Reinforcement Learning
University College London · Department of Physics and Astronomy University College London, London, UK
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
Revisiting TD Target Aggregation under Uncertainty in Q-Learning
Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-Learning Value Function Estimation
Computer Science Department, Florida State University, Tallahassee, Florida, USA
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
Aug 3, 2026Abdelghani Ghanem, Mounir GhoghoQ-Learning Temporal-Difference Learning
College of Computing Mohammed VI Polytechnic University Rabat 11103, Morocco

Oct 26, 2026J/K move · Enter open · S save

Temporal-Difference Calibration of Black-Box Simulators

Min Zeng, Xin Wen, Xi Chen, Yichen Zhang+1

College of Computing, City University of Hong Kong · Stern School of Business, New York University · Daniels School of Business, Purdue University+1

Model Calibration Temporal-Difference Learning Derivative-Free Optimization

Open paper AskPDF arXiv

We develop a derivative-free temporal-difference (TD) framework for calibrating black-box simulators using one-step transitions from a target system and noisy simulator-generated continuation values. The calibration parameter is defined through Bellman orthogonality, recovering the physical parameter under correct simulator specification and yielding a near-optimal

L_2

approximation of the policy value under misspecification. We propose three TD algorithms based on random-direction finite differences, distinguished by their query costs and reuse of noisy simulator evaluations. We establish almost-sure convergence, last-iterate mean-square error bounds, and central limit theorems for Polyak-Ruppert averages across four query-noise regimes. Our analysis reveals a specification-dependent bias phenomenon: for two algorithms, the finite-difference bias of the TD estimating equation vanishes at the calibration target under correct specification, even when the gradient surrogate remains biased. This permits a fixed perturbation radius and rate exponents approaching

1/2

in the simulator rollout budget, whereas misspecification generally recovers the classical Kiefer-Wolfowitz rates. We also develop a few-replicate online bootstrap that yields Student-

t

calibrated confidence intervals under negligible query noise, and identify a covariance mismatch that can invalidate conventional multiplier inference along Markov trajectories. Finally, a second-stage TD recursion corrects the remaining value discrepancy without additional simulator queries.