Temporal-Difference Learning

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 57

All topics

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks
Oct 6, 2026Zijie Xu, Bingrui Guo, Yiding Sun +3Deep Q-Learning Temporal-Difference Learning
Peking University Beijing, 100871, China
Fast Last-Iterate Convergence in Zero-Sum Markov Games with Bandit Feedback
Oct 5, 2026Yuheng ZhangLast-Iterate Convergence Reinforcement Learning
University of Illinois Urbana-Champaign
Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning
Oct 4, 2026Bingyun Liu, Yuheng JingHierarchical RL Temporal-Difference Learning
Institute of Automation Chinese Academy of Sciences
Temporal-Difference Learning for Dragonchess
Oct 1, 2026Jim O'Connor, Annika Hoag, Sarah Goyette +1Game-Playing Agents Temporal-Difference Learning
Connecticut College, New London CT 06320, USA
Fast Regularized Policy Mirror Descent with One-Step TD Updates
Sep 30, 2026Qipei Chen, Wenye Li, Yule Sun +1Markov Decision Processes Policy Optimization
School of Mathematical Sciences, Fudan University · School of Data Science, Fudan University
Sharp Statistical Rates for Asynchronous TD Learning with Markovian Data
Sep 30, 2026Yang PengAsynchronous RL Stochastic Approximation
Yau Mathematical Sciences Center, Tsinghua University
GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
Sep 27, 2026Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman +1Reinforcement Learning Temporal-Difference Learning
Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, Texas State University, USA · Department of Computer Science, University of Oxford, UK
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Sep 24, 2026Ids van der Werf, Sergio Rozada, Antonio G. MarquesStochastic Approximation Temporal-Difference Learning
Delft University of Technology, Dep. Microelectronics, Delft, The Netherlands
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
Sep 23, 2026James Wu, Chris R. SimsComputational Cognitive Modeling Reinforcement Learning
Department of Computer Science Rensselaer Polytechnic Institute Troy, NY, 12180 · Department of Cognitive Science Rensselaer Polytechnic Institute Troy, NY, 12180
Limiting-Kernel Q( $λ$ ): Bridging Short and Long Horizons
Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function Estimation Policy Evaluation
Delft University of Technology, The Netherlands · Alpha Brain Technologies, The Netherlands · University of Toronto, Canada
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Sep 17, 2026Xiang Zou, Shengzhu Shi, Junqi Gao +1Temporal-Difference Learning Actor-Critic Methods
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
A Finite-Sample Analysis of Quantile Temporal-Difference Learning
Aug 27, 2026Zijie Cheng, Xiang Li, Yang Peng +1Distributional RL Stochastic Approximation
School of Mathematical Sciences, Peking University · Perelman School of Medicine, University of Pennsylvania · Yau Mathematical Sciences Center, Tsinghua University
Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning
Aug 13, 2026Zijie Cheng, Yang Peng, Zhihua ZhangDistributional RL Reinforcement Learning
School of Mathematical Sciences, Peking University · Yau Mathematical Sciences Center, Tsinghua University
Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling
Aug 11, 2026Min Zeng, Yichen Zhang, Xiaofeng ShaoConfidence Region Estimation Stochastic Approximation
Department of Biostatistics, City University of Hong Kong, Hong Kong SAR, China · Department of Quantitative Methods, Purdue University, West Lafayette, Indiana, USA · Department of Statistics and Data Science, Washington University in St. Louis, St. Louis, Missouri, USA +1
Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis
Aug 7, 2026Idil GözelRL Control Reinforcement Learning
University College London · Department of Physics and Astronomy University College London, London, UK
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
Revisiting TD Target Aggregation under Uncertainty in Q-Learning
Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-Learning Value Function Estimation
Computer Science Department, Florida State University, Tallahassee, Florida, USA
Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
Aug 3, 2026Abdelghani Ghanem, Mounir GhoghoQ-Learning Temporal-Difference Learning
College of Computing Mohammed VI Polytechnic University Rabat 11103, Morocco
Gated Q-learning: Add Off-Policy Bias to Taste
Jul 31, 2026Brett DaleyCredit Assignment in RL Step-Level Credit Assignment in RL
Meta
Online Policy Evaluation for MDPs with Dynamic UBSR Measures
Jul 25, 2026Weikai Wang, Erick DelagePolicy Evaluation Risk-Sensitive RL
GERAD & Department of Decision Sciences, HEC Montr´eal, Canada
General Value Functions for Remaining Useful Life and Failure-Mode Prediction
Jul 24, 2026Hao Yan, Ali Sarabi, Qing Zou +1RUL Estimation Value Function Estimation
School of Computing and Augmented Intelligence, Arizona State University
Relative Value Learning
Jul 23, 2026Marc Höftmann, Jan Robine, Stefan HarmelingValue Function Estimation Temporal-Difference Learning
Department of Computer Science, Technical University of Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence
Generalized Kalman filter based temporal difference reinforcement learning
Jul 22, 2026Vasos Arnaoutis, Eric Lutters, Bojana RosićRL Control Reinforcement Learning
Applied Mechanics and Data Analysis, Faculty of Engineering Technology, University of Twente, Enschede, the Netherlands · Department of Design, Production and Management, Faculty of Engineering Technology, University of Twente, Enschede, the Netherlands · Digital Engineering research group, Faculty of Mechanical and Industrial Engineering, Vienna University of Technology, Vienna, Austria
Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces
Jul 20, 2026Dongming Wang, Pengcheng Dai, Wenwu Yu +1Multi-Agent Control Temporal-Difference Learning
Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA · Engineering Systems and Design Pillar, Singapore University of Technology and Design, Singapore 487372 · Frontiers Science Center for Mobile Information Communication and Security, School of Mathematics, Southeast University, Nanjing 211102, China +1
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Jul 15, 2026Leitian Tao, Baolin Peng, Wenlin Yao +5Credit Assignment in RL Step-Level Credit Assignment in RL
University of Wisconsin–Madison · Microsoft Research
Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions
Jul 6, 2026Kyohei Suzuki, Konstantinos SlavakisTemporal-Difference Learning Nonconvex Optimization
Department of Information and Communications Engineering, Institute of Science Tokyo, Yokohama, 226-8501, Japan
Mesh-RL: Coupled subgrid reinforcement learning
Jun 24, 2026Behnam Gheshlaghi, Bahador Rashidi, Shahin AtakishiyevReinforcement Learning Credit Assignment in RL
Independent Researcher · University of Alberta
A Single Stepsize Suffices for Unprojected Linear TD(0): Simultaneous Robust and Fast Rates via Polyak--Ruppert Averaging
Jun 23, 2026Wei-Cheng Lee, Francesco OrabonaStochastic Approximation Temporal-Difference Learning
King Abdullah University of Science and Technology (KAUST) Thuwal, 23955-6900, Kingdom of Saudi Arabia
On the Variance of Temporal Difference Learning and its Reduction Using Control Variates
Jun 18, 2026Hsiao-Ru Pan, Bernhard SchölkopfValue Function Estimation Temporal-Difference Learning
Max Planck Institute for Intelligent Systems, Tübingen · ELLIS Institute Tübingen · ETH Zürich
A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise
Jun 16, 2026M. Forzo, E. Monzio Compagnoni, A. Russo +1Stochastic Differential Equations Temporal-Difference Learning
Technical University of Munich (TUM), Munich, Germany · University of Basel, Basel, Switzerland · Boston University, Boston, USA.

Oct 6, 2026, cs.NEJ/K move · Enter open · S save

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

Zijie Xu, Bingrui Guo, Yiding Sun, Yiting Dong+2

Peking University Beijing, 100871, China

Deep Q-Learning Temporal-Difference Learning Spiking Neural Networks

Open paper AskPDF arXiv

Spiking neural networks (SNNs) offer sparse and event-driven computation, making them attractive for energy-constrained reinforcement learning (RL) on edge devices. In value-based RL, deep spiking Q-networks (DSQNs) combine such efficiency with action-value estimation for decision making. However, existing DSQNs often require multiple simulation timesteps for competitive performance, increasing computational and energy costs, whereas reducing the timesteps can cause substantial performance degradation. We investigate this degradation from the perspective of Q-value estimation errors. By decomposing errors across actions into common-mode and differential-mode components, we find that low-timestep DSQNs suffer disproportionately from common-mode errors shared across action values, which are particularly detrimental to temporal-difference learning through bootstrapped targets. Based on this finding, we propose Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which uses an auxiliary ANN to compensate for common-mode errors in the SNN outputs. At inference, greedy action selection can be performed directly from the SNN outputs, allowing the auxiliary ANN to be completely removed and preserving the energy efficiency of SNNs. Extensive experiments on Atari and MiniAtar environments demonstrate substantial performance improvements under low-timestep settings. CMC-DSQN outperforms state-of-the-art DSQN baselines by nearly

20\%

T=2

and further surpasses the ANN baseline at

T=4

Momentum

Related topics

Latest papers 57