cs.AIMay 2, 2026

Regularized Centered Emphatic Temporal Difference Learning

Authors: Xingguo ChenChaohui WuJinguo YeChao LiShangdong YangGuang YangTianyu LiangWenhao Wang

Organizations: Nanjing University of Posts and Telecommunications · Nanjing University · Microsoft · National University of Defense Technology

Abstract

Off-policy temporal-difference (TD) learning with function approximation faces a structural tradeoff among stability, projection geometry, and variance control. Emphatic TD (ETD) improves the off-policy projection geometry through follow-on emphasis, but the follow-on trace can have high variance. We revisit this tradeoff through Bellman-error centering. Although centering naturally removes a common drift term from TD errors, we show that a naive centered emphatic extension introduces an auxiliary coupling that can destroy the positive-definiteness of the ETD key matrix. We propose \emph{Regularized Emphatic Temporal-Difference Learning} (RETD), which preserves the follow-on trace and regularizes only the auxiliary centering recursion, corresponding to lifting the lower-right block of the coupled key matrix from 11 to 1+c1+c. We derive the RETD core matrix, prove convergence under a conservative sufficient regularization condition, and evaluate the method on diagnostic linear off-policy prediction tasks. The experiments show that RETD avoids the instability of naive centered emphatic learning, preserves favorable emphatic geometry, and exhibits a robust intermediate regime for the regularization parameter cc across the diagnostics.

Explore similar work

CardsList