Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error
Authors: Taisuke Kobayashi
Organizations: Principles of Informatics Research Division, National Institute of Informatics (NII), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan · Informatics Program, Graduate Institute for Advanced Studies (SOKENDAI), 2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Tokyo, Japan
In reinforcement learning (RL), temporal difference (TD) errors are widely adopted for optimizing value and policy functions. However, since the TD error is defined by a bootstrap method, its computation tends to be noisy and destabilize learning. Heuristics to improve the accuracy of TD errors, such as target networks and ensemble models, have been introduced so far. While these are essential approaches for the current deep RL algorithms, they cause side effects like increased computational cost and reduced learning efficiency. Therefore, this paper revisits the TD learning algorithm based on control as inference, deriving a novel algorithm capable of robust learning against noisy TD errors. First, the distribution model of optimality, a binary random variable, is represented by a sigmoid function. Alongside forward and reverse Kullback-Leibler divergences, this new model derives a robust learning rule: when the sigmoid function saturates with a large TD error probably due to noise, the gradient vanishes, implicitly excluding it from learning. Furthermore, the two divergences exhibit distinct gradient-vanishing characteristics. Building on these analyses, the optimality is decomposed into multiple levels to achieve pseudo-quantization of TD errors, aiming for further noise reduction. Additionally, a Jensen-Shannon divergence-based approach is approximately derived to inherit the characteristics of both divergences. These benefits are verified through RL benchmarks, demonstrating stable learning even when heuristics are insufficient or rewards contain noise.
Figures & tables
Figure 1: Replacement of the model for optimality probability: (a) the model with exponential function is usually employed due to its simplicity; (b) to always satisfy the definition of probability, a new model with sigmoid function is introduced in this work.
Figure 2: Analysis of nonlinear TD errors derived with forward and reverse KL divergences: in both cases, the gradients are vanished if the value estimate is closer to upper or lower bound; with reverse KL divergence (left), the contour lines are mirrored for the dashed line ( δ=0 ); with forward KL divergence (right), the contour lines are symmetric for the dashed line.
Figure 3: Decomposition of nonlinear TD errors into weight and error terms: the weight term (left) is saturated (or vanished) only in the case with reverse KL divergence; when σV≃{0,1} , the error term in the case with forward KL divergence outputs almost positive or negative values.
Figure 4: Extension of the model for optimality probability: since sigmoid function always makes its output within [0,1] , multiple optimality variables can be easily introduced; by evenly spacing them, the saturation property is expected between them as a kind of (pseudo-)quantization.
Figure 5: Analysis of nonlinear TD errors with the extension of optimality probability: if λ is too small (left), the saturation property between adjacent optimalities is canceled out; by setting λ is enough large (right), the expected pseudo-quantization with vanishment of gradients between adjacent optimalities is observed.
Figure 6: Analysis of nonlinear TD errors with JS divergence: if the alignment trick is not modified from the cases with forward and reverse KL divergences (left), the shape of contour lines is similar to the case with reverse KL divergence while widening the spaces between lines; with the modified alignment trick (right), the shape of contour lines are mixed up with two previous cases with the appropriate spaces.
Figure 7: Weight and error terms in the cases with Jeffreys and JS divergences: in both terms, the case with Jeffreys divergence loses the saturation property; on the other hand, the case with JS divergence can inherit it in both terms while relaxing its restriction.
#Hidden layers of fully connected networks
2
#Neurons for each hidden layer
100
Activation function
Squish ( Barron, 2021 ; Kobayashi and Aotani, 2023 )
Normalization
RMSNorm ( Zhang and Sennrich, 2019 )
Value estimation
Ensemble w/ underestimation ( Kobayashi, 2025 )
Policy distribution
Student’s t-distribution ( Kobayashi, 2019 )
Discount factor γ
0.99
Table 1: Learning conditions
Figure 8: Results for comparing quantization level L+1 : if L+1=2 (i.e. only saturation near Ru,l ), the scores were not improved enough in all cases; with too large L+1 (i.e. 10,20 ), the scores were slightly improved in HalfCheetah or decreased in Pusher; taking the computational cost into account as well, L+1=5 seems to be the best balance.
Figure 9: Results for comparing the integration way of two divergences: the cases with reverse and forward KL divergences improved especially HalfCheetah and Pusher, respectively; the case with Jeffreys divergence was closer to the worse scores in either case, while the case with JS divergence was closer to the better scores.
Figure 10: Performance properties for noisy TD errors: the baselines (Vanilla, RKL, and FKL) had the similar tendency found in the above experiments; although MME achieved the best robustness in HalfCheetah, it was too fragile in Pusher; the proposed method (JS) achieved the stable results in both cases.
Figure 11: Results for guided RL: MME completely failed all tasks; the proposed PQAC outperformed the standard RL in all tasks, while its performance degradation from ‘E’ (with a well-trained policy) to ‘A’ (with a poorly trained policy) was almost equivalent to the standard RL probably due to the limitation of imitation.
Figure 12: Results for high-dimensional action spaces: the standard RL could not solve these tasks; although MME achieved the best performance on Ant, it was unstable; the proposed PQAC was stable in both tasks, and achieved the best in Swimmer.
Temporal-difference learning estimates values by bootstrapping from later predictions, enabling online learning and control from sequential experience.
The temporal difference (TD) error was first formalized in Sutton (1988), where it was first characterized as the difference between temporally successive predictions, and later, in that same work, formulated as the difference between a bootstrapped target and a prediction. Since then, these two interpretations of the TD error have been used interchangeably in the literature, with the latter eventually being adopted as the standard critic loss in deep reinforcement learning (RL) architectures. In this work, we show that these two interpretations of the TD error are not always equivalent. In particular, we show that increasingly-nonlinear deep RL architectures can cause these interpretations of the TD error to yield increasingly different numerical values. Then, building on this insight, we show how choosing one interpretation of the TD error over the other can affect the performance of deep RL algorithms that utilize the TD error to compute other quantities, such as with deep differential (i.e., average-reward) RL methods. All in all, our results show that the default interpretation of the TD error as the difference between a bootstrapped target and a prediction does not always hold in deep RL settings.
Temporal difference (TD) learning is a fundamental algorithm for estimating value functions in reinforcement learning. Recent finite-time analyses of TD with linear function approximation quantify its theoretical convergence rate. However, they often require setting the algorithm parameters using problem-dependent quantities that are difficult to estimate in practice -- such as the minimum eigenvalue of the feature covariance (ω) or the mixing time of the underlying Markov chain (τmix). In addition, some analyses rely on nonstandard and impractical modifications, exacerbating the gap between theory and practice. To address these limitations, we use an exponential step-size schedule with the standard TD(0) algorithm. We analyze the resulting method under two sampling regimes: independent and identically distributed (i.i.d.) sampling from the stationary distribution, and the more practical Markovian sampling along a single trajectory. In the i.i.d.\ setting, the proposed algorithm does not require knowledge of problem-dependent quantities such as ω, and attains the optimal bias-variance trade-off for the last iterate. In the Markovian setting, we propose a regularized TD(0) algorithm with an exponential step-size schedule. The resulting algorithm achieves a comparable convergence rate to prior works, without requiring projections, iterate averaging, or knowledge of τmix or ω.
Yunxiang Li, Mark Schmidt, Reza Babanezhad +1
University of British Columbia · Samsung AI - Montreal · Simon Fraser University
Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the Successor Rollout Aggregation Deep Q-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.
Lipeng Zu, Xiaonan Zhang
Computer Science Department, Florida State University, Tallahassee, Florida, USA