cs.LGSep 26, 2025

Functional Critics for Off-Policy Actor-Critic: From Stability to Exploration

Authors: Qinxun Bai, Yuxuan Han, Yuxiao Wen, Wei Xu, Zhengyuan Zhou

Organizations: Isara Laboratories. · Stern School of Business, New York University. · Horizon Robotics.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Actor-critic methods are central to modern reinforcement learning, yet off-policy actor-critic theory remains far removed from practical algorithms, due in part to the interaction between the deadly triad and the continually changing target policy. We revisit functional critics, i.e., value functions that explicitly condition on the policy, and show that this representation provides a shared foundation for off-policy training stability and policy-space exploration. We first develop a convergent off-policy actor-critic framework under linear function approximation that accommodates target-based TD, partial coverage, and evolving behavior policies. Our analysis combines a policy-evaluation guarantee under partial coverage with policy-gradient generalization of functional critics to yield a dual trust-coverage framework that controls policy-gradient error under evolving behavior policies while enabling greater reuse of off-policy data. We further develop a replay-buffer analysis under i.i.d. sampling, yielding a principled replay-sampling design for minimizing the replay-sample complexity required to reach a prescribed evaluation accuracy. We next show that functional critics provide a natural representation for model-free posterior-sampling-style exploration. Standard critic ensembles do not naturally represent joint uncertainty over return functions across policy space, whereas functional-critic ensembles represent policy-conditioned return functions directly. This motivates a particle-based, model-free analogue of posterior-sampling-style exploration without explicit model learning. Finally, we instantiate these ideas in a deep functional actor-critic algorithm that, despite omitting several standard off-policy AC heuristics, achieves strong performance across the full Dog and Humanoid task series of the DeepMind Control Suite.

Figures & tables

Figure 1: From PSRL to functional-critic ensembles. (a) An MDP sample induces a return functional over policy space. (b) With a shared target policy, a standard critic ensemble represents uncertainty only at that policy. (c) With independent target policies, ensemble members estimate different policy-dependent quantities rather than samples from a common posterior over return functions. (d) Functional-critic ensembles represent particles over entire policy-conditioned return functions.
Figure 2: Average episodic return on Dog and Humanoid tasks from DM Control. Curves show the mean over four seeds and shaded regions indicate one standard deviation; all experiments use four parallel environments.
 Policy representationCritic trainingPolicy distributionActor update
PSSVF / PSVFraw parametersMC / TDparameter perturbationspolicy-input derivative only
PVNprobe-state actionsMC on fixed policy setpre-collected policiespolicy-input derivative only
PAVFraw parametersTDparameter perturbationsDPG ++ policy term
PeVFAlearned OPR / SPRMC, on-policysingle improvement pathstandard PPO
Oursprobe-state actions ++ hidden activations; end-to-end transformeroff-policy target-based TD ++ replayconcurrent actor ensembleDPG ++ policy term
Table 1: Comparison with prior policy-conditioned value-function methods. The listed design differences should not all be attributed to policy conditioning itself; see the discussion below.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

 ξsa′(j)←ρξsa′(j)+(1−ρ)ξsa(j)\xi_{\rm sa}^{{}^{\prime}(j)}\leftarrow\rho\xi_{\rm sa}^{{}^{\prime}(j)}+(1-\rho)\xi_{\rm sa}^{(j)} 
Appendix
Algorithm 4 Neural Ensemble Functional Actor–Critic
HyperparameterValue
optimizerAdam
learning rate3×10−43\times 10^{-4}
batch size256
discount factor ( γ\gamma )0.99
target update rate ( τ\tau )0.005
critic ensemble size10
Appendix
Table 2: Hyperparameters of our functional actor–critic implementation, shared across all reported tasks.
 OursSAC+TD3+SACTD3
Number of critics10101022
Critic-to-actor update ratio10:110:110:11:12:1
Layer normalizationYesYesYesNoNo
Actor policyDet.Stoch.Det.Stoch.Det.
Additional rollout action noiseNoNoYesNoYes
Target-policy smoothingNoNoYesNoYes
Appendix
Table 3: Key algorithmic differences among the compared methods.
Figure 3: Targeted evaluation of the replay-reweighted variant Ours_reweight on Dog_run and Humanoid_run. Starting from Algorithm 4 , we modify only the critic replay update: for each candidate replay minibatch, we compute the heuristic distribution w^\hat{\mathbf{w}} described above and use it to weight the squared TD loss in Eq. equation A.5 . We use pmax⁡=0.1p_{\max}=0.1 and Klbfgs=1K_{\mathrm{lbfgs}}=1 . On Dog_run, replay reweighting is enabled from the beginning for all four runs and improves environment-sample efficiency over the original algorithm. On Humanoid_run, early activation was not beneficial, motivating delayed activation after the initial feature-learning phase; because activation timing is not identical across all runs, this panel is intended as a diagnostic of feature-dependent activation rather than a fixed-protocol four-seed comparison.
Figure 4: Targeted evaluation of the trust-guided behavior-reuse variant Ours_trust on Dog_walk. The variant combines the neural ensemble functional actor–critic in Algorithm 4 with the evaluation-trust behavior-refresh criterion from Algorithm 2 . Before collecting a fresh rollout, we compute the neural approximation of C(π,μ)\mathcal{C}(\pi,\mu) described above and skip the rollout when the estimate remains below the chosen threshold. To study delayed activation of the learned feature-based criterion, two runs enable refresh skipping at 75k environment steps and two at 105k environment steps (dashed lines). The threshold for each run is chosen as the 3333 -percentile of empirically estimated C(π,μ)\mathcal{C}(\pi,\mu) values over random subsets of the replay buffer. The trust-based variant improves environment-sample efficiency in this targeted study while skipping a substantial fraction of behavior-policy refreshes. The improvement coincides with periods of frequent refresh skipping, providing initial evidence that the evaluation-trust criterion can serve as a practical signal for data reuse.
Figure 5: Illustration of the diversity-coverage trade-off in buffer sampling.
More in this topic

Actor-Critic Methods

Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.

See all 118 papers

Explore similar work

CardsList
May 13, 2026

Achieving ε−2ε^{-2} Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions

In this paper, we establish last-iterate convergence rates for off-policy actor--critic methods in reinforcement learning. In particular, under a single-loop, single-timescale implementation and a broad class of policy updates, including approximate policy iteration and natural policy gradient methods, we prove the first O~(ε−2)\tilde{\mathcal{O}}(ε^{-2}) sample complexity guarantee for finding an εε-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain. This stands in stark contrast to the existing literature, where an O~(ε−2)\tilde{\mathcal{O}}(ε^{-2}) sample complexity is achieved only through nested-loop updates and/or under strong, algorithm-dependent assumptions on the policies, such as uniform mixing and uniform exploration. Technically, to address the challenges posed by the coupled update equations arising from the single-loop implementation, as well as the potentially unbounded iterates induced by off-policy learning, our analysis is based on a coupled Lyapunov drift framework. Specifically, we establish a geometric convergence rate for the actor and an O~(1/T)\tilde{\mathcal{O}}(1/T) convergence rate for the critic, and combine the two Lyapunov drift inequalities through a cross-domination property. We believe this analytical framework is of independent interest and may be applicable to other coupled iterative algorithms with unbounded

Ishaq Hamza, Zaiwei Chen

IISc · Edwardson School of Industrial Engineering, Purdue University · Purdue IE

Policy GradientLast-Iterate ConvergenceNatural Policy GradientActor-Critic MethodsPolicy LearningSample-Efficient RL+1
AskPDF
Jul 29, 2026

Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates amplify early-stage estimation errors. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor-critic methods. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying bias.In this paper, we propose Collaborative Weighting Actor-Critic (CWAC), a unified framework that explicitly accounts for predictive uncertainty in value estimation. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement. CWAC can be seamlessly integrated into existing off-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks. Our code is publicly available at https://anonymous.4open.science/r/CWAC-348E.

Gong Gao, Xiao Lai, Ziqi Xie +3

School of Computer Science and Technology, Tongji University, 200092, Shanghai, China · Shanghai University of Engineering Science, 201620, Shanghai, China

Reinforcement LearningActor-Critic MethodsOff-Policy RL
AskPDF
Aug 27, 2026

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

Guhyeon Kang, Jaehwi Lee, Minhae Kwon

Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

Offline RLActor-Critic Methods
AskPDF