Functional Critics for Off-Policy Actor-Critic: From Stability to Exploration

Qinxun Bai; Yuxuan Han; Yuxiao Wen; Wei Xu; Zhengyuan Zhou

cs.LGSep 26, 2025

Functional Critics for Off-Policy Actor-Critic: From Stability to Exploration

Authors: Qinxun Bai, Yuxuan Han, Yuxiao Wen, Wei Xu, Zhengyuan Zhou

Organizations: Isara Laboratories. · Stern School of Business, New York University. · Horizon Robotics.

Abstract

Actor-critic methods are central to modern reinforcement learning, yet off-policy actor-critic theory remains far removed from practical algorithms, due in part to the interaction between the deadly triad and the continually changing target policy. We revisit functional critics, i.e., value functions that explicitly condition on the policy, and show that this representation provides a shared foundation for off-policy training stability and policy-space exploration. We first develop a convergent off-policy actor-critic framework under linear function approximation that accommodates target-based TD, partial coverage, and evolving behavior policies. Our analysis combines a policy-evaluation guarantee under partial coverage with policy-gradient generalization of functional critics to yield a dual trust-coverage framework that controls policy-gradient error under evolving behavior policies while enabling greater reuse of off-policy data. We further develop a replay-buffer analysis under i.i.d. sampling, yielding a principled replay-sampling design for minimizing the replay-sample complexity required to reach a prescribed evaluation accuracy. We next show that functional critics provide a natural representation for model-free posterior-sampling-style exploration. Standard critic ensembles do not naturally represent joint uncertainty over return functions across policy space, whereas functional-critic ensembles represent policy-conditioned return functions directly. This motivates a particle-based, model-free analogue of posterior-sampling-style exploration without explicit model learning. Finally, we instantiate these ideas in a deep functional actor-critic algorithm that, despite omitting several standard off-policy AC heuristics, achieves strong performance across the full Dog and Humanoid task series of the DeepMind Control Suite.

Figures & tables

Figure 1: From PSRL to functional-critic ensembles. (a) An MDP sample induces a return functional over policy space. (b) With a shared target policy, a standard critic ensemble represents uncertainty only at that policy. (c) With independent target policies, ensemble members estimate different policy-dependent quantities rather than samples from a common posterior over return functions. (d) Functional-critic ensembles represent particles over entire policy-conditioned return functions.

Panel of Figure 2 — Figure 2: Average episodic return on Dog and Humanoid tasks from DM Control. Curves show the mean over four seeds and shaded regions indicate one standard deviation; all experiments use four parallel environments.

	Policy representation	Critic training	Policy distribution	Actor update
PSSVF / PSVF	raw parameters	MC / TD	parameter perturbations	policy-input derivative only
PVN	probe-state actions	MC on fixed policy set	pre-collected policies	policy-input derivative only
PAVF	raw parameters	TD	parameter perturbations	DPG $+$ policy term
PeVFA	learned OPR / SPR	MC, on-policy	single improvement path	standard PPO
Ours	probe-state actions $+$ hidden activations; end-to-end transformer	off-policy target-based TD $+$ replay	concurrent actor ensemble	DPG $+$ policy term

Table 1: Comparison with prior policy-conditioned value-function methods. The listed design differences should not all be attributed to policy conditioning itself; see the discussion below.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

\xi_{\rm sa}^{{}^{\prime}(j)}\leftarrow\rho\xi_{\rm sa}^{{}^{\prime}(j)}+(1-\rho)\xi_{\rm sa}^{(j)}

Appendix

Algorithm 4 Neural Ensemble Functional Actor–Critic

Hyperparameter	Value
optimizer	Adam
learning rate	$3\times 10^{-4}$
batch size	256
discount factor ( $\gamma$ )	0.99
target update rate ( $\tau$ )	0.005
critic ensemble size	10

Appendix

Table 2: Hyperparameters of our functional actor–critic implementation, shared across all reported tasks.

Number of critics	10	10	10	2	2
	Ours	SAC+	TD3+	SAC	TD3
Critic-to-actor update ratio	10:1	10:1	10:1	1:1	2:1
Layer normalization	Yes	Yes	Yes	No	No
Actor policy	Det.	Stoch.	Det.	Stoch.	Det.
Additional rollout action noise	No	No	Yes	No	Yes
Target-policy smoothing	No	No	Yes	No	Yes

Appendix

Table 3: Key algorithmic differences among the compared methods.

Panel of Figure 3 — Figure 3: Targeted evaluation of the replay-reweighted variant Ours_reweight on Dog_run and Humanoid_run. Starting from Algorithm 4 , we modify only the critic replay update: for each candidate replay minibatch, we compute the heuristic distribution $\hat{\mathbf{w}}$ described above and use it to weight the squared TD loss in Eq. equation A.5 . We use $p_{\max}=0.1$ and $K_{\mathrm{lbfgs}}=1$ . On Dog_run, replay reweighting is enabled from the beginning for all four runs and improves environment-sample efficiency over the original algorithm. On Humanoid_run, early activation was not beneficial, motivating delayed activation after the initial feature-learning phase; because activation timing is not identical across all runs, this panel is intended as a diagnostic of feature-dependent activation rather than a fixed-protocol four-seed comparison.

Panel of Figure 4 — Figure 4: Targeted evaluation of the trust-guided behavior-reuse variant Ours_trust on Dog_walk. The variant combines the neural ensemble functional actor–critic in Algorithm 4 with the evaluation-trust behavior-refresh criterion from Algorithm 2 . Before collecting a fresh rollout, we compute the neural approximation of $\mathcal{C}(\pi,\mu)$ described above and skip the rollout when the estimate remains below the chosen threshold. To study delayed activation of the learned feature-based criterion, two runs enable refresh skipping at 75k environment steps and two at 105k environment steps (dashed lines). The threshold for each run is chosen as the $33$ -percentile of empirically estimated $\mathcal{C}(\pi,\mu)$ values over random subsets of the replay buffer. The trust-based variant improves environment-sample efficiency in this targeted study while skipping a substantial fraction of behavior-policy refreshes. The improvement coincides with periods of frequent refresh skipping, providing initial evidence that the evaluation-trust criterion can serve as a practical signal for data reuse.

Figure 5: Illustration of the diversity-coverage trade-off in buffer sampling.

Actor-Critic Methods

Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.

See all 118 papers

Achieving $ε^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions
May 13, 2026Ishaq Hamza, Zaiwei ChenPolicy Gradient Last-Iterate Convergence
IISc · Edwardson School of Industrial Engineering, Purdue University · Purdue IE
Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Jul 29, 2026Gong Gao, Xiao Lai, Ziqi Xie +3Reinforcement Learning Actor-Critic Methods
School of Computer Science and Technology, Tongji University, 200092, Shanghai, China · Shanghai University of Engineering Science, 201620, Shanghai, China
Simple Actors and Deep Critics for Scalable Reinforcement Learning
Aug 27, 2026Guhyeon Kang, Jaehwi Lee, Minhae KwonOffline RL Actor-Critic Methods
Sungkyunkwan University Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea · Soongsil University Republic of Korea

May 13, 2026J/K move · Enter open · S save

Achieving $ε^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions

Ishaq Hamza, Zaiwei Chen

IISc · Edwardson School of Industrial Engineering, Purdue University · Purdue IE

Policy Gradient Last-Iterate Convergence Natural Policy Gradient Actor-Critic Methods Policy Learning Sample-Efficient RL+1

Open paper AskPDF arXiv

In this paper, we establish last-iterate convergence rates for off-policy actor--critic methods in reinforcement learning. In particular, under a single-loop, single-timescale implementation and a broad class of policy updates, including approximate policy iteration and natural policy gradient methods, we prove the first

\tilde{\mathcal{O}}(ε^{-2})

sample complexity guarantee for finding an

ε

-optimal policy under minimal assumptions, namely, the existence of a policy that induces an irreducible Markov chain. This stands in stark contrast to the existing literature, where an

\tilde{\mathcal{O}}(ε^{-2})

sample complexity is achieved only through nested-loop updates and/or under strong, algorithm-dependent assumptions on the policies, such as uniform mixing and uniform exploration. Technically, to address the challenges posed by the coupled update equations arising from the single-loop implementation, as well as the potentially unbounded iterates induced by off-policy learning, our analysis is based on a coupled Lyapunov drift framework. Specifically, we establish a geometric convergence rate for the actor and an

\tilde{\mathcal{O}}(1/T)

convergence rate for the critic, and combine the two Lyapunov drift inequalities through a cross-domination property. We believe this analytical framework is of independent interest and may be applicable to other coupled iterative algorithms with unbounded

Abstract

Actor-Critic Methods

Explore similar work

Achieving ε−2ε^{-2}ε−2 Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions

Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Achieving $ε^{-2}$ Sample Complexity for Single-Loop Actor-Critic under Minimal Assumptions