Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

Thanh-Tuan Tran; Thanh Nguyen Canh; Nak Young Chong; Xiem HoangVan

cs.ROMar 1, 2026

Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

Authors: Thanh-Tuan Tran, Thanh Nguyen Canh, Nak Young Chong, Xiem HoangVan

Organizations: University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.

Abstract

Reinforcement learning in discrete-continuous hybrid action spaces presents fundamental challenges for robotic manipulation, where high-level task decisions and low-level joint-space execution must be jointly optimized. Existing approaches either discretize continuous components or relax discrete choices into continuous approximations, which suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization. In this paper, we propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that natively handles parameterized hybrid action spaces in a principled manner. We conduct a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants under synchronized Gaussian error assumptions. Building on this analysis, we introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization while improving policy smoothness. Experimental results demonstrate that Hybrid TD3 achieves superior training stability and competitive performance against state-of-the-art hybrid action baselines.

Figures & tables

Fig. 1 : HyTD3 exhibits superior reward performance than HySAC, HyPPO, and HyDDPG under domain randomization.

Fig. 2 : Architecture of the proposed Hybrid TD3 framework.

Panel of Figure 3 — Fig. 3 : Training (left) and test objects (right) used for zero-shot generalization evaluation

Fig. 4 : Estimation bias of the baselines (top row), estimation bias of the proposed methods (middle row), and average return (bottom row) across four manipulation tasks. Solid curves represent mean performance, while shaded areas indicate standard deviations over four independent random seeds.

Fig. 5 : Average return learning curves across four manipulation tasks.

Fig. 6 : Distribution of final episode rewards across seeds and evaluation episodes for our proposed and hybrid action baselines

Fig. 7 : Average return learning curves across four manipulation tasks for 5 different types of robotic arm.

Fig. 8 : Distribution of final episode rewards across seeds and evaluation episodes for 5 different types of robotic arm

Fig. 9 : Representative execution trajectories of 5 different types of robotic arm

Robot Platform	Action 0	Action 1	Action 2	Action 3
Standard Environment (Normal Setting)
UF850	$94.25\pm 1.92$	$89.75\pm 4.66$	$80.75\pm 2.58$	$83.25\pm 3.56$
KUKA	$96.67\pm 1.25$	$90.00\pm 2.94$	$83.00\pm 4.08$	$90.67\pm 2.05$
Panda	$94.67\pm 2.05$	$74.67\pm 2.05$	$69.00\pm 1.63$	$83.33\pm 2.05$
Kinova	$94.67\pm 2.05$	$91.33\pm 2.87$	$89.33\pm 1.70$	$87.33\pm 1.70$
UR5e	$88.00\pm 2.50$	$82.00\pm 3.00$	$75.00\pm 4.50$	$84.00\pm 2.00$

TABLE I : Task success rates (%) over 100 trials per action, averaged across four random seeds (mean

\pm

std). xx and xx indicate the best and second-best results, respectively.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Critic loss $\mathcal{L}_{\phi}$	Critic target $y$	Actor loss $\mathcal{L}_{\theta}$
TD3	$\sum_{i=1}^{2}\big(Q_{\phi_{i}}(s,a_{c},a_{d})-y\big)^{2}$	$r+\gamma\min_{i=1,2}Q_{\phi_{i}^{\prime}}(s^{\prime},a_{c}^{\prime},a_{d}^{\prime})$	$-\max_{a_{d}\in\mathcal{A}_{d}}Q_{\phi_{1}}(s,\mu_{\theta}(s),a_{d})$
DDPG	$\big(Q_{\phi}(s,a_{c},a_{d})-y\big)^{2}$	$r+\gamma Q_{\phi^{\prime}}(s^{\prime},a_{c}^{\prime},a_{d}^{\prime})$	$-\max_{a_{d}\in\mathcal{A}_{d}}Q_{\phi}(s,\mu_{\theta}(s),a_{d})$
SAC	$\sum_{i=1}^{2}\big(Q_{\phi_{i}}(s,a_{c},a_{d})-y\big)^{2}$	$\begin{aligned} r+\gamma\big(\min_{i=1,2}Q_{\phi_{i}^{\prime}}&-\alpha_{c}\log p_{c}(a^{\prime}_{c}\|s^{\prime})\\ &-\alpha_{d}\log p_{d}(a^{\prime}_{d}\|s^{\prime})\big)\end{aligned}$	$\begin{aligned} \mathbb{E}_{a_{c}\sim p_{c},\,a_{d}\sim p_{d}}[\alpha_{c}\log p_{c}(a^{\prime}_{c}\|s^{\prime})+\alpha_{d}\log p_{d}(a^{\prime}_{d}\|s^{\prime})-\min_{i=1,2}Q_{\phi_{i}}]\end{aligned}$
PPO	$(V(s)-V_{\mathrm{targ}})^{2}$	$V(s)+A(s,a)$ (GAE)	$-\mathbb{E}\bigg[\min\big(r_{t}(\theta)A(s,a),\;\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)A(s,a)\big)\bigg]$

Appendix

TABLE II : Hybrid adaptation of RL baselines. All share the same state

(s)

and action

(a_{c},a_{d})

inputs to the critics.

$k$	$kN$ (HyTQC)	$\delta Q_{\mathrm{HyTQC}}$	$\delta Q_{\mathrm{HyACC}}$ ( $kN{-}2$ )
20	100	$-0.3460$	$-0.3702$
21	105	$-0.2865$	$-0.3107$
22	110	$-0.2245$	$-0.2496$
23	115	$-0.1591$	$-0.1858$
24	120	$-0.0877$	$-0.1173$

Appendix

TABLE III : Bias coefficients for HyTQC (

\beta{=}0

) and HyACC (

\beta{=}2

N{=}5

M{=}25

Hybrid Action Space RL

See all 8 papers

Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
May 14, 2026Matias Alvo, Daniel Russo, Yash KanoriaRL Control Reinforcement Learning
Graduate School of Business Columbia University
Revisiting Action Factorization for Complex Action Spaces
Jun 25, 2026Timothy Flavin, Sandip SenReinforcement Learning Actor-Critic Methods
The University of Tulsa, Tulsa, Oklahoma, USA
Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition
Aug 10, 2026Changhao Li, Yifang Zhang, Heng Zhang +6Reinforcement Learning Robotic RL
HHCM, Istituto Italiano di Tecnologia, Genoa, Italy · DIBRIS, University of Genova, Genova, Italy · Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy +1

May 14, 2026J/K move · Enter open · S save

Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

Matias Alvo, Daniel Russo, Yash Kanoria

Graduate School of Business Columbia University

RL Control Reinforcement Learning Policy Optimization Hybrid Action Space RL

Open paper AskPDF arXiv

We study reinforcement learning in hybrid discrete-continuous action spaces, such as settings where the discrete component selects a regime (or index) and the continuous component optimizes within it -- a structure common in robotics, control, and operations problems. Standard model-free policy gradient methods rely on score-function (SF) estimators and suffer from severe credit-assignment issues in high-dimensional settings, leading to poor gradient quality. On the other hand, differentiable simulation largely sidesteps these issues by backpropagating through a simulator, but the presence of discrete actions or non-smooth dynamics yields biased or uninformative gradients. To address this, we propose Hybrid Policy Optimization (HPO), which backpropagates through the simulator wherever smoothness permits, using a mixed gradient estimator that combines pathwise and SF gradients while maintaining unbiasedness. We also show how problems with action discontinuities can be reformulated in hybrid form, further broadening its applicability. Empirically, HPO substantially outperforms PPO on inventory control and switched linear-quadratic regulator problems, with performance gaps increasing as the continuous action dimension grows. Finally, we characterize the structure of the mixed gradient, showing that its cross term -- which captures how continuous actions influence future discrete decisions -- becomes negligible near a discrete best response, thereby enabling approximate decentralized updates of the continuous and discrete components and reducing variance near optimality. All resources are available at github.com/MatiasAlvo/hybrid-rl.

Abstract

Hybrid Action Space RL

Explore similar work

Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

Revisiting Action Factorization for Complex Action Spaces

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition