Constrained Variational Policy Optimization for Hybrid Action Spaces
Abstract
Constrained reinforcement learning in hybrid action spaces must coordinate discrete modes and mode-conditioned continuous controls under long-horizon cost constraints. Existing constrained hybrid-action methods typically optimize factorized policies directly in parameter space, whereas Constrained Variational Policy Optimization (CVPO) separates nonparametric improvement from parametric projection in its original continuous-action formulation. We introduce Hybrid Constrained Variational Policy Optimization (H-CVPO), extending this variational framework to hybrid action spaces. Under boundedness and strict feasibility, its constrained E-step has zero duality gap and, when the dual optimum is attained at positive temperature, yields a closed-form joint target distribution over hybrid actions through exponential reward-cost reweighting. Using the exact hybrid KL decomposition, we formulate a factorized M-step that fits both policy components to this target under separate discrete and conditional-continuous trust regions. For exact E/M updates, we prove fixed-reference surrogate improvement and conditional reward-cost bounds with a sufficient feasibility-margin condition. For approximate updates, we establish the existence of a dual minimizer under continuity on a compact domain and use residual-based admissibility to determine whether to project the target. Ablations identify cost-aware reweighting as an important contributor to constraint control. Extensive experiments in simulated hybrid-control environments show that H-CVPO achieves favorable reward-cost trade-offs compared with hybrid-action baselines.
Figures & tables
| H-CVPO | CHPO | HPPO-Lag | PDQN-Rco | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Reward | Cost | Reward | Cost | Reward | Cost | Reward | Cost | |
| Moving | 1.0 | 2.07 0.03 | 0.85 0.97 | -0.04 0.19 | 0.98 0.73 | -0.69 0.09 | 1.30 1.51 | -0.72 0.43 | 0.12 0.20 |
| 1.5 | 1.41 1.18 | 1.07 0.60 | -0.12 0.35 | 1.83 0.52 | -0.62 0.08 | 1.33 1.50 | -0.60 0.49 | 0.28 0.30 | |
| 2.0 | 2.08 0.02 | 1.00 0.43 | -0.12 0.35 | 1.83 0.52 | -0.46 0.10 | 0.37 0.16 | 0.02 0.43 | 5.83 7.87 | |
| Sliding | 1.0 | 1.08 0.61 | 0.67 0.47 | -0.63 0.21 | 3.79 1.71 | -0.67 0.16 | 0.61 0.38 | -0.95 0.05 | 0.49 0.48 |
| 1.5 | 1.55 0.85 | 0.68 0.30 | -0.62 0.26 | 2.06 0.55 | -0.64 0.04 | 1.07 1.24 | -0.43 0.38 | 2.19 0.83 | |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Observation dimension | Modes | Continuous action | Episode horizon | Primary | Training steps |
| Moving | 2-D, | |||||
| Sliding | 2-D, | |||||
| HardMove | 4-D, , mode-masked | |||||
| Parking | 1-D, |
| Setting | Value |
|---|---|
| Policy network | , ReLU |
| Reward critic network | , ReLU |
| Cost critic network | , ReLU |
| Optimizer | Adam |
| Learning rates | actor ; reward and cost critics |
| Critic / reference batch size |
| H-CVPO | CHPO | HPPO-Lag | PDQN-Rco | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Reward | Cost | Reward | Cost | Reward | Cost | Reward | Cost | |
| Moving | 1.0 | 2.02 0.01 | 0.69 0.29 | -0.14 0.19 | 1.03 0.85 | -0.67 0.05 | 0.37 0.65 | -0.62 0.44 | 0.48 0.49 |
| 1.5 | 1.33 1.22 | 0.35 0.10 | -0.15 0.18 | 1.91 1.24 | -0.58 0.30 | 0.55 0.95 | -0.58 0.20 | 0.48 0.25 | |
| 2.0 | 1.95 0.05 | 0.72 0.84 | -0.15 0.18 | 1.91 1.24 | -0.50 0.14 | 0.04 0.07 | -0.30 0.45 | 0.35 0.18 | |
| Sliding | 1.0 | 1.20 0.57 | 3.21 4.35 | -0.62 0.17 | 2.42 3.47 | -0.68 0.14 | 0.66 0.94 | -0.88 0.13 | 0.55 0.30 |
| 1.5 | 1.57 0.79 | 1.21 0.55 | -0.72 0.21 | 1.38 1.00 | -0.67 0.10 | 1.31 2.14 | -0.67 0.14 | 1.59 0.46 | |
| H-CVPO | A1 | A2 | A3 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Environment | Reward | Cost | Reward | Cost | Reward | Cost | Reward | Cost | |
| Moving | 1.0 | ||||||||
| Moving | 1.5 | ||||||||
| Moving | 2.0 | ||||||||
| Sliding | 1.0 | ||||||||
| Sliding | 1.5 | ||||||||
Constrained RL
Constrained reinforcement learning optimizes task performance while keeping safety risks, resource use, or other costs within explicit limits.
See all 242 papers