Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback
Organizations: Purdue University · UPenn · University of Tennessee, Knoxville · RPI
Abstract
Safe reinforcement learning from human feedback (Safe-RLHF) seeks to improve helpfulness while satisfying harmlessness constraints. Existing primal-dual approaches can converge to feasible policies but do not prevent constraint violations during training. We study a stronger requirement, safe exploration for RLHF, under which every policy executed during training, including auxiliary exploration policies, remains feasible. We propose Zeroth-Order Log-Barrier Policy Gradient (ZO-LBPG), a reward-model-free method that optimizes a log-barrier objective using zeroth-order estimates derived directly from human preferences. To ensure safety under noisy feedback, we combine conservative constraint margin estimates with coupled rollouts and an adaptive step size that prevents updates from crossing constraint boundaries. We prove that, with high probability, all iterates and exploration policies remain strictly feasible, and that ZO-LBPG globally converges to an approximately optimal constrained policy. We establish a sample complexity of trajectories and human queries. Experiments on a synthetic CMDP and a language-model alignment task corroborate the training-time safety guarantees.
Figures & tables
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Algorithm | Noise | Coupling | Infeasible | Worst | Worst Executed | Final Reward Gap |
|---|---|---|---|---|---|---|
| ZO-LBPG | low | ✓ | ||||
| ZO-LBPG | high | ✓ | ||||
| ZPGPD-HF | low | ✗ | ||||
| ZPGPD-HF | high | ✗ | ||||
| ZPGPD-HF | low | ✓ | ||||
| ZO-LBPG | low | ✗ |
| Algorithm | Infeasible | Exploration | Worst | Helpfulness | Feasible |
|---|---|---|---|---|---|
| ZO-LBPG | ( ) | ||||
| ZPGPD-HF | ( ) | ||||
| Unconstrained | ( ) | — |
Safe RL
Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.
See all 238 papers