cs.LGNov 5, 2026

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

Authors: Weiqin Chen, Xinmeng Huang, Dongsheng Ding, Santiago Paternain

Organizations: Purdue University · UPenn · University of Tennessee, Knoxville · RPI

Abstract

Safe reinforcement learning from human feedback (Safe-RLHF) seeks to improve helpfulness while satisfying harmlessness constraints. Existing primal-dual approaches can converge to feasible policies but do not prevent constraint violations during training. We study a stronger requirement, safe exploration for RLHF, under which every policy executed during training, including auxiliary exploration policies, remains feasible. We propose Zeroth-Order Log-Barrier Policy Gradient (ZO-LBPG), a reward-model-free method that optimizes a log-barrier objective using zeroth-order estimates derived directly from human preferences. To ensure safety under noisy feedback, we combine conservative constraint margin estimates with coupled rollouts and an adaptive step size that prevents updates from crossing constraint boundaries. We prove that, with high probability, all iterates and exploration policies remain strictly feasible, and that ZO-LBPG globally converges to an approximately optimal constrained policy. We establish a sample complexity of O~(ε−6+d3ε−5)\widetilde{\mathcal O}(\varepsilon^{-6}+d^3\varepsilon^{-5}) trajectories and O~(ε−8+d5ε−7)\widetilde{\mathcal O}(\varepsilon^{-8}+d^5\varepsilon^{-7}) human queries. Experiments on a synthetic CMDP and a language-model alignment task corroborate the training-time safety guarantees.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

    Apr 21, 2026Qiang Liu, Adrienne Kline, Ermin WeiConstrained RLPolicy Gradient Methods

  2. Sampling-Based Safe Reinforcement Learning

    May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot SafetySafe RL

  3. SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

    Dec 29, 2025Dominik Wagner, Ankit Kanwar, Luke OngSafety-Critical ControlPolicy Optimization