Offline Constrained RLHF with Multiple Preference Oracles

Brenden Latham; Mehrdad Moharrami

cs.LGMar 31, 2026

Offline Constrained RLHF with Multiple Preference Oracles

Authors: Brenden Latham, Mehrdad Moharrami

Organizations: Computer Science Department, University of Iowa, Iowa City, IA, USA

Abstract

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.

Figures & tables

Method	Objective	Setting	Guarantees	Constraint Type
Chakraborty et al. (2024b)	Max-Min Welfare	Online	Asymptotic/None	Unconstrained
Ramesh et al. (2024)	Robust Utility	Offline	None	Robustness Penalty
Ge et al. (2024)	Social Choice	Theoretical	Axiomatic	N/A
Dai et al. (2024)	Lagrangian PPO	Online	Empirical	Expected safety constraint
Zhou et al. (2024)	Scalarization	Online	Empirical	Pareto
Wachi et al. (2024)	Constrained Alignment	Offline	Statistical optimality & violation	Expected reward

Table 1: Comparison of RLHF methods.

Parameters: $J_{\min}=-0.6473$ , $\eta=0.3$ , $T=1000$
	Training (74k)		Test (7k)
Metric	$\pi_{0}$	$\pi_{\bar{\lambda}}$	$\pi_{\bar{\lambda}}$
Safety score $\widehat{\mathbb{E}}[\widehat{r}_{1}]$	-1.8997	-1.6034	-1.6044
Helpfulness score $\widehat{\mathbb{E}}[\widehat{r}_{2}]$	-0.8867	-0.6479	-0.6287
Estimated violation	0.2394	0.0006	0.0000

Table 2: Learned reward scores in the two-candidate PKU-SafeRLHF experiment.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Performance vs. dataset size ( $N$ ) with $T=1000$ over three settings of $w$ . Top: Primal objective sub-optimality. Bottom: Constraint violation.

Explore similar work

Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences
Feb 26, 2026Junghyun Lee, Minju Hong, Kwang-Sung Jun +2Regret Minimization in RL Preference-Based RL
KAIST AI · KAIST EE · POSTECH CSE/AI
Learning Optimal and Sample-Efficient Decision Policies with Guarantees
Feb 20, 2026Daqian ShaoReinforcement Learning Sample-Efficient RL
Wolfson College University of Oxford
Safe Reinforcement Learning with Preference-based Constraint Inference
Mar 24, 2026Chenglin Li, Grant Ruan, Hua GengConstrained RL Safe RL
Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

Feb 26, 2026J/K move · Enter open · S save

Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences

Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun+1

KAIST AI · KAIST EE · POSTECH CSE/AI

Regret Minimization in RL Preference-Based RL RL from Human Feedback Preference Learning Pairwise Preference Learning

Open paper AskPDF arXiv

We consider the problem of regularized best-response max-regret minimization in online RLHF under general preferences and bandit feedback. While various regularizers are utilized to robustify alignment, known polylogarithmic regret guarantees remain heavily specific to KL. To investigate whether such fast rates extend beyond KL, we adopt the Generalized Bilinear Preference Model (GBPM) -- capturing intransitive preferences over

d

-dimensional item-wise features via a rank-

2r

skew-symmetric matrix -- to isolate the impact of generic regularization. Crucially, under GBPM, we prove that the dual gap of any greedy policy is bounded by the squared estimation error, derived using only strong convexity and skew-symmetry. Under a feature coverage assumption, we establish a generic polylogarithmic regret of

\tilde{\mathcal{O}}(ηd^4 C_{\min}^{-1} (\log T)^2 \wedge d^2 C_{\min}^{-1/2} \sqrt{T})

with Greedy Sampling, and a dimension-wise improved regret (for well-conditioned arm-sets) of

\tilde{\mathcal{O}}(C_{\min}^{-2} \sqrt{ηr T} \wedge r^{1/3} C_{\min}^{-4/3} T^{2/3})

with Explore-Then-Commit, where

η^{-1}

is the regularization coefficient,

T

is the time horizon, and

C_{\min}

is an arm-set dependent quantity. This demonstrates that ``fast'' regrets are not KL-specific, but rather a fundamental consequence of generic strongly convex geometry.

Abstract

Explore similar work

Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences

Learning Optimal and Sample-Efficient Decision Policies with Guarantees

Safe Reinforcement Learning with Preference-based Constraint Inference