cs.LGFeb 25, 2026

Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual

Authors: Yining Li, Peizhong Ju, Ness Shroff

Organizations: The Ohio State University · University of Kentucky

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement Learning from Human Feedback (RLHF) plays a significant role in aligning Large Language Models (LLMs) with human preferences. While RLHF with expected reward constraints can be formulated as a primal-dual optimization problem, standard primal-dual methods only guarantee convergence with a distributional policy where the saddle-point problem is in convex-concave form. Moreover, standard primal-dual methods may exhibit instability or divergence in the last iterate under policy parameterization in practical applications. In this work, we propose a universal primal-dual framework for safe RLHF that unifies a broad class of existing alignment algorithms, including safe-RLHF, one-shot, and multi-shot based methods. Building on this framework, we introduce an optimistic primal-dual (OPD) algorithm that incorporates predictive updates for both primal and dual variables to stabilize saddle-point dynamics. We establish last-iterate convergence guarantees for the proposed method, covering both exact policy optimization in the distributional space and convergence to a neighborhood of the optimal solution whose gap is related to approximation error and bias under parameterized policies. Our analysis reveals that optimism plays a crucial role in mitigating oscillations inherent to constrained alignment objectives, thereby closing a key theoretical gap between constrained RL and practical RLHF.

Figures & tables

 πt=arg⁡\displaystyle\pi_{t}=\argmax⁡π(L(π,λt−1)\displaystyle\max_{\pi}\left(\mathcal{L}(\pi,\lambda_{t-1})\right. (4)
  −Ex∼D[ηθKL(π(⋅∣x)∥π^t(⋅∣x))]),\displaystyle\left.-\mathbb{E}_{x\sim\mathcal{D}}\left[\eta_{\theta}\mathrm{KL}\left(\pi(\cdot|x)\|\hat{\pi}_{t}(\cdot|x)\right)\right]\right), 
Algorithm 2 OPD with Primal Distributional Policies
 θt=ProjΘ(θ^t+1ηθ+βF(θ^t)†∇θL(πθ^t,λt−1)).\theta_{t}=\text{Proj}_{\Theta}\left(\hat{\theta}_{t}+\frac{1}{\eta_{\theta}+\beta}F(\hat{\theta}_{t})^{\dagger}\nabla_{\theta}\mathcal{L}(\pi_{\hat{\theta}_{t}},\lambda_{t-1})\right). (8)
Algorithm 3 OPD with Primal Parameterized Policies
Figure 1: Comparison of OPD and PD under a softmax tabular parameterization in a single-state, two-action RLHF toy problem. OPD (red) converges to the optimal solution in the last iterate, while PD (blue) exhibits persistent oscillations and fails to converge.
Figure 2: Comparison of PD and OPD on reward and constrained reward during the training phase.
Figure 3: Inference comparison of PD and OPD on reward and cost.
More in this topic

LLM Safety Alignment

Large language model safety alignment trains and evaluates systems to follow human policies while resisting harmful requests and misuse.

See all 344 papers

Explore similar work

CardsList
Apr 21, 2026

Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

Safe Reinforcement Learning from Human Feedback (Safe RLHF) has recently achieved empirical success in developing helpful and harmless large language models by decoupling human preferences regarding helpfulness and harmlessness. Existing approaches typically rely on fitting fixed horizon reward models from human feedback and have only been validated empirically. In this paper, we formulate safe RLHF as an infinite horizon discounted Con- strained Markov Decision Process (CMDP), since humans may interact with the model over a continuing sequence of interactions rather than within a single finite episode. We propose two Safe RLHF algorithms that do not require reward model fitting and, in contrast to prior work assuming fixed-length trajectories, support flexible trajectory lengths for training. Both algo- rithms are based on the primal-dual method and achieve global convergence guarantees with polynomial rates in terms of policy gradient iterations, trajectory sample lengths, and human preference queries. To the best of our knowledge, this is the first work to study infinite horizon discounted CMDP under human feedback and establish global, non-asymptotic convergence.

Qiang Liu, Adrienne Kline, Ermin Wei

Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA · Center for Artificial Intelligence, Bluhm Cardiovascular Institute, Northwestern Medicine, Chicago, IL, USA · Department of Surgery, Northwestern University, Chicago, IL, USA +3

Constrained RLPolicy Gradient MethodsConstrained OptimizationSafe RLRL from Human Feedback
AskPDF
May 5, 2026

Explaining and Preventing Alignment Collapse in Iterative RLHF

Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on which the RM is retrained, creating a feedback loop. Building on the Stackelberg game formulation of this interaction, we derive an analytical decomposition of the policy's true optimization gradient into a standard policy gradient and a parameter-steering term that captures the policy's influence on the RM's future parameters. We show that standard iterative RLHF, which drops this steering term entirely, suffers from alignment collapse: the policy systematically exploits the RM's blind spots, producing low-quality, high-reward outputs whose feedback reinforces the very errors it exploits. To mitigate this, we propose foresighted policy optimization (FPO), a mechanism-design intervention that restores the missing steering term by regularizing the policy's parameter-steering effect on RM updates. We instantiate FPO via a scalable first-order approximation and demonstrate that it prevents alignment collapse on both controlled environments and an LLM alignment pipeline using Llama-3.2-1B.

Etienne Gauthier, Francis Bach, Michael I. Jordan

Inria, Ecole Normale Sup´erieure, PSL Research University · Ecole Normale Sup´erieure, PSL Research University · PSL Research University +1

Reward HackingPolicy GradientRL from Human Feedback
AskPDF
Mar 24, 2026

Improving Safety Alignment via Balanced Direct Preference Optimization

With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. However, safety alignment still suffers from severe overfitting, which limits its actual performance. This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pairs, which compromises the model's safety performance. To address this, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on mutual information. A series of experimental results show that B-DPO can enhance the safety capability while maintaining the competitive general capabilities of LLMs on various mainstream benchmarks compared to state-of-the-art methods. \color{red}{Warning: This paper contains examples of harmful texts, and reader discretion is recommended.

Shiji Zhao, Mengyang Wang, Shukun Xiong +7

Institute of Artificial Intelligence, Beihang University, Beijing, China.

LLM Safety AlignmentLLM AlignmentDirect Preference Optimization
AskPDF