Oracle-Robust Online Alignment for Large Language Models

Zimeng Li; Mudit Gaur; Vaneet Aggarwal

cs.LGFeb 24, 2026

Oracle-Robust Online Alignment for Large Language Models

Authors: Zimeng Li, Mudit Gaur, Vaneet Aggarwal

Organizations: Purdue University

Abstract

We study online alignment of large language models under misspecified preference feedback, where the observed preference oracle deviates from an ideal but unknown ground-truth oracle. The online LLM alignment problem is a bi-level reinforcement problem due to the coupling between data collection and policy updates. Recently, the problem has been reduced to tractable single-level objective in the SAIL (Self-Improving Efficient Online Alignment) framework. In this paper, we introduce a pointwise oracle uncertainty set in this problem and formulate an oracle-robust online alignment objective as a worst-case optimization problem. For log-linear policies, we show that this robust objective admits an exact closed-form decomposition into the original loss function plus an explicit sensitivity penalty. We develop projected stochastic composite updates for the resulting weakly convex objective and prove $\widetilde{O}(\varepsilon^{-2})$ oracle complexity for reaching approximate stationarity.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Distributionally Robust Listwise Preference Optimization
Jul 2, 2026Xudong Wu, Jian Qian, Pangpang Liu +2LLM Alignment Distributionally Robust Optimization
The University of Hong Kong · Hong Kong SAR · Yale University +3
Doubly Robust Alignment for Large Language Models
Jun 1, 2025Erhan Xu, Kai Ye, Hongyi Zhou +3LLM Alignment RL from Human Feedback
Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK +1
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Jul 2, 2026J/K move · Enter open · S save

Distributionally Robust Listwise Preference Optimization

Xudong Wu, Jian Qian, Pangpang Liu, Vaneet Aggarwal+1

The University of Hong Kong · Hong Kong SAR · Yale University+3

LLM Alignment Distributionally Robust Optimization Robust Optimization Robust Loss Functions Preference Optimization

Open paper AskPDF arXiv

Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from

K!

enumeration to

O(K\log K)

. This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global

ε

-suboptimality with

O(ε^{-2})

sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and

\widetilde O(ε^{-2})

Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.

Abstract

LLM Alignment

Explore similar work

Distributionally Robust Listwise Preference Optimization

Doubly Robust Alignment for Large Language Models

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment