On the Convergence of Self-Improving Online LLM Alignment

Xudong Wu; Pangpang Liu; Vaneet Aggarwal; Jiayu Chen

cs.LGJun 30, 2026

On the Convergence of Self-Improving Online LLM Alignment

Authors: Xudong Wu, Pangpang Liu, Vaneet Aggarwal, Jiayu Chen

Organizations: The University of Hong Kong, Hong Kong SAR · Yale University, New Haven, CT, USA · Purdue University, West Lafayette, IN, USA

Abstract

The Self-Improving Alignment (SAIL) algorithm addresses distribution shift by reducing a bilevel formulation of the problem to an efficient, single-level method. Empirically, SAIL has demonstrated strong performance on this task. However, a formal analysis of its convergence properties has been lacking. We identify a key theoretical challenge: the standard SAIL objective function is not guaranteed to be strongly concave due to unfavorable properties of its Hessian. To address this limitation, we propose a regularized objective, SAIL-RevKL, which incorporates a reverse Kullback-Leibler (KL) divergence penalty to improve the optimization landscape. Our central theoretical contribution is to prove that this regularized objective satisfies the Polyak-Lojasiewicz (PL) condition within a bounded parameter space. We establish global convergence guarantees, achieving a near-linear sample complexity. We further validate the effectiveness and stability of SAIL-RevKL through empirical evaluations, demonstrating that it outperforms the vanilla SAIL on both MuJoCo benchmarks and LLM alignment tasks.

Explore similar work

Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs
May 11, 2026Wu Li, Yigeng Zhou, Zesheng Shi +3Large Language Model Alignment Self-Play
Demystifying the unreasonable effectiveness of online alignment methods
Apr 19, 2026Enoch Hyunwook KangTraining-Free Reward Alignment Kullback-Leibler Regularization
Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Large Language Model Alignment Bregman Divergences

May 11, 2026 · cs.CLJ/K move · Enter open · S save

Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs

Wu Li, Yigeng Zhou, Zesheng Shi, Yequan Wang+2

Harbin Institute of Technology, Shenzhen, China · Beijing Academy of Artificial Intelligence, Beijing, China

While recent self-training approaches have reduced reliance on human-labeled data for aligning LLMs, they still face critical limitations: (i) sensitivity to synthetic data quality, leading to instability and bias amplification in iterative training; (ii) ineffective optimization due to a diminishing gap between positive and negative responses over successive training iterations. In this paper, we propose Team-based self-Play with dual Adaptive Weighting (TPAW), a novel self-play algorithm designed to improve alignment in a fully self-supervised setting. TPAW adopts a team-based framework in which the current policy model both collaborates with and competes against historical checkpoints, promoting more stable and efficient optimization. To further enhance learning, we design two adaptive weighting mechanisms: (i) a response reweighting scheme that adjusts the importance of target responses, and (ii) a player weighting strategy that dynamically modulates each team member's contribution during training. Initialized from a SFT model, TPAW iteratively refines alignment without requiring additional human supervision. Experimental results demonstrate that TPAW consistently outperforms existing baselines across various base models and LLM benchmarks. Our code is publicly available at https://github.com/lab-klc/TPAW.

Large Language Model Alignment Self-Play Large Language Model Fine-Tuning Unsupervised On-Policy Self-Distillation Self-Supervised

Open paper Ask about it PDF

Abstract

Explore similar work

Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs

Demystifying the unreasonable effectiveness of online alignment methods

Theoretical Limits of Language Model Alignment