Divergence based Ranked Preference Alignment for LLMs

Rajdeep Haldar; Yue Xing; Qifan Song

cs.LGOct 18, 2026

Divergence based Ranked Preference Alignment for LLMs

Authors: Rajdeep Haldar, Yue Xing, Qifan Song

Organizations: Purdue University · Michigan State University

Abstract

Alignment of large language models typically relies on pairwise preference feedback, though ranked feedback can capture more comprehensive information across multiple responses within a single annotation. However, ranked-preference alignment is underexplored in practice compared to simpler pairwise approaches, potentially resulting in suboptimal behavior. Concurrently, recent research has conceptualized large language model alignment objectives as divergence estimators between aligned (preferred) and unaligned (less-preferred) response distributions, resulting in a family of objectives with an interpretable theoretical foundation and strong empirical performance. This work unifies these perspectives by introducing $f$ -RPO, a $f$ -divergence objective for ranked preference optimization that extends divergence-based alignment to ranked feedback and recovers pairwise preference alignment as a special case. Our theoretical analysis establishes rank-order consistency, showing that $f$ -RPO preserves the underlying preference ordering at convergence. Empirical results indicate that total-variation and reverse-KL variants of the proposed objective consistently outperform ranked-alignment baselines in both ranking and generation quality across Qwen, Llama, & Gemma models ranging from 0.5B to 8B parameters.

Figures & tables

Figure 1: Overview of $f$ -RPO. Ranked feedback implicitly induces aligned and unaligned response distributions. Alignment then scales the reference policy to reflect ranking.

Divergence	$\mathcal{L}_{f\text{-RPO}}(\theta)$
Reverse KL	$\displaystyle\mathbb{E}_{x,\mathbf{y}_{(1:k)}}\sum_{i=1}^{k-1}\rho_{x}(i)\left[e^{-r_{\theta,i}}+\sum_{j=i+1}^{k}q_{x}(j\mid i)\,r_{\theta,j}\right]$
Total variation	$\displaystyle\frac{1}{2}\mathbb{E}_{x,\mathbf{y}_{(1:k)}}\sum_{i=1}^{k-1}\rho_{x}(i)\left[-\sigma(r_{\theta,i})+\sum_{j=i+1}^{k}q_{x}(j\mid i)\sigma(r_{\theta,j})\right]$

Table 1: Realization of

f

-RPO objective for TV and RKL divergence.

Figure 2: comparison of ranked alignment methods. We report improvement over the corresponding unaligned base model in Spearman correlation (left) and Top-1 accuracy (right). Error bars denote

\pm

one standard error over five training seeds.

Method	Qwen 1.5B	Llama 8B	Gemma 2B	Llama 3B
$f$ -RPO-RKL	70.10	51.18	69.13	88.26
$f$ -RPO-TV	81.24	55.27	65.84	94.84
Mallows-RCPO	58.80	50.80	61.73	83.16
PL-DPO	56.50	48.57	51.30	68.91

Table 2: AlpacaEval win rate (%) against the corresponding base model. Best and second-best results in each column are bold and underlined , respectively.

Figure 3: Effect of ranking depth

k

on Qwen2.5-1.5B. Bars show ranking performance for PL-DPO and

f

-RPO-TV, while the overlaid curves highlight the trend across increasing

k

Divergence	Spearman $\rho$	Pairwise (%)	Top-1 (%)
Reverse KL	0.1648 (0.0039)	56.70 (0.16)	28.94 (0.10)
Total variation	0.1492 (0.0017)	56.16 (0.08)	28.08 (0.07)
KL	0.0928 (0.0019)	53.98 (0.08)	26.94 (0.29)
Pearson $\chi^{2}$	0.1048 (0.0012)	54.40 (0.05)	27.70 (0.26)
Squared Hellinger	0.0937 (0.0005)	53.96 (0.03)	27.80 (0.18)
Jensen–Shannon	0.1044 (0.0010)	54.41 (0.05)	27.97 (0.16)

Table 3: Ablation on the choice of

f

-divergence on ranking performance. Mean (SE) reported over five seeds. Best and second-best aligned results are bold and underlined , respectively. Model: Qwen2.5-0.5B.

Method	Qwen2.5 0.5B	Qwen2.5 1.5B	Qwen2.5 7B
Base	0.0314	-0.0431	0.4390
DPO (pairwise)	0.0996 (0.0014)	0.0420 (0.0041)	0.4480 (0.0005)
PL-DPO (ranked)	0.1131 (0.0017)	0.0471 (0.0045)	0.4530 (0.0006)
$f$ -RPO-TV (ranked)	0.1492 (0.0017)	0.1288 (0.0035)	0.4858 (0.0005)
$f$ -RPO-RKL (ranked)	0.1648 (0.0039)	0.1538 (0.0068)	0.4827 (0.0008)
$\Delta$ PL-DPO $-$ DPO	+0.0135	+0.0051	+0.0050

Table 4: Pairwise versus ranked preference alignment under the same optimization-step budget. PL-DPO is the ranked counterpart of DPO. Entries report mean Spearman correlation

\rho

(SE) over five seeds.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Divergence	$f(t)$	$f^{*}(u)$	$\operatorname{dom}(f^{*})$	$g_{f}(r)$
KL	$t\log t$	$e^{u-1}$	$\mathbb{R}$	$r$
Reverse KL	$-\log t$	$-1-\log(-u)$	$u<0$	$-e^{-r}$
Pearson $\chi^{2}$	$(t-1)^{2}$	$\frac{1}{4}u^{2}+u$	$\mathbb{R}$	$r$
Hellinger	$(\sqrt{t}-1)^{2}$	$\dfrac{u}{1-u}$	$u<1$	$1-e^{-r}$
Jensen–Shannon	$t\log t-(t+1)\log\frac{t+1}{2}$	$-\log(2-e^{u})$	$u<\log 2$	$\log 2-\log(1+e^{-r})$
Total variation	$\frac{1}{2}\|t-1\|$	$u$	$\|u\|\leq\frac{1}{2}$	$\frac{1}{2}(1+e^{-r})^{-1}$

Appendix

Table 5: Common

f

-divergences, convex conjugates, effective domains, and canonical link functions used in our objectives.

Method	Feedback	Aligned $\mathcal{D}^{+}$	Unaligned $\mathcal{D}^{-}$
FDO	Pairwise preference	Chosen responses	Rejected responses
$f$ -GRPO	Scalar reward	Positive-advantage responses	Non-positive-advantage responses
$f$ -HAL	Pairwise Preference $+$ reward	Chosen $+$ positive-advantage	Rejected $+$ non-positive-advantage
$f$ -RPO (ours)	Ranked preference	Higher-ranked responses	Lower-ranked responses

Appendix

Table 6: Existing divergence-based alignment objectives differ primarily in how the aligned and unaligned response distributions are defined.

Divergence	$f(t)$	$g_{f}(r)$	$\mathcal{L}_{f\text{-RPO}}(\theta)$	$g_{f}^{-1}(f^{\prime}(t))$
Reverse KL	$-\log t$	$-e^{-r}$	$\displaystyle\mathbb{E}\sum_{i}\rho_{x}(i)\left[e^{-r_{\theta,i}}+\sum_{j>i}q_{x}(j\mid i)r_{\theta,j}\right]$	$\log t$
Total variation	$\frac{1}{2}\|t-1\|$	$\frac{1}{2}\sigma(r)$	$\displaystyle\frac{1}{2}\mathbb{E}\sum_{i}\rho_{x}(i)\left[-\sigma(r_{\theta,i})+\sum_{j>i}q_{x}(j\mid i)\sigma(r_{\theta,j})\right]$	$\displaystyle-\log\!\left((\operatorname{sign}(t-1))^{-1}-1\right)$
KL	$t\log t$	$r$	$\displaystyle\mathbb{E}\sum_{i}\rho_{x}(i)\left[-r_{\theta,i}+\sum_{j>i}q_{x}(j\mid i)e^{r_{\theta,j}-1}\right]$	$\log t+1$
Pearson $\chi^{2}$	$(t-1)^{2}$	$r$	$\displaystyle\mathbb{E}\sum_{i}\rho_{x}(i)\left[-r_{\theta,i}+\sum_{j>i}q_{x}(j\mid i)\left(\frac{1}{4}r_{\theta,j}^{2}+r_{\theta,j}\right)\right]$	$2(t-1)$
Squared Hellinger	$(\sqrt{t}-1)^{2}$	$1-e^{-r}$	$\displaystyle\mathbb{E}\sum_{i}\rho_{x}(i)\left[e^{-r_{\theta,i}}+\sum_{j>i}q_{x}(j\mid i)e^{r_{\theta,j}}\right]$	$\frac{1}{2}\log t$
Jensen–Shannon	$\displaystyle t\log t-(t+1)\log\frac{t+1}{2}$	$\displaystyle\log 2-\log(1+e^{-r})$	$\displaystyle\mathbb{E}\sum_{i}\rho_{x}(i)\left[-\log\!\bigl(2\sigma(r_{\theta,i})\bigr)-\sum_{j>i}q_{x}(j\mid i)\log\!\bigl(2\sigma(-r_{\theta,j})\bigr)\right]$	$\log t$

Appendix

Table 7:

f

-divergence instantiations considered in this work. The final column gives the population scaling

g_{f}^{-1}(f^{\prime}(t))

, where

t=d\mathcal{D}^{+}/d\mathcal{D}^{-}

. Terms independent of

\theta

are omitted from the displayed objectives.

Generator	Temperature	Top- $p$	Max. new tokens
Qwen2.5-14B-Instruct	0.25	0.90	1,536
Qwen2.5-7B-Instruct	0.70	0.90	1,536
Qwen2.5-7B-Instruct	1.00	0.95	1,536
Llama-3.1-8B-Instruct	0.70	0.90	1,536
Qwen2.5-1.5B-Instruct	0.80	0.95	1,536

Appendix

Table 8: Models and decoding parameters used to generate the five response candidates.

Candidate	Mean reward	Mean rank	Top-1 (%)	Bottom (%)
Qwen 14B, conservative	18.67	2.31	31.76	2.83
Qwen 7B, balanced	17.58	2.55	22.08	3.30
Llama 8B, balanced	17.32	2.68	28.23	8.85
Qwen 7B, diverse	16.74	2.82	16.19	5.27
Qwen 1.5B, weaker	4.02	4.64	1.74	79.76

Appendix

Table 9: Reward and rank statistics before high-confidence filtering, computed over the 3,854 complete five-way rows. Lower mean rank is better.

Source	Train	Validation	Test	Total
Evol-Instruct	395	53	82	530
FLAN	365	59	93	517
FalseQA	253	33	57	343
ShareGPT	303	49	62	414
UltraChat	359	41	75	475
Total	1,675	235	369	2,279

Appendix

Table 10: Composition of the high-confidence dataset used in all main experiments.

Hyperparameter	Value
Training epochs	1
Random seeds	$\{1,2,3,4,5\}$
Learning rate	$5\times 10^{-5}$
Optimizer	8-bit AdamW
Weight decay	0.1
LR schedule / warmup	cosine / 0.1 fraction

Appendix

Table 11: Training hyperparameters shared across methods unless noted.

Method	Spearman $\rho$ ( $\uparrow$ )	Kendall $\tau$ ( $\uparrow$ )	Pairwise (%) ( $\uparrow$ )	Top-1 (%) ( $\uparrow$ )	Exact match (%) ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )
$f$ -RPO-RKL	0.1538 (0.0068)	0.1316 (0.0058)	56.58 (0.29)	27.32 (0.10)	2.82 (0.11)	1.4298 (0.0088)	1.7224 (0.0085)
$f$ -RPO-TV	0.1288 (0.0035)	0.1079 (0.0026)	55.39 (0.13)	26.72 (0.24)	2.87 (0.07)	1.4448 (0.0037)	1.7521 (0.0040)
PL-DPO	0.0471 (0.0045)	0.0441 (0.0036)	52.21 (0.18)	24.66 (0.30)	2.38 (0.05)	1.5261 (0.0047)	1.8431 (0.0052)
Mallows-RCPO	0.0394 (0.0111)	0.0375 (0.0090)	51.88 (0.45)	24.12 (0.59)	2.38 (0.05)	1.5343 (0.0115)	1.8533 (0.0122)
Base	-0.0431	-0.0287	48.56	20.33	1.36	1.6260	1.9557

Appendix

Table 12: Expanded ranking performance across alignment method for Qwen2.5-1.5B. Parentheses denote standard errors over five seeds; the base model is evaluated deterministically. Best and second-best aligned results in each column are bold and underlined , respectively. Shaded rows denote our

f

-RPO variants.

Method	Spearman $\rho$ ( $\uparrow$ )	Kendall $\tau$ ( $\uparrow$ )	Pairwise (%) ( $\uparrow$ )	Top-1 (%) ( $\uparrow$ )	Exact match (%) ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )
$f$ -RPO-RKL	0.4827 (0.0008)	0.3951 (0.0009)	69.76 (0.04)	29.76 (0.20)	5.09 (0.18)	1.0363 (0.0012)	1.3261 (0.0018)
$f$ -RPO-TV	0.4858 (0.0005)	0.3989 (0.0006)	69.95 (0.03)	28.40 (0.10)	4.82 (0.13)	1.0383 (0.0015)	1.3260 (0.0018)
PL-DPO	0.4530 (0.0006)	0.3708 (0.0004)	68.54 (0.02)	30.46 (0.14)	4.39 (0.16)	1.0810 (0.0011)	1.3680 (0.0010)
Mallows-RCPO	0.4674 (0.0023)	0.3828 (0.0023)	69.14 (0.12)	29.76 (0.23)	4.28 (0.10)	1.0606 (0.0036)	1.3451 (0.0037)
Base	0.4390	0.3588	67.94	29.27	2.98	1.1003	1.3919

Appendix

Table 13: Expanded ranking performance across alignment method for Qwen2.5-7B. Parentheses denote standard errors over five seeds; the base model is evaluated deterministically. Best and second-best aligned results in each column are bold and underlined , respectively. Shaded rows denote our

f

-RPO variants.

Divergence	Spearman $\rho$	Kendall $\tau$	Pairwise acc. $\%$ .	Top-1 (%)	Exact match	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )
Reverse KL	0.1648 (0.0039)	0.1340 (0.0031)	56.70 (0.16)	28.94 (0.10)	0.0320 (0.0005)	1.4131 (0.0053)	1.7066 (0.0048)
Total variation	0.1492 (0.0017)	0.1233 (0.0016)	56.16 (0.08)	28.08 (0.07)	0.0309 (0.0007)	1.4333 (0.0019)	1.7252 (0.0020)
KL	0.0928 (0.0019)	0.0796 (0.0017)	53.98 (0.08)	26.94 (0.29)	0.0298 (0.0000)	1.4955 (0.0021)	1.7899 (0.0019)
Pearson $\chi^{2}$	0.1048 (0.0012)	0.0879 (0.0010)	54.40 (0.05)	27.70 (0.26)	0.0304 (0.0010)	1.4784 (0.0020)	1.7749 (0.0019)
Squared Hellinger	0.0937 (0.0005)	0.0792 (0.0006)	53.96 (0.03)	27.80 (0.18)	0.0314 (0.0018)	1.4862 (0.0011)	1.7861 (0.0015)
Jensen–Shannon	0.1044 (0.0010)	0.0881 (0.0011)	54.41 (0.05)	27.97 (0.16)	0.0314 (0.0011)	1.4782 (0.0012)	1.7745 (0.0013)

Appendix

Table 14: Expanded

f

-divergence ablation for ranked alignment on Qwen2.5-0.5B base model. Parentheses denote standard errors over five seeds; the base model is evaluated deterministically. Best and second-best aligned results in each column are bold and underlined , respectively. For MAE and RMSE, lower is better.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
May 10, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuMulti-Objective Evolutionary Optimization Multi-Objective Language Model Alignment
The University of Hong Kong, China · Stellaris AI Limited, China
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
Aug 24, 2026Seungyoon Lee, Minhyuk Kim, Jungseob Lee +1LLM Alignment Preference Alignment
Department of Computer Science and Engineering, Korea University

May 10, 2026J/K move · Enter open · S save

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong, China · Stellaris AI Limited, China

Multi-Objective Evolutionary Optimization Multi-Objective Language Model Alignment Preference Alignment

Open paper AskPDF arXiv

Gradient-based preference optimization methods for large language model (LLM) alignment suffer from preference collapse, converging to narrow behavioral modes while neglecting preference diversity. We introduce EvoPref, a multi-objective evolutionary algorithm that maintains populations of Low-Rank Adaptation (LoRA) adapters optimized across helpfulness, harmlessness, and honesty objectives using Non-dominated Sorting Genetic Algorithm II (NSGA-II) selection with archive-based diversity preservation. Our primary contribution is demonstrating that population-based methods discover substantially more diverse alignments than gradient descent. On standard benchmarks, EvoPref improves preference coverage by 18% (median 82.5% vs. 70.0% for ORPO,

p<0.001

, Wilcoxon,

n=30

) and reduces collapse rates by 47% (11.0% vs. 20.6%,

p<0.001

), while achieving competitive alignment quality (median 75.5% RewardBench vs. 75.0% for ORPO,

p<0.05

). We provide theoretical motivation extending recent multi-objective evolutionary algorithm (MOEA) runtime analysis (Dang et al., 2025) suggesting why archive-based methods escape collapse more effectively than single-trajectory optimization. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines (DPO, IPO, KTO, ORPO) with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes, median with IQR) confirm that multi-objective selection with diversity preservation is essential. This work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment.

Abstract

LLM Alignment

Explore similar work

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent

A Zeroth-Order Paradigm for LLM Preference Alignment

Language Chain in Alignment: Cross-lingual Ranking Preference Optimization