PORT: Preference Optimization via Robust Token-Level Reweighting

Ding Zhu; Xiukun Wei; Tian Xie; Zhihui Zhu; Xueru Zhang; Mohammad Mahdi Khalili

cs.CLNov 9, 2026

PORT: Preference Optimization via Robust Token-Level Reweighting

Authors: Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili

Organizations: Department of Computer Science and Engineering, The Ohio State University

Abstract

Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose Preference Optimization via Robust Token-level reweighting (PORT), a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT

Figures & tables

Figure 1: Overview of the PORT training pipeline: 1) Given a batch

\mathcal{B}=\{(\tilde{x},\tilde{y}^{[b]}_{w},\tilde{y}^{[b]}_{l})\}_{b=1}^{|\mathcal{B}|}

of preference pairs, where

\tilde{y}^{[b]}_{w}

and

\tilde{y}^{[b]}_{l}

denote chosen and rejected responses. 2) Forward pass: the policy

\pi_{\theta}

and reference policy

\pi_{\mathrm{ref}}

are evaluated to obtain per-token log-ratios

\delta_{t}^{[b]}(\theta)

and token-level logits

\tilde{s}_{w,t}^{[b]},\tilde{s}_{l,t}^{[b]}

. 3)

\Delta

CDF estimation: token log-probabilities are pooled across the batch to form a global empirical CDF

F(\cdot)

, which gives percentile ranks

F_{w,t}^{[b]}

and

F_{l,t}^{[b]}

for the chosen and rejected tokens. A large gap indicates that the rejected token is assigned anomalously high probability relative to the chosen token, signaling potential token-level label noise. 4) Weight conversion: gap

\Delta_{t}^{[b]}=F_{l,t}^{[b]}-F_{w,t}^{[b]}

is mapped to a weight

w_{t}^{[b]}=\gamma\exp(\rho\Delta_{t}^{[b]})

. 5) Weighted token-level update: these weights are incorporated into the preference objective

\widetilde{\mathcal{L}}_{w}

, yielding a robust gradient update that suppresses corrupted tokens. The procedure uses only a single forward pass and introduces no extra backward-pass overhead.

Figure 2: Scatter plot of

\Delta

CDF versus token-level gradient discrepancy. Since

\|\nabla\delta_{t}(z;\theta)\|=\|\nabla_{\theta}\pi_{\theta}(y_{w},t|x)-\nabla_{\theta}\pi_{\theta}(y_{l},t|x)\|

, we compute this quantity by taking the SFT gradients of the chosen and rejected responses separately and evaluating the norm of their difference. Across a large population of tokens, an implicit linear trend emerges between the two quantities (Pearson

r=0.4526

n=8{,}752

), suggesting that

\Delta\mathrm{CDF}

may serve as a lightweight proxy for the noise-induced gradient magnitude in Theorem 1 , without requiring access to the unobservable statistics.

Method	LLaMA2-7B		Qwen3-4B
Method	Gold RM Score $\uparrow$	GPT Win Rate (%) $\uparrow$	Gold RM Score $\uparrow$	GPT Win Rate (%) $\uparrow$
SLiC-HF	2.58 $\pm$ 0.11	55.8 $\pm$ 1.2	2.08 $\pm$ 0.07	45.5 $\pm$ 0.8
IPO	2.75 $\pm$ 0.09	57.4 $\pm$ 0.5	2.15 $\pm$ 0.13	46.7 $\pm$ 0.7
DPO	2.66 $\pm$ 0.12	56.8 $\pm$ 0.9	2.18 $\pm$ 0.11	46.3 $\pm$ 1.2
fDPO	3.20 $\pm$ 0.06	60.7 $\pm$ 0.8	2.54 $\pm$ 0.08	48.7 $\pm$ 1.0
SamS	3.68 $\pm$ 0.11	64.8 $\pm$ 0.9	2.84 $\pm$ 0.10	53.8 $\pm$ 1.1

Table 1: Evaluation results on the AlpacaFarm dataset under two model settings. Bold indicates the best performance, and underline denotes the second-best performance. The results is reported under 3 different seed and as the format of mean

\pm

standard deviation.

Method	PKU-Beaver SR $\uparrow$	GPT Win Rate (%) $\uparrow$
SLiC-HF	0.41	56.0
IPO	0.43	57.6
DPO	0.42	56.8
fDPO	0.48	61.8
SamS	0.53	65.8
cDPO	0.48	61.8

Table 2: Results on Anthropic HH dataset (Harmless split). Bold denotes the best performance, underline indicates the second-best performance.

Method	$p=0.2$		$p=0.3$		$p=0.4$
Method	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$
DPO	0.45	59.3	0.42	56.8	0.38	53.4
SamS	0.53	65.9	0.53	65.8	0.46	60.1
$\beta$ -DPO	0.54	66.7	0.51	64.3	0.47	60.9
PORT	0.56	68.5	0.55	67.6	0.53	65.9

Table 3: Ablation of Noise Intensity on Anthropic HH (evaluated by PKU-Beaver SR

\uparrow

). Bold denotes the best performance, underline indicates the second-best performance.

Method	$\epsilon$ -mislabeled noise		$\alpha$ -uncertain noise
Method	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$
DPO	0.38	53.4	0.38	53.5
SamS	0.46	60.1	0.43	57.0
Dr.DPO	0.48	61.8	0.44	58.5
PORT	0.53	65.9	0.49	62.5

Table 4: Impact of Noise Type on Model Performance (evaluated by PKU-Beaver SR

\uparrow

) on Anthropic HH with

p=0.4

noise level. Bold denotes the best performance, underline indicates the second-best performance. PORT achieves consistent gains of

{\sim}4\%

WR and

0.05

SR over the strongest baseline across both noise types.

Panel of Figure 3 — Figure 3: Comparison of per-sample training loss trajectories between standard DPO and our proposed PORT method across $30$ gradient steps. The training samples are partitioned into High $\Delta$ CDF (more reliable) and Low $\Delta$ CDF (more noisy) groups.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Scatter plot of

\Delta

CDF versus token-level gradient discrepancy. Since

\|\nabla h(z;\theta)\|=\|\nabla_{\theta}\pi_{\theta}(y_{w}|x)-\nabla_{\theta}\pi_{\theta}(y_{l}|x)\|

\Delta\mathrm{CDF}

may serve as a lightweight proxy for the noise-induced gradient magnitude

\eta(z)\|\nabla h(z;\theta)\|

in Theorem 1 , without requiring access to the unobservable statistics.

$\rho$ $\backslash$ $\gamma$	$0.5$	$1.0$	$1.5$
$0.1$	0.0804	0.2315	0.2928
$0.3$	0.0921	0.2624	0.3088
$0.5$	0.0797	0.2448	0.2649

Appendix

Table 5: Sensitivity analysis of

\gamma

and

\rho

on LLaMA2-7B. Performance is measured by the mean ground-truth log-probability margin between chosen and rejected responses on a held-out validation set. Higher is better.

Batch Size ( $B$ )	GPT Win Rate (%)
$1$	39.1
$4$	42.8
$16$	44.3

Appendix

Table 6: Effect of batch size on GPT win rate using Qwen2.5-0.5B. The reference set

\mathcal{R}

pools all non-padding chosen and rejected token logits across the batch. Higher is better.

Panel of Figure 5 — Figure 5: Qualitative comparison of PORT and the baseline model on a safety-related prompt from PKU-Beaver. PORT’s response is classified as harmless, whereas the baseline response is classified as harmful.

Method	$\alpha$ -uncertain noise		$\epsilon$ -mislabeled noise
Method	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$
SLiC-HF	0.36	52.0	0.37	52.6
IPO	0.37	52.6	0.39	54.2
DPO	0.38	53.5	0.38	53.4
fDPO	0.41	56.1	0.45	59.3
SamS	0.43	57.0	0.46	60.1

Appendix

Table 7: Impact of Noise Type on Model Performance (evaluated by PKU-Beaver SR

\uparrow

) on Anthropic HH with

p=0.4

noise level. Bold denotes the best performance, underline indicates the second-best performance.

Method	$p=0.2$		$p=0.3$		$p=0.4$
Method	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$	SR $\uparrow$	GPT Win Rate (%) $\uparrow$
SLiC-HF	0.44	58.4	0.41	56.0	0.37	52.6
IPO	0.46	60.1	0.43	57.6	0.39	54.2
DPO	0.45	59.3	0.42	56.8	0.38	53.4
fDPO	0.51	64.2	0.48	61.8	0.45	59.3
SamS	0.53	65.9	0.53	65.8	0.46	60.1

Appendix

Table 8: Ablation of Noise Intensity on Anthropic HH (evaluated by PKU-Beaver SR

\uparrow

). Bold denotes the best performance, underline indicates the second-best performance. We have improved 19.2

\%

compared with the SFT baseline and

2\%

compared with the best SOTA method.

Preference Learning for LLMs

See all 14 papers

Unbiased Alignment for Large Language Models with Noisy Preferences
Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Pairwise Preference Learning LLM Alignment
Harbin Institute of Technology · City University of Hong Kong
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference Learning LLM Alignment
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3

Jul 3, 2026J/K move · Enter open · S save

Unbiased Alignment for Large Language Models with Noisy Preferences

Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu+1

Harbin Institute of Technology · City University of Hong Kong

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Learning with Noisy Labels Preference Learning

Open paper AskPDF arXiv

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.

Abstract

Preference Learning for LLMs

Explore similar work

Unbiased Alignment for Large Language Models with Noisy Preferences

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching