PLOT: Enhancing Preference Learning via Optimal Transport

Liang Zhu; Yuelin Bai; Xiankun Ren; Jiaxi Yang; Lei Zhang; Feiteng Fang; Hamid Alinejad-Rokny; Minghuan Tan

cs.CLApr 2, 2026

PLOT: Enhancing Preference Learning via Optimal Transport

Authors: Liang Zhu, Yuelin Bai, Xiankun Ren, Jiaxi Yang, Lei Zhang, Feiteng Fang, Hamid Alinejad-Rokny, Minghuan Tan,

Organizations: Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Chongqing University · University of Science and Technology of China ·

Abstract

Preference learning in Large Language Models (LLMs) has advanced significantly, yet existing methods remain limited by modest performance gains, high computational costs, hyperparameter sensitivity, and insufficient modeling of global token-level relationships. We introduce PLOT, which enhances Preference Learning in fine-tuning-based alignment through a token-level loss derived from Optimal Transport. By formulating preference learning as an Optimal Transport Problem, PLOT aligns model outputs with human preferences while preserving the original distribution of LLMs, ensuring stability and robustness. Furthermore, PLOT leverages token embeddings to capture semantic relationships, enabling globally informed optimization. Experiments across two preference categories - Human Values and Logic & Problem Solving - spanning seven subpreferences demonstrate that PLOT consistently improves alignment performance while maintaining fluency and coherence. These results substantiate optimal transport as a principled methodology for preference learning, establishing a theoretically grounded framework that provides new insights for preference learning of LLMs.

Figures & tables

	Red Teaming Attack
Method	ZS	PEZ	GBDA	UAT	SFS	GCG
	$n$ = $500$	$n$ = $5$ , $T$ = $500$	$n$ = $5$ , $T$ = $500$	$n$ = $1$ , $k$ = $3$ , $T$ = $100$	$k$ = $5$ , $T$ = $50$	$n$ = $1$ , $T$ = $500$
Llama3.2-3B-Instruct	$31.18$ $\pm$ $1.12$	$21.65$ $\pm$ $0.54$	$20.38$ $\pm$ $0.53$	$19.67$ $\pm$ $0.51$	$38.42$ $\pm$ $1.36$	$47.17$ $\pm$ $1.43$
SFT	$15.38$ $\pm$ $0.92$	$9.74$ $\pm$ $0.62$	$10.11$ $\pm$ $0.72$	$11.42$ $\pm$ $0.92$	$35.35$ $\pm$ $1.72$	$42.26$ $\pm$ $1.75$
w/ PLOT	$11.96$ $\pm$ $0.78$	$7.60$ $\pm$ $0.52$	$8.94$ $\pm$ $0.79$	$9.87$ $\pm$ $0.49$	$30.47$ $\pm$ $0.88$	$40.22$ $\pm$ $1.34$
DPO	$8.46$ $\pm$ $0.81$	$5.45$ $\pm$ $0.25$	$5.85$ $\pm$ $0.37$	$6.75$ $\pm$ $0.20$	$25.75$ $\pm$ $1.24$	$30.08$ $\pm$ $0.82$

Table 1: The Attack Success Rate (ASR) results for various models and preference learning methods under multiple red teaming attack scenarios consistently demonstrate that integrating

\mathcal{L}_{\text{PLOT}}

leads to a significant reduction in ASR across all evaluated models (Llama3.2-3B, Llama3.1-8B, and Qwen2.5-7B) and alignment techniques (SFT, DPO, PRO, and AOT). This highlights the universal effectiveness and robustness of PLOT in enhancing the preference learning capabilities of LLMs.

Method	Human Values			Logic & Problem Solving
	Helpfulness		Humanity	Mathematics		Reasoning	Coding	STEM
	Reward	GPT-4	Humanity	GSM8K	MATH	Reasoning	Coding	STEM
Instruct	$65.53$	$5.51$	$7.80$	$65.49$	$43.89$	$3.80$	$5.45$	$6.50$
DPO	$70.63$	$7.87$	$8.65$	$67.52$	$47.68$	$4.05$	$5.95$	$7.45$
w/ PLOT	$72.14$	$8.74$	$9.15$	$68.36$	$48.28$	$4.50$	$6.35$	$8.25$

Table 2: Comparative evaluation of Instruct, DPO, and PLOT models across Human Values and Logic & Problem Solving, demonstrating consistent superiority of PLOT in multifaceted preference learning domains.

Method	ZS	PEZ	GBDA	UAT	SFS	GCG
Method	$n$ = $500$	$n$ = $5$ , $T$ = $500$	$n$ = $5$ , $T$ = $500$	$n$ = $1$ , $k$ = $3$ , $T$ = $100$	$k$ = $5$ , $T$ = $50$	$n$ = $1$ , $T$ = $500$
DPO	$8.46\pm 0.81$	$5.45\pm 0.25$	$5.85\pm 0.37$	$6.75\pm 0.20$	$25.75\pm 1.24$	$30.08\pm 0.82$
+ $\mathcal{R}_{\mathcal{Q}}$ (DEFT)	$5.65\pm 0.61$	$5.08\pm 0.51$	$5.27\pm 0.41$	$5.25\pm 0.20$	$19.25\pm 1.27$	$28.42\pm 1.12$
+ $\mathcal{L}_{\text{PLOT}}$ w/o $\mathbf{E}$	$4.91\pm 0.30$	$4.83\pm 0.17$	$4.40\pm 0.12$	$5.17\pm 0.12$	$17.83\pm 0.31$	$27.25\pm 0.54$
+ $\mathcal{L}_{\text{PLOT}}$	$4.39\pm 0.20$	$4.23\pm 0.20$	$4.32\pm 0.13$	$4.92\pm 0.24$	$16.92\pm 0.51$	$26.83\pm 0.42$

Table 3: A comparison of ASR across different loss components and experimental settings under various attack methods reveals that transitioning from DEFT to OT-formulated problem yields performance improvements, which are further enhanced by the inclusion of token embeddings.

Method	50%	75%	100%	125%	150%
DEFT	5.75	5.35	5.22	5.92	6.54
PLOT	4.59	4.15	4.12	4.16	4.14

Table 4: ASR under Zero-Shot (n=50) for DEFT and PLOT across varying hyperparameter scales.

Method	ZS ( $n$ = $50$ )	GCG ( $T$ = $50$ )
Instruct	$29.78\pm 2.31$	$29.00\pm 0.54$
DPO	$8.63\pm 0.89$	$14.00\pm 0.35$
w/ PLOT	$4.28\pm 0.09$	$9.17\pm 0.66$
OT $\rightarrow$ KL	inf error	inf error
OT $\rightarrow$ JS	$5.82\pm 0.12$	$12.08\pm 0.59$

Table 5: Compared to other distribution measures, OT achieves enhanced preference learning results, owing to its computational stability and the design of the cost matrix.

Figure 1: Comparison of the LC Win Rate shows that

\mathcal{L}_{\text{PLOT}}

preserves the general capabilities of the model under the original fine-tuning method.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Behavior	#Sample	Source & Description
Standard	200	Based on AdvBench ( 2023 ) and TDC 2023 ( 2023 )
Copyright	100	Manually crafted requests for copyrighted content
Contextual	100	Manually crafted complex requests with context
Total	400	Manually filtered to ensure clearly harmful with no legitimate use

Appendix

Table 6: Details of the HarmBench evaluation dataset.

Figure 2: The ASR curves of three models under different case counts for the Zero-Shot method (Left) and varying update steps of GCG (Right). PLOT consistently demonstrates superior defense capabilities and stability compared to DPO.

Method	DPO	PLOT
Training Time (s)	487.03	500.80
Increase	–	2.67%

Appendix

Table 7: PLOT incurs only a marginal increase in computational cost compared to the original method.

Attack Method

Harmful Behavior Type

Total

Standard

Contextual

Instruct

DPO

PLOT

Instruct

DPO

PLOT

Instruct

DPO

PLOT

Instruct

DPO

PLOT

n=1

26.33

\pm

3.06

4.17

\pm

0.85

1.00

\pm

0.41

13.00

\pm

1.41

6.00

\pm

0.82

5.00

\pm

0.82

50.00

\pm

4.55

18.00

\pm

1.41

8.00

\pm

0.82

28.92

\pm

2.54

8.08

\pm

0.85

3.75

\pm

0.41

n=500

27.28

\pm

1.41

3.50

\pm

0.43

0.47

\pm

0.05

12.97

\pm

0.21

9.20

\pm

0.37

6.33

\pm

0.66

57.22

\pm

1.88

17.63

\pm

2.09

10.30

\pm

1.56

31.18

\pm

1.12

8.46

\pm

0.81

4.39

\pm

0.20

PEZ

n=5

21.87

\pm

0.90

5.37

\pm

0.33

4.73

\pm

0.12

22.60

\pm

0.16

7.13

\pm

0.98

4.20

\pm

0.43

20.27

\pm

1.64

3.93

\pm

0.52

3.27

\pm

0.62

21.65

\pm

0.54

5.45

\pm

0.25

4.23

\pm

0.20

T=500

Appendix

Table 8: Detailed experimental results of Llama3.2-3B-Instruct for the Harmlessness preference learning. Attack Success Rate (ASR) is used to assess the defense capability of the model under various attack methods. PLOT demonstrates significant performance enhancements over DPO across all attack methods, showing improvements in each test data subset as well as the overall performance with better stability.

Preference Learning

Preference learning infers people’s values and trade-offs from choices, comparisons, rankings, and feedback to guide predictions and decisions.

See all 149 papers

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
Apr 14, 2026Shanyong Wang, Shuhang Lin, Yining Zhao +2LLM Fine-Tuning Personalized Language Model Alignment
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1

Apr 14, 2026J/K move · Enter open · S save

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Shanyong Wang, Shuhang Lin, Yining Zhao, Xi Zhu+1

Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign

LLM Fine-Tuning Personalized Language Model Alignment Preference Alignment Value Alignment User Preference Modeling Multi-Objective Language Model Alignment

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.

Abstract

Preference Learning

Explore similar work

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

A Zeroth-Order Paradigm for LLM Preference Alignment