TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

Chengtao Jian; Kai Yang

cs.LGOct 20, 2026

TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

Authors: Chengtao Jian, Kai Yang

Organizations: School of Computer Science and Technology, Tongji University, Shanghai, China · Shenzhen Loop Area Institute, Shenzhen, China · Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai, China

Abstract

Direct Preference Optimization (DPO) minimizes empirical risk over observed preference comparisons, implicitly assuming that the training preference distribution matches the deployment preference distribution. In practice, this i.i.d. assumption often fails, as preference distribution shift exhibits a natural hierarchical structure across three levels: group-level shift, prompt-level shift, and preference-level shift. Such multi-level distributional mismatch poses a critical challenge for safe LLM deployment, yet existing robust DPO methods typically either model uncertainty over the joint preference distribution or focus on individual sources of shift, leaving the hierarchical structure of OOD preference shifts underexplored. We propose TPO, a tri-level distributionally robust framework for OOD preference alignment, which jointly models all three levels of distributional uncertainty through a hierarchically structured rectangular ambiguity set. By instantiating it with Wasserstein distance, we derive a tractable and interpretable objective that combines a closed-form preference-level margin penalty, prompt-level adversarial embedding perturbation, and worst-case group reweighting, each governed by an independent radius. We establish finite-sample convergence guarantees and empirically validate the effectiveness of TPO over existing robust DPO methods.

Figures & tables

Methods	Overall	Group-level	Prompt-level	Preference-level
DPO ( Rafailov et al., 2023 )	✗	✗	✗	✗
WDPO / KLDPO ( Xu et al., 2025 )	✓	✗	✗	✗
GRPO ( Ramesh et al., 2024 )	✗	✓	✗	✗
rDPO ( Chowdhury et al., 2024 )	✗	✗	✗	✓
Dr.DPO ( Wu et al., 2025 )	✗	✗	✗	✓
TPO (Ours)	✓	✓	✓	✓

Table 1: Comparison of robust DPO methods by the shift levels explicitly modeled in their objectives. ✓ denotes a dedicated robustness component, while ✗ denotes no explicit level-specific component.

Qwen-2.5-0.5B	ID		Group		Prompt		Preference		Combined		Avg. Rank
Qwen-2.5-0.5B	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg. Rank
DPO	86.87	77.78	81.80	76.15	83.17	76.98	82.93	78.16	82.60	76.40	5.30
Dr.DPO	84.81	81.36	81.65	77.06	82.75	76.84	82.24	75.86	81.27	75.22	6.50
WDPO	84.87	84.13	82.72	78.90	85.08	82.20	84.01	81.46	84.21	78.76	3.20
TPO (Group)	84.90	77.27	82.96	76.76	83.46	78.28	82.31	79.24	82.06	76.11	5.25
TPO (Prompt)	85.93	83.77	82.84	80.43	84.69	81.05	85.02	83.03	84.37	79.65	2.40

Table 2: Average and worst-group preference accuracy (%) under different OOD shifts. The best result is in bold and the second-best is underlined .

	LLaMA-3.2-1B				LLaMA-3.2-3B				LLaMA-3.1-8B
Task	DPO	TPO			DPO	TPO			DPO	TPO
Task	DPO	$\rho_{g}^{S}=0$	$\rho_{g}^{S}=0.01$	$\rho_{g}^{S}=0.05$	DPO	$\rho_{g}^{S}=0$	$\rho_{g}^{S}=0.01$	$\rho_{g}^{S}=0.05$	DPO	$\rho_{g}^{S}=0$	$\rho_{g}^{S}=0.005$	$\rho_{g}^{S}=0.01$
MMLU-Pro	20.0	20.5	20.5	20.2	31.2	31.5	31.5	31.7	36.8	36.7	37.1	37.1
BBH	35.6	35.8	35.9	35.6	45.2	45.6	45.9	45.7	51.1	50.9	51.7	51.3
GPQA	23.2	23.5	23.7	23.3	28.0	28.4	28.3	28.1	30.0	30.2	30.8	30.1
MATH-Hard	8.1	9.4	8.3	8.2	21.4	21.1	20.1	21.0	19.5	20.9	20.7	19.3

Table 3: Performance comparison on Open LLM Leaderboard v2, which evaluates LLMs across six tasks: MMLU-Pro, BBH, GPQA, MATH-Hard, IFEval, and MuSR.

Figure 1: Preference accuracy under preference poisoning.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Group Proportions	Prompt Perturbation	Reward Mixture $\alpha$
train	Skewed	None	$0.2$
val	Skewed	None	$0.2$
id_test	Skewed	None	$0.2$
group_test	Uniform	None	$0.2$
prompt_test	Skewed	Synonym substitution	$0.2$
preference_test	Skewed	None	$0.8$

Appendix

Table 4: Environment construction for the Emotion preference benchmark.

Hyperparameter	Search Range
$\gamma_{g}$	$\{0.1,\ 1.0\}$
$\eta_{\omega}$	$\{0.01,\ 0.03,\ 0.05,\ 0.1\}$
$\rho^{\omega}$	$\{0.1,0.01,\ 0.05\}$
$\rho_{g}^{X}$	$\{0.01,\ 0.001,\ 0.0005,\ 0.0001\}$
$\rho_{g}^{S}$	$\{0.2,0.1,\ 0.01,\ 0.005,\ 0.001\}$

Appendix

Table 5: Hyperparameter search ranges.

Qwen-2.5-0.5B	ID		Global		Prompt		Preference		Combined
Qwen-2.5-0.5B	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc	Avg Acc	Worst Acc
DPO	86.87 $\pm$ 0.32	77.78 $\pm$ 0.87	81.80 $\pm$ 0.15	76.15 $\pm$ 0.92	83.17 $\pm$ 0.46	76.98 $\pm$ 1.10	82.93 $\pm$ 0.49	78.16 $\pm$ 1.99	82.60 $\pm$ 0.92	76.40 $\pm$ 0.51
Dr.DPO	84.81 $\pm$ 0.23	81.36 $\pm$ 1.02	81.65 $\pm$ 0.35	77.06 $\pm$ 0.78	82.75 $\pm$ 0.69	76.84 $\pm$ 1.22	82.24 $\pm$ 0.44	75.86 $\pm$ 1.65	81.27 $\pm$ 0.94	75.22 $\pm$ 0.73
WDPO	84.87 $\pm$ 0.29	84.13 $\pm$ 1.18	82.72 $\pm$ 0.16	78.90 $\pm$ 0.49	85.08 $\pm$ 0.22	82.20 $\pm$ 2.36	84.01 $\pm$ 0.65	81.46 $\pm$ 1.83	84.21 $\pm$ 0.31	78.76 $\pm$ 1.14
TPO (Group)	84.90 $\pm$ 0.32	77.27 $\pm$ 1.52	82.96 $\pm$ 0.23	76.76 $\pm$ 1.40	83.46 $\pm$ 0.32	78.28 $\pm$ 0.87	82.31 $\pm$ 0.22	79.24 $\pm$ 0.12	82.06 $\pm$ 0.23	76.11 $\pm$ 0.88
TPO (Prompt)	85.93 $\pm$ 0.55	83.77 $\pm$ 0.76	82.84 $\pm$ 0.26	80.43 $\pm$ 2.12	84.69 $\pm$ 0.70	81.05 $\pm$ 2.11	85.02 $\pm$ 0.97	83.03 $\pm$ 1.61	84.37 $\pm$ 0.39	79.65 $\pm$ 0.88

Appendix

Table 6: Average and worst-group accuracy (%) on the Emotion preference benchmark.

Panel of Figure 2 — Figure 2: Sensitivity of group-, prompt-, and preference-level robustness to their corresponding radii.

Panel of Figure 3 — Figure 3: Sensitivity of overall robustness to the three robustness radii.

Model	Method	ID		Group		Prompt		Preference		Combined
Model	Method	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst
Qwen2.5-0.5B	DPO	86.87	77.78	81.80	76.15	83.17	76.98	82.93	78.16	82.60	76.40
Qwen2.5-0.5B	TPO	86.95	85.12	84.61	81.65	85.47	82.56	85.44	82.34	85.25	79.94
LLaMA-3.2-1B	DPO	88.75	86.19	88.23	83.44	87.81	85.43	86.91	81.42	87.44	82.61
LLaMA-3.2-1B	TPO	90.57	87.55	89.11	83.50	89.14	86.70	88.25	81.31	88.31	83.19
Qwen2.5-3B	DPO	78.60	77.48	76.04	74.53	77.38	76.92	73.97	72.62	76.92	71.99

Appendix

Table 7: Preference accuracy (%) across model families and scales.

Model	OOD Average			OOD Worst
Model	DPO	TPO	Gain	DPO	TPO	Gain
Qwen2.5-0.5B	82.63	85.19	+2.57	76.92	81.62	+4.70
Qwen2.5-3B	76.08	80.03	+3.95	74.02	77.27	+3.26
LLaMA-3.2-1B	87.60	88.70	+1.11	83.23	83.68	+0.45
LLaMA-3.2-3B	83.98	89.22	+5.24	81.55	86.27	+4.72

Appendix

Table 8: Average OOD performance and absolute gain of TPO over DPO.

Method	MMLU-Pro	BBH	GPQA	MATH-Hard	IFEval	MuSR	Avg.
DPO	20.0	35.6	23.2	8.1	51.3	31.6	28.30
WDPO	19.4	36.1	24.1	8.0	47.9	32.5	28.00
Dr.DPO	19.7	36.4	24.3	8.6	52.1	32.6	28.95
TPO ( $\rho_{g}^{S}=0.01$ )	20.5	35.9	23.7	8.3	53.0	31.2	28.77

Appendix

Table 9: Downstream evaluation on the Open LLM Leaderboard v2 using LLaMA-3.2-1B-Instruct. The best result is shown in bold and the second-best is underlined.

Figure 4: Computational cost on UltraFeedback with LLaMA-3.2-1B-Instruct.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
Sep 7, 2026Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai +1LLM Alignment Preference Alignment
Department of Statistics University of California, Irvine · School of Business George Washington University
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Apr 30, 2026Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili +1LLM Alignment Direct Preference Optimization
Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq · Department of Computer Engineering, University of Kurdistan, Iran · Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia +2
Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Pairwise Preference Learning LLM Alignment
Cornell University · Vanderbilt University

Sep 7, 2026J/K move · Enter open · S save

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

Department of Statistics University of California, Irvine · School of Business George Washington University

LLM Alignment Preference Alignment

Open paper AskPDF arXiv

Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about

42\%

, demonstrating both its effectiveness and efficiency.

Abstract

LLM Alignment

Explore similar work

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

Token-weighted Direct Preference Optimization with Attention