A recurring challenge in preference fine-tuning (PFT) is handling intransitive (i.e., cyclic) preferences. Intransitive preferences often stem from either (i) inconsistent rankings along a single objective or (ii) scalarizing multiple objectives into a single metric. Regardless of their source, the downstream implication of intransitive preferences is the same: there is no well-defined optimal policy, breaking a core assumption of the standard PFT pipeline. In response, we propose a novel, game-theoretic solution concept, the Maximum Entropy Blackwell Winner (MaxEntBW), that is well-defined under multi-objective intransitive preferences. To enable computing MaxEntBWs at scale, we derive PROSPER: a provably efficient PFT algorithm. Unlike prior self-play techniques, PROSPER directly handles multiple objectives without requiring scalarization. We then apply PROSPER to the problem of fine-tuning large language models (LLMs) from multi-objective LLM-as-a-Judge feedback (e.g., rubric-based judges), a setting where both sources of intransitivity arise. We find that PROSPER outperforms all baselines considered across both instruction following and general chat benchmarks, releasing trained model checkpoints at the 7B and 3B parameter scales.
Figures & tables
Figure 1: We study the problem of learning from multiple preferences over different objectives or criteria, each of which might be intransitive (i.e., inconsistent). Such preferences are common when learning from large language model (LLM) “judges” ( Zheng et al., 2023 ) that evaluate a response along multiple objectives. We propose a novel solution concept for multi-object preference fine-tuning (PFT), the Maximum Entropy Blackwell Winner (MaxEntBW) , that remains well-defined under multiple intransitive preferences. We then derive PROSPER : a regression-based algorithm for computing MaxEntBWs, before using it to fine-tune LLMs from multi-objective LLM judge feedback on multiple problems.
Algorithm 1 PROSPER : PReference Optimization with a Single Player over Entire Rubrics
Figure 2: On a held-out set of prompts from WildChecklists , we report the fraction of prompts with no Condorcet Winner (left) and Intransitive Preferences (right) for both the PJC (joint check) and PSC (single check) judges, where N denotes the number of generated responses. We see that splitting up a rubric into multiple items before passing it to the LLM judge (i.e., using PSC rather than PJC ) reduces but doesn’t eliminate inconsistent preferences.
Figure 3: We consistently see policies trained via PROSPER outperform RLCF (roughly 2/3 of the time), baseline (roughly 3/4 of the time), and ablation method policies, as measured by LLM judge win-rates on held-out prompts. This indicates that PROSPER is able to more effectively optimize nuanced, multi-criteria LLM judge preferences than the other methods we consider.
Model
Arena-Hard
AlpacaEval
Vanilla
Style-Controlled
Vanilla
Length-Controlled
Qwen2.5-7B-Instruct
42.4
44.2
37.1
25.32
+ RLCF
42.5
43.9
41.4
17.24
+ PROSPER-JC
44.2
42.0
55.3
38.21
+ PROSPER-VB
47.6
45.5
51.2
33.64
+ PROSPER
49.2
46.1
55.4
37.61
Table 1: Highest score is bolded and second highest underlined . Across instruction following and general chat benchmarks, PROSPER consistently outperforms the RLCF baseline and PROSPER-JC and PROSPER-VB baselines at the 7B parameter scale.
Model
IFEval
MMLU
ARC
HellaSwag
TruthfulQA
Avg
Qwen2.5-7B-Instruct
71.16
63.39
85.23
74.54
68.42
72.55
+ RLCF
67.83
63.47
85.07
74.18
65.97
71.30
+ PROSPER-JC
69.87
63.37
85.41
74.18
65.61
71.69
+ PROSPER-VB
73.75
63.36
85.32
74.04
65.6
72.32
+ PROSPER
69.50
63.42
85.49
74.12
65.97
71.70
Table 2: Highest score is bolded and second highest underlined . By comparing the first and last row of the table, we see that PROSPER does not lead to significant model commonsense reasoning and general knowledge QA capability degradation at the 7B parameter scale.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Arena-Hard
AlpacaEval
Vanilla
Style-Controlled
Vanilla
Length-Controlled
Qwen2.5-3B-Instruct
20.6
19.6
24.1
12.79
+ RLCF
22.8
22.9
26.2
11.62
+ PROSPER-JC
22.9
23.4
24.7
7.4
+ PROSPER-VB
25.1
23.6
34.8
19.99
+ PROSPER
26.0
24.6
35.8
20.94
Appendix
Table 3: Model performances on instruction-following and general-chat benchmarks (fine tuning Qwen2.5-3B-Instruct). Highest score presented in bold and second highest underlined . Across instruction-following and preference-alignment benchmarks (AlpacaEval 2.0 and Arena-Hard) evaluated using gpt-5-mini as the judge, PROSPER consistently outperforms the PROSPER ablations (joint-check and variational-bound), while also improving over the RLCF baseline.
Model
IFEval
MMLU
ARC
HellaSwag
TruthfulQA
Avg
Qwen2.5-3B-Instruct
59.70
56.45
79.43
67.68
64.62
65.58
+ RLCF
58.04
56.28
79.35
66.88
62.17
64.54
+ PROSPER-JC
53.24
56.16
79.27
66.87
61.69
63.45
+ PROSPER-VB
57.30
56.34
79.52
67.36
62.67
64.64
+ PROSPER
58.78
56.31
79.27
67.14
62.18
64.74
Appendix
Table 4: Model performance on instruction, knowledge, and commonsense benchmarks. The highest score is shown in bold and the second-highest is underlined . Across these benchmarks, post-training methods incur small performance drops; among them, PROSPER ranks first on Qwen2.5-3B-Instruct (by average score).
Model
Arena-Hard
AlpacaEval
Vanilla
Style-Controlled
Vanilla
Length-Controlled
Qwen2.5-7B-Instruct
42.4
44.2
37.1
25.32
+ RLCF (Released)
42.2
44.6
42.3
28.08
Appendix
Table 9
Parameter
Value
Temperature
0.8
Top- p
0.9
Max new tokens
2048
Hardware
1×(16×NVIDIA H100 80GB)
Total compute time
≈2 hours per model per epoch
Appendix
Table 5: Hyperparameters and compute configuration for response generation.
Parameter
Value
Temperature
0.8
Top- p
0.9
Top- k
20
Max new tokens
256
Selection pairs
4
Base pairs
2
Appendix
Table 6: Hyperparameters and compute configuration for score generation.
Parameter
Value
Batch size
128
Max input length
1024
Max sequence length
2048
Optimizer
AdamW
AdamW ϵ
1×10−8
Learning rate
3×10−7
Appendix
Table 7: Hyperparameters and compute configuration for policy training.
Figure 4: Prompt for generating preference score according to specific criteria.
Figure 5: Prompt for generating preference score according to multiple criterion.
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.
Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based methods also suppress the chosen response when they try to suppress the rejected one, and there is no general way to prevent this across different objectives. We address this issue with a unified incentive-score decomposition of preference optimization, revealing that different objectives share the same local update directions and differ only in their scalar weights. This decomposition provides a common framework for analyzing objectives that were previously studied in separate settings. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the disentanglement band (DB), a simple, testable condition that tells us when training can follow the desired path: suppress the loser while preserving the winner, possibly after an early stage. Using the DB, we propose reward calibration (RC), a plug-and-play method that adaptively rebalances the updates for chosen and rejected responses to satisfy the DB, without redesigning the base objective. Empirical results show that RC leads to more disentangled dynamics, with better downstream performance observed across several settings. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.
Wei Chen, Yubing Wu, Junmei Yang +5
1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1
Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.
Minjae Lee, Kyunghyun Cho, Sangdon Park
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH