Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning

Jiahao Zhang; Lujing Zhang; Keltin Grimes; Zhuohao Yu; Gokul Swamy; Zhiwei Steven Wu

cs.LGFeb 22, 2026

Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning

Authors: Jiahao Zhang, Lujing Zhang, Keltin Grimes, Zhuohao Yu, Gokul Swamy, Zhiwei Steven Wu

Organizations: Carnegie Mellon University Pittsburgh, PA 15213

Abstract

A recurring challenge in preference fine-tuning (PFT) is handling $\textit{intransitive}$ (i.e., cyclic) preferences. Intransitive preferences often stem from either $\textit{(i)}$ inconsistent rankings along a single objective or $\textit{(ii)}$ scalarizing multiple objectives into a single metric. Regardless of their source, the downstream implication of intransitive preferences is the same: there is no well-defined optimal policy, breaking a core assumption of the standard PFT pipeline. In response, we propose a novel, game-theoretic solution concept, the $\textit{Maximum Entropy Blackwell Winner}$ ( $\textit{MaxEntBW}$ ), that is well-defined under multi-objective intransitive preferences. To enable computing MaxEntBWs at scale, we derive $\texttt{PROSPER}$ : a provably efficient PFT algorithm. Unlike prior self-play techniques, $\texttt{PROSPER}$ directly handles multiple objectives without requiring scalarization. We then apply $\texttt{PROSPER}$ to the problem of fine-tuning large language models (LLMs) from multi-objective LLM-as-a-Judge feedback (e.g., rubric-based judges), a setting where both sources of intransitivity arise. We find that $\texttt{PROSPER}$ outperforms all baselines considered across both instruction following and general chat benchmarks, releasing trained model checkpoints at the 7B and 3B parameter scales.

Figures & tables

Figure 1: We study the problem of learning from multiple preferences over different objectives or criteria, each of which might be intransitive (i.e., inconsistent). Such preferences are common when learning from large language model (LLM) “judges” ( Zheng et al., 2023 ) that evaluate a response along multiple objectives. We propose a novel solution concept for multi-object preference fine-tuning (PFT), the Maximum Entropy Blackwell Winner (MaxEntBW) , that remains well-defined under multiple intransitive preferences. We then derive PROSPER : a regression-based algorithm for computing MaxEntBWs, before using it to fine-tune LLMs from multi-objective LLM judge feedback on multiple problems.

\widehat{g_{t}}(x,z)=\frac{\sum_{j=1}^{M}\mathcal{P}^{\widehat{k}(x)}(z\succ y_{j}^{\prime}\mid x)\exp{\mathchoice{\left(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x)\right)}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}}}{\sum_{j=1}^{M}\exp{\mathchoice{\left(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x)\right)}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}{(-\frac{1}{M\beta}\sum_{i=1}^{M}\mathcal{P}^{\widehat{k}(x)}(y_{i}\succ y_{j}^{\prime}\mid x))}}}.

Algorithm 1 PROSPER : PReference Optimization with a Single Player over Entire Rubrics

Figure 2: On a held-out set of prompts from WildChecklists , we report the fraction of prompts with no Condorcet Winner (left) and Intransitive Preferences (right) for both the

\mathcal{P}_{\mathsf{JC}}

(joint check) and

\mathcal{P}_{\mathsf{SC}}

(single check) judges, where

N

denotes the number of generated responses. We see that splitting up a rubric into multiple items before passing it to the LLM judge (i.e., using

\mathcal{P}_{\mathsf{SC}}

rather than

\mathcal{P}_{\mathsf{JC}}

) reduces but doesn’t eliminate inconsistent preferences.

Figure 3: We consistently see policies trained via PROSPER outperform RLCF (roughly $2/3$ of the time), baseline (roughly $3/4$ of the time), and ablation method policies, as measured by LLM judge win-rates on held-out prompts. This indicates that PROSPER is able to more effectively optimize nuanced, multi-criteria LLM judge preferences than the other methods we consider.

Model	Arena-Hard		AlpacaEval
	Vanilla	Style-Controlled	Vanilla	Length-Controlled
Qwen2.5-7B-Instruct	42.4	44.2	37.1	25.32
+ RLCF	42.5	43.9	41.4	17.24
+ PROSPER-JC	44.2	42.0	55.3	38.21
+ PROSPER-VB	47.6	45.5	51.2	33.64
+ PROSPER	49.2	46.1	55.4	37.61

Table 1: Highest score is bolded and second highest underlined . Across instruction following and general chat benchmarks, PROSPER consistently outperforms the RLCF baseline and PROSPER-JC and PROSPER-VB baselines at the 7B parameter scale.

Model	IFEval	MMLU	ARC	HellaSwag	TruthfulQA	Avg
Qwen2.5-7B-Instruct	71.16	63.39	85.23	74.54	68.42	72.55
+ RLCF	67.83	63.47	85.07	74.18	65.97	71.30
+ PROSPER-JC	69.87	63.37	85.41	74.18	65.61	71.69
+ PROSPER-VB	73.75	63.36	85.32	74.04	65.6	72.32
+ PROSPER	69.50	63.42	85.49	74.12	65.97	71.70

Table 2: Highest score is bolded and second highest underlined . By comparing the first and last row of the table, we see that PROSPER does not lead to significant model commonsense reasoning and general knowledge QA capability degradation at the 7B parameter scale.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Arena-Hard		AlpacaEval
	Vanilla	Style-Controlled	Vanilla	Length-Controlled
Qwen2.5-3B-Instruct	20.6	19.6	24.1	12.79
+ RLCF	22.8	22.9	26.2	11.62
+ PROSPER-JC	22.9	23.4	24.7	7.4
+ PROSPER-VB	25.1	23.6	34.8	19.99
+ PROSPER	26.0	24.6	35.8	20.94

Appendix

Table 3: Model performances on instruction-following and general-chat benchmarks (fine tuning Qwen2.5-3B-Instruct). Highest score presented in bold and second highest underlined . Across instruction-following and preference-alignment benchmarks (AlpacaEval 2.0 and Arena-Hard) evaluated using gpt-5-mini as the judge, PROSPER consistently outperforms the PROSPER ablations (joint-check and variational-bound), while also improving over the RLCF baseline.

Model	IFEval	MMLU	ARC	HellaSwag	TruthfulQA	Avg
Qwen2.5-3B-Instruct	59.70	56.45	79.43	67.68	64.62	65.58
+ RLCF	58.04	56.28	79.35	66.88	62.17	64.54
+ PROSPER-JC	53.24	56.16	79.27	66.87	61.69	63.45
+ PROSPER-VB	57.30	56.34	79.52	67.36	62.67	64.64
+ PROSPER	58.78	56.31	79.27	67.14	62.18	64.74

Appendix

Table 4: Model performance on instruction, knowledge, and commonsense benchmarks. The highest score is shown in bold and the second-highest is underlined . Across these benchmarks, post-training methods incur small performance drops; among them, PROSPER ranks first on Qwen2.5-3B-Instruct (by average score).

Model	Arena-Hard		AlpacaEval
	Vanilla	Style-Controlled	Vanilla	Length-Controlled
Qwen2.5-7B-Instruct	42.4	44.2	37.1	25.32
+ RLCF (Released)	42.2	44.6	42.3	28.08

Appendix

Table 9

Parameter	Value
Temperature	0.8
Top- $p$	0.9
Max new tokens	2048
Hardware	$1\times(16\times\text{NVIDIA H100 80GB})$
Total compute time	$\approx 2$ hours per model per epoch

Appendix

Table 5: Hyperparameters and compute configuration for response generation.

Parameter	Value
Temperature	0.8
Top- $p$	0.9
Top- $k$	20
Max new tokens	256
Selection pairs	4
Base pairs	2

Appendix

Table 6: Hyperparameters and compute configuration for score generation.

Parameter	Value
Batch size	128
Max input length	1024
Max sequence length	2048
Optimizer	AdamW
AdamW $\epsilon$	$1\times 10^{-8}$
Learning rate	$3\times 10^{-7}$

Appendix

Table 7: Hyperparameters and compute configuration for policy training.

Figure 4: Prompt for generating preference score according to specific criteria.

Figure 5: Prompt for generating preference score according to multiple criterion.

Blackwell Approachability

See all 10 papers

Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
Apr 20, 2026Wei Chen, Yubing Wu, Junmei Yang +5Pairwise Preference Learning LLM Alignment
1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1
Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

Aug 26, 2026J/K move · Enter open · S save

Multiple latent orderings better predict language model preferences

Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young

Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403

Preference Modeling Pairwise Preference Learning Bradley–Terry Model

Open paper AskPDF arXiv

Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item

A

B

and

B

C

, while also preferring

C

A

. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.

Abstract

Blackwell Approachability

Explore similar work

Multiple latent orderings better predict language model preferences

Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution