Uniform Information Density-Based Preference Optimization

Yongyuan Li; Yang Xu

cs.CLOct 10, 2026

Uniform Information Density-Based Preference Optimization

Authors: Yongyuan Li, Yang Xu

Organizations: Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen, Guangdong, China

Abstract

To enable large language models to generate outputs that align with human preferences, optimization algorithms based on the comparison between chosen and rejected output pairs have been developed and been proven effective. Some recent endeavors have utilized certain linguistically-derived assumptions about the preference pairs to guide the design of more precise optimization directions. However, human preferences are complex and hard to measure by simplistic criteria, and it is important to incorporate broader and more sophisticated insights about preferred vs. not-preferred language. In this study, we adopt a well-established psycholinguistic theory, Uniform Information Density, to develop a novel preference optimization algorithm, UIDPO. The basic assumption is that humans prefer to produce language in which information is uniformly distributed, because uniformity induces less cognitive processing effort. To implement this idea into the algorithm, we derive a compound metric for information uniformity by combining the global standard deviation and the local second-order difference, and integrate it to the objective of preference optimization. We compare UIDPO with SimPO, a previous state-of-the-art method, obtaining comparable and sometimes better performance on AlpacaEval2, ArenaHard, and MT-Bench. Additional linguistic analysis on complement clauses also shows that UIDPO can better reflect the tendencies of keeping information uniform in natural language than baselines. Our results show that the subtle human preference over the information rate in language is a helpful source for achieving better human alignment for LLMs. The code repository is published here: https://github.com/CLCS-SUSTech/UIDPO.

Figures & tables

Figure 1: Comparison of metrics for local information uniformity.

Benchmark	Baseline Model	Judge Model	Metric
AlpacaEval 2	GPT-4 1106 Preview	GPT-4 1106 Preview	LC & raw win rate
AlpacaEval 2*	GPT-4 1106 Preview	Llama-3.3 70B IT	LC & raw win rate
Arena-Hard v0.1	GPT-4 0314	Llama-3.3 70B IT	Win rate
MT-Bench	–	Llama-3.3 70B IT	0-10 rating

Table 1: Detailed benchmark setups. Llama-3.3 70B IT refers to Llama-3.3 70B Instruct Turbo. LC means length controlled win rate. For AlpacaEval2 and Arena-Hard, the baseline responses to be compared with the subject are generated with the baseline model. All the comparison and rating are conducted by the judge model. Note the difference between AlpacaEval 2 and AlpacaEval 2*.

Method	AlpacaEval 2		AlpacaEval 2*		Arena-Hard v0.1	MT-Bench
Method	LC(%)	WR(%)	LC(%)	WR(%)	WR(%)	Score(0-10)
Instruct	26.0	25.3	45.80	43.66	40.8	7.2
SimPO v0.2	53.7	47.5	–	–	–	–
SimPO reprod.	51.9	47.4	60.8	56.1	66.6	7.3
UIDPO	49.0	44.8	63.2	56.2	62.7	7.5

Table 2: Benchmark scores of UIDPO and SimPO. SimPO v0.2 is the best performing SimPO model for reference. Its scores are only available under the “AlpacaEval 2” setup. The row of Instruct is from the Llama-3-8B-Instruct baseline without any post-training. The row of SimPO reprod. is from the reproduced model under exactly the same parameters of SimPO v0.2. LC is the length-controlled win rate, and WR is the raw win rate. Better scores are in bold and underlined.

	Llama-3-8B-Instruct				Mistral-7B-Instruct
$\alpha$	LC(%)	WR(%)	SE	avg. length	LC(%)	WR(%)	SE	avg. length
0.25	63.2	56.2	1.75	1799	34.3	23.2	1.49	1550
0.25 orig.	21.0	25.0	11.4	1893	31.29	33.29	1.66	2895
1.00	32.0	27.1	1.57	1743	9.91	4.66	0.74	812
1.00 orig.	40.8	35.8	1.69	1829	5.4	7.6	0.93	7221
Inst.	45.8	43.7	1.75	1946	19.24	13.42	1.20	1500

Table 3: AlpacaEval 2* scores of UIDPO models following the original UID assumption and the UIDPO assumption. Rows with “orig.” are from models trained under the original UID assumption. The Inst. row is from the Instruct models without any post-training. SE is the standard error (normalized by

N-1

) of the win rate. Avg. length is the average token length of responses. Better scores in the same group are in bold and underlined.

	Llama-3-8B-Instruct				Mistral-7B-Instruct
$\alpha$ / $\frac{\gamma}{\beta}$	LC(%)	WR(%)	SE	avg. length	LC(%)	WR(%)	SE	avg. length
0.000	59.9	52.7	1.92	1839	34.0	32.4	1.65	1970
0.125	60.7	54.2	1.76	1847	34.1	30.7	1.63	1836
0.250	63.2	56.2	1.75	1799	34.3	23.2	1.49	1550
0.375	62.3	55.3	1.75	1816	28.0	21.4	1.45	1616
0.500	41.0	33.6	1.66	1745	25.7	20.0	1.41	1621

Table 4: AlpacaEval 2* scores of models trained with different

\alpha

and

\gamma

. The top six rows are ablations on

\alpha

, and the rests are ablations on

\gamma

(controlled with

\frac{\gamma}{\beta}

). Best scores in the same group are in bold and underlined.

Figure 2: Exhibition of our linguistic analysis method. Assuming the conjunction word “ that ” is at position

k

, whose probability is

p_{k}(\textit{that})

. The probabilities of words before and after are

p_{k-1}(\textit{pre})

and

p_{k+1}(\textit{sub})

, respectively. The probability of a subsequent word at position

k

immediately following the preceding (thus, omitting that ) is

p_{k}(\textit{sub})

. Left curves showcase a necessary that . Right curves showcases a redundant that .

Metric	SFT	SimPO	UIDPO
# of necessary that :
present/correct ( $\uparrow$ )	495 (0%)	617 (24.65%)	676 (36.57%)
absent/mistaken ( $\downarrow$ )	186 (0%)	181 (-2.69%)	191 (2.69%)
# of redundant that :
absent/correct ( $\uparrow$ )	348 (0%)	343 (-1.44%)	345 (-0.86%)
present/mistaken ( $\downarrow$ )	3719 (0%)	3607 (-3.01%)	3536 (-4.92%)

Table 5: The numbers of necessary and redundant that (present vs. absent). All three models have Llama-3-8B-Instruct backbone. Up arrow means the larger the better, and down arrow means the opposite. The percentages in the brackets are the rates of change relative to the SFT baseline. Best scores are in bold.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	LC(%)	WR(%)	Length
RRHF ( Yuan et al., 2023 )	37.9	31.6	1700
SLiC-HF ( Zhao et al., 2023 )	33.9	32.5	1938
DPO ( Rafailov et al., 2024 )	48.2	47.5	2000
IPO ( Azar et al., 2023 )	46.8	42.4	1830
CPO ( Xu et al., 2024 )	34.1	36.4	2086
KTO ( Ethayarajh et al., 2024 )	34.1	32.1	1878

Appendix

Table 6: AlpacaEval 2 results of different post-training methods on Llama-3-8B-Instruct, utilizing SimPO v0.2 preference labels. All the scores are reported in SimPO paper. LC is the length-controlled win rate, and WR is the raw win rate. Length is the average token length of the responses.

answer extraction	Llama-3-8B-Instruct	SimPO run 1	SimPO run 2	UIDPO
Level 1	0.4098	0.3935	0.3910	0.4122
Level 2	0.4098	0.3983	0.3965	0.4122

Appendix

Table 7: MMLU results. Level 1 & 2 are different answer extraction mechanisms. The score of Llama-3-8B-Instruct is retrieved from the Leader-board, where only the best score is reported.

Method	AlpacaEval 2		AlpacaEval 2*		Arena-Hard v0.1
Method	SE	avg. length	SE	avg. length	CI(%)
SimPO reprod.	1.54	1861	1.75	1861	-2.0/+1.8
UIDPO	1.50	1825	1.75	1799	-1.9/+1.7

Appendix

Table 8: Key statistics reported by main experiment benchmarks. SE is the standard error (normalized by

N-1

) of the win rate. Avg. length is the average token length of responses. CI is the non-overlapping confidence interval with the baseline.

Figure 3: Average log probability of chosen responses during training, when $\gamma=0.5,0.9,1.3$ .

Figure 4: The reward margins between chosen responses and rejected responses during training, under the best performing setup.

Figure 5: The global and local UID margins between chosen responses and rejected responses during training, under the best performing setup.

Preference Optimization

Preference optimization aligns AI systems with human or task-specific judgments by training them to favor better-ranked outputs.

See all 150 papers

Intrinsic Mutual Information as a Modulator for Preference Optimization
Apr 27, 2026Peng Liao, Peijia Zheng, Lingbo Li +2LLM Alignment Offline Preference Optimization
1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM Alignment Direct Preference Optimization
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA

Apr 27, 2026J/K move · Enter open · S save

Intrinsic Mutual Information as a Modulator for Preference Optimization

Peng Liao, Peijia Zheng, Lingbo Li, Shangsong Liang+1

1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University

LLM Alignment Offline Preference Optimization LLM Training

Open paper AskPDF arXiv

Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.

Abstract

Preference Optimization

Explore similar work

Intrinsic Mutual Information as a Modulator for Preference Optimization

A Zeroth-Order Paradigm for LLM Preference Alignment

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates