Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

Austin Tsai; Saurav Nanda; Ganapathy Parthasarathy; Bhuvnesh Kumar; Parivesh Choudhary

cs.LGNov 3, 2026

Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

Authors: Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary

Organizations: Synopsys, Inc., Sunnyvale, CA, USA.

Abstract

Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.

Figures & tables

Figure 1: One DNO-Prct iteration (Algorithm 2 ). Blue: training-time steps performed by GPT-4o in the main runs. Look-back pairs are reused without re-rating; LoRA adapters trained on

\pi_{t}

are merged to give

\pi_{t+1}

Algorithm	Epoch or iter.	LC / raw win rate (%) vs. GPT-4o	LC / raw win rate (%) vs. GPT-4-0613	Avg. length (chars)
DNO	Iter. 1	39.01 / 43.04	61.44 / 68.45	3281
DNO	Iter. 2	41.57 / 44.66	64.87 / 71.37	3626
DNO	Iter. 3	45.32 / 48.26	70.18 / 74.60	3786
DNO-Restrictive	Iter. 3	37.98 / 38.62	60.16 / 60.51	4802
SPIN-style	Epoch 3	35.17 / 41.99	50.30 / 66.81	3207
SPPO	Epoch 3	31.28 / 34.22	45.85 / 59.44	2241

Table 1: Llama 3.1 8B Instruct: DNO versus our reproduced baselines, with GPT-4o as annotator and gold-response source. AlpacaEval 2.0 LC / raw win rate (%) against two opponents, judged by GPT-4o, and average length. The SPIN-style and DNO-Restrictive rows are ablation runs (Table 7 ; full-DNO reference 43.90% LC). Bold: best DNO value per column.

Algorithm	Epoch or iter.	LC / raw win rate (%) vs. GPT-4o	LC / raw win rate (%) vs. GPT-4-0613	Reported by Rosset et al. (%)	Avg. length (chars)
DNO	Iter. 1	9.94 / 10.64	33.62 / 35.12	$\sim$ 10	2424
DNO	Iter. 2	13.62 / 13.45	38.67 / 40.17	$\sim$ 21	5365
DNO	Iter. 3	18.44 / 18.53	38.20 / 37.85	24.97	7506
Offline DPO	Epoch 3	7.05 / 5.85	23.27 / 24.63	18.22	1465
SFT	Epoch 3	5.37 / 4.98	14.76 / 15.69	7.96	1804
SFT	Epoch 10	4.28 / 4.11	13.77 / 14.82	–	1808

Table 2: Mistral 7B Instruct v0.1: DNO versus our reproduced baselines. Columns as in Table 1 ; the Rosset et al. column lists raw win rates from their own Mistral-based run, for reference only (

\sim

: approximate; see Section 5.2 ). Bold: best DNO value per column.

Algorithm	Direct DNO, annotator GPT-4o			Direct DNO, annotator Llama 3.1 70B			SFT + DNO, annotator GPT-4o
Algorithm	vs. GPT-4o	vs. GPT-4-0613	Len.	vs. GPT-4o	vs. GPT-4-0613	Len.	vs. GPT-4o	vs. GPT-4-0613	Len.
DNO Iter. 1	39.01 / 43.04	61.44 / 68.45	3281	37.35 / 44.34	56.95 / 71.08	2763	36.85 / 38.20	50.52 / 65.53	2092
DNO Iter. 2	41.57 / 44.66	64.87 / 71.37	3626	40.01 / 47.45	58.55 / 72.55	3025	38.90 / 44.22	55.06 / 72.42	2285
DNO Iter. 3	45.32 / 48.26	70.18 / 74.60	3786	40.13 / 47.45	58.91 / 71.80	3185	44.92 / 45.14	68.96 / 69.30	4342
Llama 3.1 8B Instruct (base)	31.71 / 35.51	49.64 / 62.19	2237	–	–	–	–	–	–
Llama 3.1 70B Instruct (larger)	41.40 / 39.94	55.23 / 67.64	1873	–	–	–	–	–	–

Table 3: Llama 3.1 8B Instruct under three DNO settings: direct DNO annotated by GPT-4o (the run of Table 1 ), direct DNO annotated by Llama 3.1 70B Instruct, and SFT followed by DNO annotated by GPT-4o. Cells: LC / raw win rate (%); Len.: average length (chars). Bold: best DNO value per column within each block.

Algorithm	Direct DNO, annotator GPT-4o			Direct DNO, annotator Mistral Large			SFT + DNO, annotator GPT-4o
Algorithm	vs. GPT-4o	vs. GPT-4-0613	Len.	vs. GPT-4o	vs. GPT-4-0613	Len.	vs. GPT-4o	vs. GPT-4-0613	Len.
DNO Iter. 1	9.94 / 10.64	33.62 / 35.12	2424	16.40 / 17.36	38.72 / 40.21	8264	10.18 / 10.43	30.83 / 34.53	2078
DNO Iter. 2	13.62 / 13.45	38.67 / 40.17	5365	14.29 / 15.50	39.16 / 38.81	5365	11.43 / 10.68	29.29 / 32.92	1768
DNO Iter. 3	18.44 / 18.53	38.20 / 37.85	7506	16.30 / 17.81	38.71 / 39.09	7699	12.58 / 13.29	30.56 / 37.27	2030
Mistral 7B Instruct v0.1 (base)	6.77 / 5.47	18.40 / 18.66	1375	–	–	–	–	–	–
Mixtral 8x7B Instruct v0.1 (larger)	23.09 / 18.76	41.78 / 50.68	1464	–	–	–	–	–	–

Table 4: Mistral 7B Instruct v0.1 under the three DNO settings of Table 3 , with Mistral Large as the open-weight annotator.

Samples per prompt	LC / raw win rate (%) vs. GPT-4o	Avg. length (chars)	$T\succ S$	$S\succ T$	$S\succ S$	Total	Avg. rating pos. / neg.
1	44.90 / 39.68	3251	4,025	2,708	0	6,733	5.87 / 3.39
3	45.96 / 41.37	2700	14,984	5,569	7,793	28,346	5.84 / 3.41
5	43.90 / 41.77	5324	20,978	14,167	33,379	68,524	5.92 / 3.44
7	44.15 / 38.75	3116	14,145	9,060	45,614	68,819	5.79 / 3.35

Table 5: Number of on-policy samples per prompt (ablation configuration). Selected pairs by type (

T

: teacher’s gold response;

S

: student response); ratings on the 0–6 scale.

Margin	LC / raw win rate (%) vs. GPT-4o	Avg. length (chars)	$T\succ S$	$S\succ T$	$S\succ S$	Total	Share of pairs with teacher positive (%)	Avg. rating pos. / neg.
1	36.60 / 41.23	2978	45,344	24,643	78,777	148,764	30.48	5.78 / 3.70
2	37.72 / 42.92	2931	16,601	13,089	25,712	55,402	29.96	5.81 / 3.39
3	38.99 / 43.54	2619	9,377	6,784	8,765	24,926	37.62	5.79 / 2.01
4	37.38 / 40.98	2551	5,169	4,580	3,826	13,575	38.08	5.82 / 1.42
5	38.13 / 41.93	2542	638	923	254	1,815	35.15	5.97 / 0.92

Table 6: Minimum rating margin (separate pipeline, at most 2,000 training steps per iteration). Columns as in Table 5 , plus the share of pairs whose positive is the teacher’s response.

Variant	vs. GPT-4o (%)			vs. GPT-4-0613 (%)			Avg. length (chars)
Variant	LC	$\Delta$ LC	raw	LC	$\Delta$ LC	raw
DNO (full)	43.90	–	41.77	69.46	–	64.21	5324
Remove $y^{\mathrm{gold}}$	36.97	$-$ 6.93	40.91	55.36	$-$ 14.10	66.15	2648
Mark $T$ always positive	35.17	$-$ 8.73	41.99	50.30	$-$ 19.16	66.81	3207
Accept only $T\succ S$	37.98	$-$ 5.92	38.62	60.16	$-$ 9.30	60.51	4802
Remove data look-back	35.00	$-$ 8.90	41.61	50.10	$-$ 19.36	67.45	2489

Table 7: Gold responses, pair-selection rule and data look-back (ablation configuration).

\Delta

LC: change relative to full DNO. “Mark

T

always positive” is the SPIN-style rule and “Accept only

T\succ S

” the DNO-Restrictive rule. The look-back variant starts from the SFT checkpoint; its matched comparison is given in Section 6.4 . Bold: best value per win-rate column.

	Avg. length $T$ / $S$ (chars)	Avg. rating $T$ / $S$	$T\succ S$	$S\succ T$	$S\succ S$	Total	Share of pairs with student positive (%)	Avg. rating pos. / neg.
Llama 3.1 8B, Iter. 1	1116.45 / 1521.30	4.86 / 3.28	25,093	2,373	21,543	49,009	48.80	5.74 / 1.68
Llama 3.1 8B, Iter. 2	1121.66 / 1744.86	4.42 / 4.43	12,852	11,342	14,058	38,252	66.40	5.62 / 2.70
Llama 3.1 8B, Iter. 3	1103.87 / 2003.26	4.26 / 4.52	6,057	7,602	7,747	21,406	71.70	5.62 / 2.67
Mistral 7B, Iter. 1	1147.87 / 0 806.08	5.03 / 3.48	42,576	2,273	19,129	63,978	33.45	5.69 / 2.60
Mistral 7B, Iter. 2	1252.91 / 2048.87	4.63 / 3.93	20,450	5,518	24,943	50,911	59.83	5.61 / 2.51
Mistral 7B, Iter. 3	1446.46 / 2951.22	4.66 / 4.13	6,268	2,581	18,412	27,261	77.01	5.58 / 3.07

Table 8: Teacher (

T

, GPT-4o gold response) versus student (

S

) across DNO iterations, with GPT-4o as annotator. Row “Iter.

t

” describes the responses sampled from

\pi_{t}

at the start of iteration

t

, on that iteration’s prompt partition (Section 4.1 ); pair counts are new pairs only, excluding look-back.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Fine-tuning	LoRA: rank 16, $\alpha$ 32, dropout 0.05; adapters merged after each DNO iteration
Learning rate (SFT)	$5\times 10^{-5}$
Learning rate (DPO, DNO)	$5\times 10^{-6}$ (most DNO runs)
Learning-rate search	$\{5\times 10^{-5},5\times 10^{-6}\}$ , first epoch / iteration
Optimizer; scheduler	AdamW; constant after 100 warm-up steps; weight decay 0
Effective batch size	$2$ per device $\times$ $4$ GPUs $\times$ $4$ accumulation steps $=32$

Appendix

Table 9: Training, generation and evaluation hyperparameters.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 620 papers

Jun 10, 2026

Boosting Direct Preference Optimization with Penalization

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Pengwei Sun

Stanford University.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Offline Preference Optimization

Ask PDF

May 21, 2026

Token-weighted Direct Preference Optimization with Attention

Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.

Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1

Cornell University · Vanderbilt University

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Preference Optimization Preference-Based RL

Ask PDF

Jun 11, 2026

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study

We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computational efficiency, and achieves competitive performance. The evaluation using BLEU, ROUGE, and cosine similarity metrics indicates effective learning and convergence, though further investigation is needed to address observed training instability.

Dezhi Yu, Yvonne Qiu, ShuoJia Fu

LLM Alignment Direct Preference Optimization LLM Fine-Tuning LLM Post-Training Fine-Tuning Language Models

Ask PDF

Abstract

LLM Alignment

Explore similar work

Boosting Direct Preference Optimization with Penalization

Token-weighted Direct Preference Optimization with Attention

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study