Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.
Figures & tables
Figure 1: One DNO-Prct iteration (Algorithm 2 ). Blue: training-time steps performed by GPT-4o in the main runs. Look-back pairs are reused without re-rating; LoRA adapters trained on πt are merged to give πt+1 .
Algorithm
Epoch or iter.
LC / raw win rate (%) vs. GPT-4o
LC / raw win rate (%) vs. GPT-4-0613
Avg. length (chars)
DNO
Iter. 1
39.01 / 43.04
61.44 / 68.45
3281
DNO
Iter. 2
41.57 / 44.66
64.87 / 71.37
3626
DNO
Iter. 3
45.32 / 48.26
70.18 / 74.60
3786
DNO-Restrictive
Iter. 3
37.98 / 38.62
60.16 / 60.51
4802
SPIN-style
Epoch 3
35.17 / 41.99
50.30 / 66.81
3207
SPPO
Epoch 3
31.28 / 34.22
45.85 / 59.44
2241
Table 1: Llama 3.1 8B Instruct: DNO versus our reproduced baselines, with GPT-4o as annotator and gold-response source. AlpacaEval 2.0 LC / raw win rate (%) against two opponents, judged by GPT-4o, and average length. The SPIN-style and DNO-Restrictive rows are ablation runs (Table 7 ; full-DNO reference 43.90% LC). Bold: best DNO value per column.
Algorithm
Epoch or iter.
LC / raw win rate (%) vs. GPT-4o
LC / raw win rate (%) vs. GPT-4-0613
Reported by Rosset et al. (%)
Avg. length (chars)
DNO
Iter. 1
9.94 / 10.64
33.62 / 35.12
∼ 10
2424
DNO
Iter. 2
13.62 / 13.45
38.67 / 40.17
∼ 21
5365
DNO
Iter. 3
18.44 / 18.53
38.20 / 37.85
24.97
7506
Offline DPO
Epoch 3
7.05 / 5.85
23.27 / 24.63
18.22
1465
SFT
Epoch 3
5.37 / 4.98
14.76 / 15.69
7.96
1804
SFT
Epoch 10
4.28 / 4.11
13.77 / 14.82
–
1808
Table 2: Mistral 7B Instruct v0.1: DNO versus our reproduced baselines. Columns as in Table 1 ; the Rosset et al. column lists raw win rates from their own Mistral-based run, for reference only ( ∼ : approximate; see Section 5.2 ). Bold: best DNO value per column.
Algorithm
Direct DNO, annotator GPT-4o
Direct DNO, annotator Llama 3.1 70B
SFT + DNO, annotator GPT-4o
vs. GPT-4o
vs. GPT-4-0613
Len.
vs. GPT-4o
vs. GPT-4-0613
Len.
vs. GPT-4o
vs. GPT-4-0613
Len.
DNO Iter. 1
39.01 / 43.04
61.44 / 68.45
3281
37.35 / 44.34
56.95 / 71.08
2763
36.85 / 38.20
50.52 / 65.53
2092
DNO Iter. 2
41.57 / 44.66
64.87 / 71.37
3626
40.01 / 47.45
58.55 / 72.55
3025
38.90 / 44.22
55.06 / 72.42
2285
DNO Iter. 3
45.32 / 48.26
70.18 / 74.60
3786
40.13 / 47.45
58.91 / 71.80
3185
44.92 / 45.14
68.96 / 69.30
4342
Llama 3.1 8B Instruct (base)
31.71 / 35.51
49.64 / 62.19
2237
–
–
–
–
–
–
Llama 3.1 70B Instruct (larger)
41.40 / 39.94
55.23 / 67.64
1873
–
–
–
–
–
–
Table 3: Llama 3.1 8B Instruct under three DNO settings: direct DNO annotated by GPT-4o (the run of Table 1 ), direct DNO annotated by Llama 3.1 70B Instruct, and SFT followed by DNO annotated by GPT-4o. Cells: LC / raw win rate (%); Len.: average length (chars). Bold: best DNO value per column within each block.
Algorithm
Direct DNO, annotator GPT-4o
Direct DNO, annotator Mistral Large
SFT + DNO, annotator GPT-4o
vs. GPT-4o
vs. GPT-4-0613
Len.
vs. GPT-4o
vs. GPT-4-0613
Len.
vs. GPT-4o
vs. GPT-4-0613
Len.
DNO Iter. 1
9.94 / 10.64
33.62 / 35.12
2424
16.40 / 17.36
38.72 / 40.21
8264
10.18 / 10.43
30.83 / 34.53
2078
DNO Iter. 2
13.62 / 13.45
38.67 / 40.17
5365
14.29 / 15.50
39.16 / 38.81
5365
11.43 / 10.68
29.29 / 32.92
1768
DNO Iter. 3
18.44 / 18.53
38.20 / 37.85
7506
16.30 / 17.81
38.71 / 39.09
7699
12.58 / 13.29
30.56 / 37.27
2030
Mistral 7B Instruct v0.1 (base)
6.77 / 5.47
18.40 / 18.66
1375
–
–
–
–
–
–
Mixtral 8x7B Instruct v0.1 (larger)
23.09 / 18.76
41.78 / 50.68
1464
–
–
–
–
–
–
Table 4: Mistral 7B Instruct v0.1 under the three DNO settings of Table 3 , with Mistral Large as the open-weight annotator.
Samples per prompt
LC / raw win rate (%) vs. GPT-4o
Avg. length (chars)
T≻S
S≻T
S≻S
Total
Avg. rating pos. / neg.
1
44.90 / 39.68
3251
4,025
2,708
0
6,733
5.87 / 3.39
3
45.96 / 41.37
2700
14,984
5,569
7,793
28,346
5.84 / 3.41
5
43.90 / 41.77
5324
20,978
14,167
33,379
68,524
5.92 / 3.44
7
44.15 / 38.75
3116
14,145
9,060
45,614
68,819
5.79 / 3.35
Table 5: Number of on-policy samples per prompt (ablation configuration). Selected pairs by type ( T : teacher’s gold response; S : student response); ratings on the 0–6 scale.
Margin
LC / raw win rate (%) vs. GPT-4o
Avg. length (chars)
T≻S
S≻T
S≻S
Total
Share of pairs with teacher positive (%)
Avg. rating pos. / neg.
1
36.60 / 41.23
2978
45,344
24,643
78,777
148,764
30.48
5.78 / 3.70
2
37.72 / 42.92
2931
16,601
13,089
25,712
55,402
29.96
5.81 / 3.39
3
38.99 / 43.54
2619
9,377
6,784
8,765
24,926
37.62
5.79 / 2.01
4
37.38 / 40.98
2551
5,169
4,580
3,826
13,575
38.08
5.82 / 1.42
5
38.13 / 41.93
2542
638
923
254
1,815
35.15
5.97 / 0.92
Table 6: Minimum rating margin (separate pipeline, at most 2,000 training steps per iteration). Columns as in Table 5 , plus the share of pairs whose positive is the teacher’s response.
Variant
vs. GPT-4o (%)
vs. GPT-4-0613 (%)
Avg. length (chars)
LC
Δ LC
raw
LC
Δ LC
raw
DNO (full)
43.90
–
41.77
69.46
–
64.21
5324
Remove ygold
36.97
− 6.93
40.91
55.36
− 14.10
66.15
2648
Mark T always positive
35.17
− 8.73
41.99
50.30
− 19.16
66.81
3207
Accept only T≻S
37.98
− 5.92
38.62
60.16
− 9.30
60.51
4802
Remove data look-back
35.00
− 8.90
41.61
50.10
− 19.36
67.45
2489
Table 7: Gold responses, pair-selection rule and data look-back (ablation configuration). Δ LC: change relative to full DNO. “Mark T always positive” is the SPIN-style rule and “Accept only T≻S ” the DNO-Restrictive rule. The look-back variant starts from the SFT checkpoint; its matched comparison is given in Section 6.4 . Bold: best value per win-rate column.
Avg. length T / S (chars)
Avg. rating T / S
T≻S
S≻T
S≻S
Total
Share of pairs with student positive (%)
Avg. rating pos. / neg.
Llama 3.1 8B, Iter. 1
1116.45 / 1521.30
4.86 / 3.28
25,093
2,373
21,543
49,009
48.80
5.74 / 1.68
Llama 3.1 8B, Iter. 2
1121.66 / 1744.86
4.42 / 4.43
12,852
11,342
14,058
38,252
66.40
5.62 / 2.70
Llama 3.1 8B, Iter. 3
1103.87 / 2003.26
4.26 / 4.52
6,057
7,602
7,747
21,406
71.70
5.62 / 2.67
Mistral 7B, Iter. 1
1147.87 / 0 806.08
5.03 / 3.48
42,576
2,273
19,129
63,978
33.45
5.69 / 2.60
Mistral 7B, Iter. 2
1252.91 / 2048.87
4.63 / 3.93
20,450
5,518
24,943
50,911
59.83
5.61 / 2.51
Mistral 7B, Iter. 3
1446.46 / 2951.22
4.66 / 4.13
6,268
2,581
18,412
27,261
77.01
5.58 / 3.07
Table 8: Teacher ( T , GPT-4o gold response) versus student ( S ) across DNO iterations, with GPT-4o as annotator. Row “Iter. t ” describes the responses sampled from πt at the start of iteration t , on that iteration’s prompt partition (Section 4.1 ); pair counts are new pairs only, excluding look-back.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Fine-tuning
LoRA: rank 16, α 32, dropout 0.05; adapters merged after each DNO iteration
Learning rate (SFT)
5×10−5
Learning rate (DPO, DNO)
5×10−6 (most DNO runs)
Learning-rate search
{5×10−5,5×10−6} , first epoch / iteration
Optimizer; scheduler
AdamW; constant after 100 warm-up steps; weight decay 0
Effective batch size
2 per device ×4 GPUs ×4 accumulation steps =32
Appendix
Table 9: Training, generation and evaluation hyperparameters.
Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.
Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computational efficiency, and achieves competitive performance. The evaluation using BLEU, ROUGE, and cosine similarity metrics indicates effective learning and convergence, though further investigation is needed to address observed training instability.