Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually destroy generation. We trace this failure to the common-mode offset, a shared directional shift in the model's hidden states that appears across token positions, prompts, and both chosen and rejected responses. Because it moves chosen and rejected representations together, the relative ranking loss barely reacts to its accumulation. As this offset grows, it projects onto the output layer to form a vocabulary well, elevating a single token's probability until it dominates decoding and causes generation collapse. The isolated offset alone reconstructs the vocabulary well, injecting the learned offset into an untreated base model recreates the collapse where norm-matched random directions do not, and masking this common component from the loss prevents the well in our flagship setting. Under the per-token ranking loss we study, the same collapse appears in 14 of 17 base-model lineages and under full-parameter fine-tuning. Moreover, the offset is not merely an artifact of collapse, since it is already present in textbook DPO and SimPO runs that form no well. Textbook DPO keeps it weakly aligned with any single vocabulary direction even with three times our base budget. Trained from base models for that budget, however, full-parameter SimPO drives it into collapse on both Llama-2 and Llama-3.1, and the same kind of offset grows, without a well, from a public SFT checkpoint under the official SimPO recipe. Across 17 UltraFeedback runs, the better a run ranks held-out responses, the less often two public reward models prefer its samples to the base model's. These findings expose a blind spot in standard ranking metrics and introduce a representation-level signal that can warn of generation collapse before it occurs.
Figures & tables
Figure 1: (a) Two Llama-2 runs on the same driving lists rank held-out responses equally well, but only the one without offset control collapses onto one token. (b) The common-mode offset {\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} moves chosen and rejected hidden states together, so the margin the loss sees barely changes. (c) The offset lifts a rare token above the base model’s top token once its gain gt exceeds the base gap mt .
Base
Budget
Δ chosen (nat)
Capture / tail
Degen. / ring
\|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\|
Well
Llama-2-7B
1×
−10.97
89.4% / 89.5%
96.0% / –
92.4
textt
Mistral-7B-v0.3
1×
−17.51
95.1% / 95.1%
100% / 0%
300.4
/******/
Qwen2.5-7B
3×
−21.67
93.9% / 86.7%
99.3% / –
328.7
CJK fragment
Qwen2.5-14B
3×
−3.85
27.0% / 7.7%
80.3% / 78.7%
198.6
collision -fam.
Gemma-2-9B
1×
+0.62
12.5% / 37.5%
0% / 11 of 12 †
163.7
crash -fam.
Llama-3.1-8B
1×
−14.27
87.6% / 88.5%
18.3% / 22.0%
115.2
.scalablytyped
Table 1: Every base here but Llama-3.2-3B collapses, onto one token or the forms of one word. Untreated runs at their onset budget, or at 6× for Llama-3.2-3B. Ring counts samples dominated by one word stem, on the 12 saved samples where marked † and unmeasured where marked –.
Figure 2: (a) At every position of a rejected response, the shift of the final hidden state splits into the shared offset {\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} and a position-specific residual {\color[rgb]{0.7852,0.3906,0.1758}r_{t}} . (b) Through the unembedding W , only {\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} singles out one vocabulary row. (c) Adding only {\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} to the base hidden state puts the well on top at nearly every rejected position, while adding only {\color[rgb]{0.7852,0.3906,0.1758}r_{t}} rarely does; UF denotes UltraFeedback.
Figure 3: (a) The offset overrides the base model at a position once its gain gt on the would-be well exceeds the base gap mt , that is, once the dose Dt exceeds one. (b) Across 510 snapshots of ten collapsing lineages, colored by training progress, and three summed-score runs that stay healthy, in gray, the well never tops a majority of rejected positions while the median dose stays below one. (c) Median doses of Table 2 at the usual and the three-fold budget.
Objective
Budget
Demand (nat)
\|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\|
Dose
Capture
Degen.
Win
Per-token ranking, β=0.12
1×
42
84
1.92
97%
99%
25%
with a reference model
1×
42
79
1.91
86%
99%
26%
with a reference model
3×
42
91
4.53 †
89%
99%
17%
SimPO
1×
2.4
114
0.29
0.4%
0.7%
7%
SimPO
3×
2.4
82
0.53
9%
28%
6%
SimPO, full-parameter
1×
2.4
65
0.33
3%
0.7%
10%
Table 2: Each loss at its usual setting and, where listed, with three times the budget, with doses above one in bold. Runs start from the base model unless marked SFT, and all train on the same UltraFeedback pairs. SimPO uses β=2.5 and γ=1.0 from base models and the official β=2.0 and γ=1.0 from the SFT checkpoint. Degen. counts repetitive or nearly empty samples, and Win compares a run’s samples with those of the model it starts from.
Base
Penalty
Steps
Capture / tail
Degen.
Held-out top-1 / pairwise
Llama-2-7B
loudness
1512
19.8% / 1.5%
6–12%
1.00 / 0.95
Mistral-7B
both
504
14.6% / 1.4%
3.0%
1.00 / 0.94
Qwen2.5-7B
both
1512
0 8.0% / 0.2%
1.7%
1.00 / 0.96
Llama-3.1-8B
both
1512
13.0% / 5.5%
6.7%
1.00 / 0.96
Llama-2-7B without offset control
NLL only
–
1512
70.6% / 71.0%
2.0%
1.00 / 0.97
Table 3: The recipe, centering plus a norm penalty, a loudness penalty, or both, at or beyond each base’s onset budget, where capture goes to an ordinary token. Three Llama-2 baselines without offset control keep 71 to 81% of the released mass on one rare token.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Base
Vocabulary
Onset multiple
Capture
Degen. / ring
Collapse
Llama-2-7B ( Touvron et al., 2023b )
32k
1×
89.4%
96.0% / –
well
LLaMA-1-7B ( Touvron et al., 2023a )
32k
1×
53.0%
86.3% / –
well
TinyLlama-1.1B ( Zhang et al., 2024 )
32k
6×
99.0%
100% / 100%
well
Llama-3.2-1B-Instruct ( Grattafiori et al., 2024 )
128k
1×
88.8%
61.0% / –
well
Llama-3.2-3B
128k
none within 6×
43.7%
10.7% / 7.7%
no
Llama-3.1-8B
128k
1×
87.6%
18.3% / 22.0%
well
Appendix
Table 4: Fourteen of the 17 base-model lineages collapse within six times the base budget (§ 4 ). A run collapses once one token takes most of the released mass, a well, or once most of its samples degenerate or ring on one word stem, a word ring. Each row shows the untreated run at the onset budget, or at the largest budget tested when no collapse occurs. The onset multiple is relative to the 504-step base budget under the fixed schedule. TinyLlama, the three smallest Qwen2.5 models, and Llama-3.2-3B were trained at one and six times the budget only, Qwen2.5-14B first at three times, Phi-3-mini and Gemma-2-2B at one and three times, and Gemma-2-9B at the base budget only. Ring rates marked – come from audits that predate the ring measure, and † marks a count on the 12 saved samples. Each family is cited at its first row.
Figure 4: Training curves of six Llama-2-7B runs with K=4 sequential PL on the driving lists at three times the base budget, smoothed with a 25-step running median. (a) Total loss and (b) ranking loss fall lowest in the three runs without offset control, NLL only, N-DPO, and DPO-Positive (DPOP), which end with 71 to 81% of the released mass in a single well. The ranking loss of the SFT control ( β=0 ) stays at its initial value. (c) Without offset control the training margin reaches 67 to 87 nat, against 30 and 37 nat with it, and (d) almost all of this difference comes from pushing the rejected responses down.
Figure 5: Six-fold-budget training trajectories of OLMo-2-7B (a–c) and OLMo-3-7B (d–f) in six conditions, namely untreated, NLL only, and four offset-control variants that penalize loudness, loudness and norm, per-position loudness, or per-position loudness and norm. OLMo-2 has no NLL-only run. (a) and (d) Well capture crosses the dotted 50% line in the untreated and NLL-only runs and ends at 81 to 94%, while no offset-control variant exceeds 45%. (b) and (e) The tail beyond rank 10,000 rises with the capture in those runs. With offset control it ends at 28 to 37% on OLMo-2, but near 5% on OLMo-3 once the norm penalty is added. (c) and (f) Pairwise accuracy on 300 held-out lists stays between 0.93 and 0.96 in every run.
Figure 6: (a) Across seven Llama-2-7B runs, \|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\| stays between 70 and 85 without a penalty, grows to 116 under the loudness penalty, and falls to 6 to 8 once the norm penalty is added. (b) Divided by \|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\| , the largest entry of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} stays near zero only under the loudness penalty. (c) NLL only and N-DPO lift the largest entry of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} to 76 and 95 by the end of training, while the loudness penalty holds it below 4. (d) Across seven Qwen2.5-7B runs, \|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\| grows to 490 to 550 under every loudness penalty but stays near 40 once the norm penalty is added. (e) Divided by \|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\| , the largest entry of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} falls below zero under every loudness penalty. (f) Without a penalty, the largest entry of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} climbs to 66, while every penalized run stays below 5.
Figure 7: (a) On 600 held-out lists, pairwise K=2 training orders the unseen rejected responses below chance (dashed line), with or without offset control (blue), while every sequential listwise K=4 run orders them better than the base models. (b) The Kendall τ over the rejected responses alone is negative for SFT and every pairwise run, and at least 0.64 for every listwise K=4 run. Triangles mark runs that form a well.
Subtracted
Degen.
Chosen log-prob (nat)
Rejected log-prob (nat)
Nothing
100%
−13.37
−65.01
Only the well’s logit
22%
−8.73
−54.64
The eight largest entries of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}
65%
−7.17
−48.76
Half of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}
0%
−4.70
−39.44
All of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}
7%
−5.34
−24.98
Appendix
Table 5: Subtracting the offset’s logit bias from the collapsed flagship model at decoding time. Degen. is the share of degenerate greedy samples, and the log-probabilities are per-token means under teacher forcing with the same subtraction.
Figure 8: A z -loss penalty on logZ compresses absolute logit scale, relative peak, and common-mode vocabulary projection without preventing well capture. Blue lines show three strengths of λz on the Llama-2 flagship setting, darker for stronger, and the gray line shows the untreated run. (a) The strongest penalty, λz=10−4 , lowers logZ from 76.0 to 30.5 by step 1500. (b) It lowers the relative peak logit by the same factor of 2.5, from 70.0 to 28.2. (c) It lowers the largest entry of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} from 74.6 to 31.1. (d) The top token still takes 65 to 66% of the released mass at every strength, against 74% untreated, and at λz=10−4 the mass beyond rank 10,000 (dotted) rises to 80%.
Figure 9: Dose along natural training runs, with a dot at the first snapshot at which the well is the top token at a majority of rejected positions. (a) In all nine collapsing lineages, the median dose Dt=gt/mt , measured every 50 steps, passes one before the dot and lies between 1.1 and 4.1 at the dot, while the three summed-score runs that stay healthy, in gray, remain below 0.2. (b) The share of rejected positions with the well on top climbs from under 10% to a majority within 100 steps in seven lineages, and within 200 and 300 steps in Qwen2.5-3B and TinyLlama.
Figure 10: Outcome of pairwise training on UltraFeedback against the per-token margin the objective demands, 5/βeff with βeff=β for per-token-average scores and β∣y∣ for summed scores. The dashed vertical line marks the median base gap at the rejected positions, 17.5 nat, and dotted lines mark the base model at 1.48 nat and 27%. (a) Under per-token averaging the well takes more than 95% of the released mass once the demand passes the base gap, whereas summed scores concentrate mass gradually and send almost none beyond rank 10,000. (b) Samples from the two lowest-demand per-token runs have a high negative log-likelihood under the base model, samples from per-token runs beyond the base gap a low one, and summed runs stay near the base model. (c) Every per-token run repeats under greedy decoding on at least 78% of prompts, against at most 37% for summed runs. (d) The final dose passes one only under per-token averaging with a demand above the base gap. Red marks runs with more than half of the released mass on one token and at least 30% degenerate samples, blue marks clean runs, and orange the runs in between. Doses are for each run’s leading token, or for Conf when that token takes less than 5% of the released mass.
Figure 11: Standard recipes build the same offset at a smaller dose. All runs train Llama-2-7B pairwise on UltraFeedback and are plotted against training step on log axes. Per-token averaging at β≤0.2 collapses, while textbook DPO at β=0.1 and SimPO at β∈{2.0,2.5} do not within the base budget. (a) By the end of the base budget, SimPO has built an offset about as long as that of the collapsing runs, and textbook DPO one at most a sixth as long. (b) Only per-token averaging at β≤0.2 gives Conf , the well of the collapsing run, a logit gain above 20 nat, while textbook DPO and SimPO stay below 3 nat within the base budget. (c) By the end of the base budget, the doses of textbook DPO and SimPO sit four to twenty-five times below those of the collapsing runs, and full-parameter SimPO crosses D=1 at step 3000 under the three-fold budget. Squares mark full-parameter fine-tuning, and dashed lines mark three-fold-budget runs, which train for nine epochs under their own schedule. Full-parameter per-token averaging, which collapses onto the space token, and full-parameter textbook DPO appear only in (a).
Budget
Steps
Top token
Capture / tail
Degen.
NLL
\|{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}}\|
Dose
Held-out top-1 / pairwise
1×
369
,
2.5% / 0.6%
1.3%
1.35
32.5
0.15
0.41 / 0.67
3×
1107
(
2.7% / 4.1%
0.0%
3.86
77.3
0.19
0.58 / 0.75
Appendix
Table 6: SimPO from the public SFT checkpoint with the official hyperparameters. NLL is the SFT model’s per-token negative log-likelihood of sampled responses, the dose is for Conf , and held-out pairwise accuracy is 0.619 for the SFT checkpoint itself.
Run
Skywork win
ArmoRM win
Pairwise
Top-10
Llama-2-7B
Per-token ranking, β=0.12
25%
6%
0.68
0.02
with a reference model, 1× / 3×
26% / 17%
7% / 3%
0.68 / 0.71
0.04 / 0.05
full-parameter
24%
45%
0.70
0.14
SimPO, 1× / 3×
7% / 6%
3% / 4%
0.67 / 0.73
0.04 / 0.04
SimPO, β=2.0 , γ=1.6
8%
6%
0.67
0.05
Appendix
Table 7: External reward models on 300 held-out UltraFeedback prompts. Win is the share of prompts on which the reward model prefers the run’s sample to that of the model the run starts from. Pairwise is held-out pairwise accuracy. Top-10 is the share of released mass that goes to the base model’s ten most likely tokens under teacher forcing. The base Llama-2-7B has a held-out pairwise accuracy of 0.60, the base Llama-3.1-8B 0.61, and the SFT checkpoint 0.62. The last four Llama-2 rows train on four-response lists and the others on pairs.
Figure 12: (a) Across 17 UltraFeedback runs, the better a run ranks held-out responses, the less often a public reward model prefers its samples to the base model’s; the glyphs show where each group’s released mass goes. (b) In seven of nine collapsing lineages, the eventual well enters the eight largest entries of W{\color[rgb]{0.2031,0.5313,0.2813}\bar{b}} before it takes over the rejected positions.
Figure 13: (a) Held-out ranking accuracy and (b) the top token’s share of released mass against the preference margin along training. Ranking saturates early for every run, while the top token’s share keeps growing with the margin only in the runs without offset control.
Base
Data, objective
∥δ∥
a+
a−
c
∣d∣
random ∣c∣ / ∣d∣
cos
Llama-2-7B
UltraFeedback, per-token β=0.12 (collapses)
84
−14.5
−15.2
−14.9
0.68
0.30 / 0.06
0.03
Llama-2-7B
driving, K=4 (collapses)
93
−65.1
−65.2
−65.2
0.12
0.49 / 0.02
0.16
Qwen2.5-7B
UltraFeedback, per-token β=0.12 (collapses)
282
−26.0
−27.1
−26.6
1.36
2.15 / 0.31
−0.02
Llama-3.1-8B
UltraFeedback, per-token β=0.12 (collapses)
109
−29.2
−30.1
−29.6
0.94
0.54 / 0.10
0.05
Llama-2-7B
UltraFeedback, textbook DPO β=0.1
14
−0.005
−0.013
−0.009
0.014
0.011 / 0.008
0.07
Llama-2-7B
UltraFeedback, SimPO β=2.5 , γ=1.0
114
−0.23
−0.26
−0.25
0.06
0.57 / 0.09
0.01
Appendix
Table 8: The trained offset δ added to the base model’s final hidden states on 200 held-out lists. a+ and a− are the mean changes of the chosen and mean rejected per-token scores in nat, c the mean of (a++a−)/2 and ∣d∣ the mean of ∣a+−a−∣/2 over lists, “random” the same two means for three norm-matched random directions, and cos the cosine between δ and uˉ , the base model’s chosen-minus-rejected score gradient.
Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples.