Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive pathology in this formulation: a disproportionately small subset of high-frequency token types dominates cumulative sequence scores while appearing symmetrically across both preferred and dispreferred responses. Specifically, under canonical Qwen tokenization on Anthropic HH-RLHF, merely 69 token types account for 55.1% of all response tokens and 85.9% of within-pair shared token mass, exhibiting substantially lower preference-side specificity than the remaining vocabulary. This symmetric ubiquity induces gradient entanglement and dilutes the discriminative preference signal propagated through the objective. To resolve this issue, we introduce Anisotropic DPO (ADPO) and its canonical realization, Frequency-Hard DPO. Using a fixed, label-agnostic vocabulary mask, our method zeroes the implicit reward contribution of high-frequency response tokens while assigning unit weight to informative positions, thereby suppressing gradient interference without modifying preference pairs, discarding context, or introducing learned parameters. Here, anisotropy designates non-uniform token-level objective weighting rather than representational geometry. Extensive empirical evaluations on AlpacaEval, MT-Bench, and Arena-Hard demonstrate that Frequency-Hard DPO consistently outperforms standard DPO across Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct, establishing that selectively masking shared high-frequency tokens offers an effective, zero-overhead mechanism for robust preference alignment.
Figures & tables
Figure 1: ADPO masks score terms, not context. Coral marks frequent types; teal marks retained types. Letters identify token types, not score values. Frequencies and coefficients are illustrative, not a shared positional mask. Each response keeps its full context and softmax; only coral score terms are zeroed. With beta-scaled retained and frequent margins k and h , selection changes the sigmoid gate from σ(−k−h) to σ(−k) . Zero direct score weight does not eliminate indirect gradients through shared parameters.
Figure 2: Frequency-mask coverage and held-out side signal. (a) We measure the frequent-type mask’s coverage of observed vocabulary types, response occurrences, and shared chosen–rejected occurrences in HH calibration data. (b) We compare retained and frequent types using held-out preference-side associations estimated on separate training pairs. Error bars show paired confidence intervals. The panels use distinct populations.
Figure 3: Gradient alignment and preference signal. (a) We compute mean chosen–rejected cosine for full and frequency-hard response-score gradients at saved DPO checkpoints, using identical weights, response pairs, and input context.(b) We score complete HH responses, frequent-only fragments, and frequent- or count-matched random-deletion variants with frozen PairRM, then compare rankings with the original human labels. Error bars show prompt-bootstrap intervals, and the vertical line marks chance. The panels use different samples and interventions: score masking versus response editing.
Condition
Agreement (%)
Margin
Full
67.8
1.89969
Frequent-only
51.7
0.08779
Without frequent
61.4
0.97335
Without random
56.67
0.60148
Table 1: Human-label agreement and reward margins. We evaluate the response conditions in Figure 3 (b) against the original HH preference labels. Ties receive half credit; margins are order-symmetrized raw PairRM logits, not reward probabilities.
Token type
Tokens masked
Function words / stopwords
the , is , The , and , for , to , that , a , with , it , of , or , are , in , can , be , on , by , as
Table 2: Token types removed by the frequency-hard mask at τ=0.002 . The mask contains 50 vocabulary types whose empirical frequency exceeds the threshold.
Method
FsFairX RM
MT-Bench
AlpacaEval 2.0
Arena-Hard
Mean Reward
Score
LC Win Rate
LC Win Rate
DPO
1.231 ± 0.045
8.43 ± 0.060
49.72% ± 0.65%
52.1% ± 0.65%
SimPO
1.180 ± 0.055
8.15 ± 0.072
47.80% ± 0.72%
51.6% ± 0.69%
TGDPO
1.275 ± 0.045
8.42 ± 0.060
51.40% ± 0.65%
52.8% ± 0.64%
TDPO
1.220 ± 0.051
8.475 ± 0.055
49.90% ± 0.71%
53.0% ± 0.55%
SparsePO
1.255 ± 0.053
8.46 ± 0.061
51.10% ± 0.63%
52.6% ± 0.51%
Table 3: Main benchmark results for Qwen2.5-7B-Instruct. MT-Bench and AlpacaEval 2.0 use GPT-4-turbo as the API judge, while Arena-Hard uses GPT-4.1-mini as the API judge. ± values denote respective uncertainity estimates.
Method
FsFairX RM
MT-Bench
AlpacaEval 2.0
Arena-Hard
Mean Reward
Score
LC Win Rate
LC Win Rate
DPO
2.15 ± 0.058
7.66 ± 0.067
50.43% ± 0.72%
34.9% ± 0.68%
SimPO
1.95 ± 0.063
7.60 ± 0.074
48.50% ± 0.77%
32.8% ± 0.73%
TGDPO
2.17 ± 0.052
7.75 ± 0.069
52.00% ± 0.64%
35.2% ± 0.62%
TDPO
2.14 ± 0.057
7.85 ± 0.061
50.00% ± 0.71%
35.0% ± 0.66%
SparsePO
2.17 ± 0.054
7.80 ± 0.068
51.40% ± 0.67%
35.4% ± 0.59%
Table 4: Main benchmark results for Llama-3-8B-Instruct. MT-Bench and AlpacaEval 2.0 use GPT-4-turbo as the API judge, while Arena-Hard uses GPT-4.1-mini as the API judge.
τ
Masked Tokens
FsFairX RM
AlpacaEval 2.0
(Approx.)
Mean Reward
LC Win Rate
0.00413
25
1.654
48.15%
0.00200
50
2.200
53.06%
0.00084
100
1.440
42.70%
Table 5: Sensitivity of ADPO to the frequency threshold τ on Llama-3-8B-Instruct. Each threshold is selected to mask approximately the indicated number of vocabulary tokens. FsFairX RM reports the mean reward assigned by the FsFairX Llama-3 RM v0.1, while AlpacaEval 2.0 reports length-controlled (LC) win rate.
Method
FsFairX RM
AlpacaEval 2.0
Mean Reward
LC Win Rate
DPO
2.15
50.43%
Random mask (50 tokens)
2.14
50.18%
Mid-frequency mask (50 tokens)
2.11
49.72%
Low-frequency mask (50 tokens)
2.14
49.55%
Soft inverse-frequency weights
2.16
50.62%
Table 6: Ablation of token-selection strategies on Llama-3-8B-Instruct using approximately 50 token types. Hard-mask variants remove the indicated token set from the preference score while preserving the full context. Soft inverse-frequency weighting assigns larger weights to less frequent token types.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Tokenizer
∣Hτ∣
F(Hτ) (%)
J(Hτ) (%)
q(Hτ)
q(Kτ)
Mistral
53
40.94
52.49
0.0126
0.1033
Llama 3
50
36.06
47.43
0.0133
0.1406
Qwen2.5
52
37.77
49.22
0.0135
0.1404
Appendix
Table 7: Held-out coverage and side specificity of training-derived frequency masks. Coverage is computed from response occurrences; shared coverage uses within-pair multiset overlap. The same test pairs are tokenized separately for each row.
Tokenizer
C(Hτ)
C(Kτ)
Difference
95% CI
Mistral
0.326
1.234
0.908
[0.382,1.439]
Llama 3
0.315
1.602
1.287
[0.563,2.022]
Qwen2.5
0.292
1.558
1.266
[0.546,1.981]
Appendix
Table 8: Cross-fitted preference-side signal. All entries are scaled by 103 . Intervals are paired percentile bootstrap intervals for the retained-minus-frequent difference, computed separately per tokenizer.
Mistral
Llama 3
Qwen2.5
ID
Vocabulary piece
ID
Vocabulary piece
ID
Vocabulary piece
13
"<0x0A>"
11
","
11
","
28723
"."
279
"\u0120the"
279
"\u0120the"
28725
","
323
"\u0120and"
323
"\u0120and"
272
"\u2581the"
13
"."
13
"."
304
"\u2581and"
311
"\u0120to"
311
"\u0120to"
Appendix
Table 9: Complete frequent-type inventories for the structural audit. Token IDs and escaped vocabulary pieces preserve whitespace, byte, and special-token distinctions.
Statistic
Frequency
Random mean
Random 95% range
p
Selected types
52
20,911.4
[16,756.4,24,749.7]
—
Occurrence coverage (%)
37.60
37.60
[37.60,37.60]
—
Shared coverage (%)
49.03
37.46
[34.62,40.71]
0.001
Shared enrichment
1.304
0.996
[0.921,1.083]
0.001
Side specificity
0.0152
0.0748
[0.0625,0.0864]
0.001
Appendix
Table 10: Frequency selection and occurrence-matched randomized selection on the training-calibration population. The range is the central empirical range across randomized selections, not a confidence interval for a trained-model effect. One-sided Monte Carlo values are unadjusted.
Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter β decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.
Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA