Masking Frequent Tokens Sharpens Direct Preference Optimization

Harshvardhan Saini; Samyak Jha; Yiming Tang; Dianbo Liu

cs.CLSep 26, 2026

Masking Frequent Tokens Sharpens Direct Preference Optimization

Authors: Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

Organizations: Indian Institute of Technology Dhanbad · National University of Singapore

Abstract

Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive pathology in this formulation: a disproportionately small subset of high-frequency token types dominates cumulative sequence scores while appearing symmetrically across both preferred and dispreferred responses. Specifically, under canonical Qwen tokenization on Anthropic HH-RLHF, merely 69 token types account for $55.1\%$ of all response tokens and $85.9\%$ of within-pair shared token mass, exhibiting substantially lower preference-side specificity than the remaining vocabulary. This symmetric ubiquity induces gradient entanglement and dilutes the discriminative preference signal propagated through the objective. To resolve this issue, we introduce Anisotropic DPO (ADPO) and its canonical realization, Frequency-Hard DPO. Using a fixed, label-agnostic vocabulary mask, our method zeroes the implicit reward contribution of high-frequency response tokens while assigning unit weight to informative positions, thereby suppressing gradient interference without modifying preference pairs, discarding context, or introducing learned parameters. Here, anisotropy designates non-uniform token-level objective weighting rather than representational geometry. Extensive empirical evaluations on AlpacaEval, MT-Bench, and Arena-Hard demonstrate that Frequency-Hard DPO consistently outperforms standard DPO across Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct, establishing that selectively masking shared high-frequency tokens offers an effective, zero-overhead mechanism for robust preference alignment.

Figures & tables

Figure 1: ADPO masks score terms, not context. Coral marks frequent types; teal marks retained types. Letters identify token types, not score values. Frequencies and coefficients are illustrative, not a shared positional mask. Each response keeps its full context and softmax; only coral score terms are zeroed. With beta-scaled retained and frequent margins

k

and

h

, selection changes the sigmoid gate from

\sigma(-k-h)

\sigma(-k)

. Zero direct score weight does not eliminate indirect gradients through shared parameters.

Figure 2: Frequency-mask coverage and held-out side signal. (a) We measure the frequent-type mask’s coverage of observed vocabulary types, response occurrences, and shared chosen–rejected occurrences in HH calibration data. (b) We compare retained and frequent types using held-out preference-side associations estimated on separate training pairs. Error bars show paired confidence intervals. The panels use distinct populations.

Figure 3: Gradient alignment and preference signal. (a) We compute mean chosen–rejected cosine for full and frequency-hard response-score gradients at saved DPO checkpoints, using identical weights, response pairs, and input context.(b) We score complete HH responses, frequent-only fragments, and frequent- or count-matched random-deletion variants with frozen PairRM, then compare rankings with the original human labels. Error bars show prompt-bootstrap intervals, and the vertical line marks chance. The panels use different samples and interventions: score masking versus response editing.

Condition	Agreement (%)	Margin
Full	67.8	1.89969
Frequent-only	51.7	0.08779
Without frequent	61.4	0.97335
Without random	56.67	0.60148

Table 1: Human-label agreement and reward margins. We evaluate the response conditions in Figure 3 (b) against the original HH preference labels. Ties receive half credit; margins are order-symmetrized raw PairRM logits, not reward probabilities.

Token type	Tokens masked
Function words / stopwords	the , is , The , and , for , to , that , a , with , it , of , or , are , in , can , be , on , by , as
Punctuation	, , . , : , ) , "
Markdown / code markup	* , , : , ** , ` , ``
Whitespace / Newline composites	space , \n , three-space indent , \t , .\n , .\n\n , :\n\n , **\n\n

Table 2: Token types removed by the frequency-hard mask at

\tau=0.002

. The mask contains 50 vocabulary types whose empirical frequency exceeds the threshold.

Method	FsFairX RM	MT-Bench	AlpacaEval 2.0	Arena-Hard
	Mean Reward	Score	LC Win Rate	LC Win Rate
DPO	1.231 $\pm$ 0.045	8.43 $\pm$ 0.060	49.72% $\pm$ 0.65%	52.1% $\pm$ 0.65%
SimPO	1.180 $\pm$ 0.055	8.15 $\pm$ 0.072	47.80% $\pm$ 0.72%	51.6% $\pm$ 0.69%
TGDPO	1.275 $\pm$ 0.045	8.42 $\pm$ 0.060	51.40% $\pm$ 0.65%	52.8% $\pm$ 0.64%
TDPO	1.220 $\pm$ 0.051	8.475 $\pm$ 0.055	49.90% $\pm$ 0.71%	53.0% $\pm$ 0.55%
SparsePO	1.255 $\pm$ 0.053	8.46 $\pm$ 0.061	51.10% $\pm$ 0.63%	52.6% $\pm$ 0.51%

Table 3: Main benchmark results for Qwen2.5-7B-Instruct. MT-Bench and AlpacaEval 2.0 use GPT-4-turbo as the API judge, while Arena-Hard uses GPT-4.1-mini as the API judge.

\pm

values denote respective uncertainity estimates.

Method	FsFairX RM	MT-Bench	AlpacaEval 2.0	Arena-Hard
	Mean Reward	Score	LC Win Rate	LC Win Rate
DPO	2.15 $\pm$ 0.058	7.66 $\pm$ 0.067	50.43% $\pm$ 0.72%	34.9% $\pm$ 0.68%
SimPO	1.95 $\pm$ 0.063	7.60 $\pm$ 0.074	48.50% $\pm$ 0.77%	32.8% $\pm$ 0.73%
TGDPO	2.17 $\pm$ 0.052	7.75 $\pm$ 0.069	52.00% $\pm$ 0.64%	35.2% $\pm$ 0.62%
TDPO	2.14 $\pm$ 0.057	7.85 $\pm$ 0.061	50.00% $\pm$ 0.71%	35.0% $\pm$ 0.66%
SparsePO	2.17 $\pm$ 0.054	7.80 $\pm$ 0.068	51.40% $\pm$ 0.67%	35.4% $\pm$ 0.59%

Table 4: Main benchmark results for Llama-3-8B-Instruct. MT-Bench and AlpacaEval 2.0 use GPT-4-turbo as the API judge, while Arena-Hard uses GPT-4.1-mini as the API judge.

$\tau$	Masked Tokens	FsFairX RM	AlpacaEval 2.0
	(Approx.)	Mean Reward	LC Win Rate
0.00413	25	1.654	48.15%
0.00200	50	2.200	53.06%
0.00084	100	1.440	42.70%

Table 5: Sensitivity of ADPO to the frequency threshold

\tau

on Llama-3-8B-Instruct. Each threshold is selected to mask approximately the indicated number of vocabulary tokens. FsFairX RM reports the mean reward assigned by the FsFairX Llama-3 RM v0.1, while AlpacaEval 2.0 reports length-controlled (LC) win rate.

Method	FsFairX RM	AlpacaEval 2.0
	Mean Reward	LC Win Rate
DPO	2.15	50.43%
Random mask (50 tokens)	2.14	50.18%
Mid-frequency mask (50 tokens)	2.11	49.72%
Low-frequency mask (50 tokens)	2.14	49.55%
Soft inverse-frequency weights	2.16	50.62%

Table 6: Ablation of token-selection strategies on Llama-3-8B-Instruct using approximately 50 token types. Hard-mask variants remove the indicated token set from the preference score while preserving the full context. Soft inverse-frequency weighting assigns larger weights to less frequent token types.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Tokenizer	$\|\mathcal{H}_{\tau}\|$	$F(\mathcal{H}_{\tau})$ (%)	$J(\mathcal{H}_{\tau})$ (%)	$\overline{q}(\mathcal{H}_{\tau})$	$\overline{q}(\mathcal{K}_{\tau})$
Mistral	53	40.94	52.49	0.0126	0.1033
Llama 3	50	36.06	47.43	0.0133	0.1406
Qwen2.5	52	37.77	49.22	0.0135	0.1404

Appendix

Table 7: Held-out coverage and side specificity of training-derived frequency masks. Coverage is computed from response occurrences; shared coverage uses within-pair multiset overlap. The same test pairs are tokenized separately for each row.

Tokenizer	$C(\mathcal{H}_{\tau})$	$C(\mathcal{K}_{\tau})$	Difference	95% CI
Mistral	0.326	1.234	0.908	$[0.382,1.439]$
Llama 3	0.315	1.602	1.287	$[0.563,2.022]$
Qwen2.5	0.292	1.558	1.266	$[0.546,1.981]$

Appendix

Table 8: Cross-fitted preference-side signal. All entries are scaled by

10^{3}

. Intervals are paired percentile bootstrap intervals for the retained-minus-frequent difference, computed separately per tokenizer.

Mistral		Llama 3		Qwen2.5
ID	Vocabulary piece	ID	Vocabulary piece	ID	Vocabulary piece
13	"<0x0A>"	11	","	11	","
28723	"."	279	"\u0120the"	279	"\u0120the"
28725	","	323	"\u0120and"	323	"\u0120and"
272	"\u2581the"	13	"."	13	"."
304	"\u2581and"	311	"\u0120to"	311	"\u0120to"

Appendix

Table 9: Complete frequent-type inventories for the structural audit. Token IDs and escaped vocabulary pieces preserve whitespace, byte, and special-token distinctions.

Statistic	Frequency	Random mean	Random 95% range	$p$
Selected types	52	20,911.4	$[16{,}756.4,24{,}749.7]$	—
Occurrence coverage (%)	37.60	37.60	$[37.60,37.60]$	—
Shared coverage (%)	49.03	37.46	$[34.62,40.71]$	0.001
Shared enrichment	1.304	0.996	$[0.921,1.083]$	0.001
Side specificity	0.0152	0.0748	$[0.0625,0.0864]$	0.001

Appendix

Table 10: Frequency selection and occurrence-matched randomized selection on the training-calibration population. The range is the central empirical range across randomized selections, not a confidence interval for a trained-model effect. One-sided Monte Carlo values are unadjusted.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 607 papers

Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Pairwise Preference Learning LLM Alignment
Cornell University · Vanderbilt University
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference Learning LLM Alignment
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA

May 21, 2026J/K move · Enter open · S save

Token-weighted Direct Preference Optimization with Attention

Chengyu Huang, Zhuohang Li, Sheng-Yen Chou, Claire Cardie

Cornell University · Vanderbilt University

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.

Abstract

LLM Alignment

Explore similar work

Token-weighted Direct Preference Optimization with Attention

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights