Direct Preference Optimization (DPO) and related methods align large language models from pairwise preferences by regularizing updates against a fixed reference policy. As the policy drifts, a static reference, however, can become increasingly miscalibrated, leading to distributional mismatch and amplifying spurious preference signals under noisy supervision. Conversely, reference-free variants avoid mismatch but often suffer from unconstrained reward drift. We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor: an adversarial local perturbation of the current policy within a small radius that serves as a pessimistic baseline. This anchor enables an adaptive reweighting mechanism, modulating the importance of each preference pair based on its local sensitivity. We further introduce the Anchor Gap, the reward discrepancy between the policy and its anchor, and show under smoothness conditions that it approximates worst-case local margin degradation. Optimizing a logistic objective weighted by this gap downweights geometrically brittle instances while emphasizing robust preference signals. Across diverse noise settings, GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.
Figures & tables
Figure 1: Conceptual comparison of DPO, SimPO, and GAPO. (a) DPO’s frozen reference causes distributional mismatch as πθ drifts. (b) SimPO has no anchor, allowing unconstrained reward drift . (c) GAPO’s anchor tracks the policy and probes local geometry.
Method
Mistral-Instruct (7B)
Llama-3-Base (8B)
Llama-3-Instruct (8B)
Gemma-2-Instruct (9B)
AlpacaEval 2
Arena- Hard
AlpacaEval 2
Arena- Hard
AlpacaEval 2
Arena- Hard
AlpacaEval 2
Arena- Hard
LC (%)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
LC (%)
WR (%)
WR (%)
SFT
17.1
14.7
12.6
6.2
4.6
3.3
26.0
25.3
22.3
48.7
36.5
42.1
DPO
26.8
24.9
16.3
18.2
15.5
15.9
40.3
37.9
32.6
70.4
66.9
58.8
IPO
20.3
20.3
16.2
14.4
14.2
17.8
35.6
35.6
30.5
62.6
58.4
53.5
CPO
23.8
28.8
22.6
10.8
8.1
5.8
28.9
32.2
28.8
56.4
53.4
55.2
Table 1: Main Results on Instruction-Following Benchmarks. We compare GAPO with various baselines across four models. GAPO is competitive with or improves over strong baselines across most model-benchmark settings. LC : Length-Controlled, WR : Win Rate.
Model
Mistral-7B
Llama-3-8B
Method
AlpacaEval 2
GSM8k
ARC-C
AlpacaEval 2
GSM8k
ARC-C
DPO
26.8
41.8
66.7
18.2
55.5
65.6
RDPO
27.3
42.6
67.5
17.6
54.9
65.2
SimPO
32.1
36.6
66.3
22.0
53.0
65.2
α -DPO
32.3
29.3
65.2
18.3
45.4
66.0
GAPO
35.7
40.3
66.3
25.0
56.0
67.2
Table 2: Reasoning vs. Alignment Trade-off across Models Performance comparison on Mistral-Instruct and Llama-3-Base.
Figure 2: Robustness against label noise. Reward accuracy under (a) random label flips and (b) length-dependent label flips, evaluated on Pythia-2.8B trained on Anthropic HH. The shaded region in each panel marks the gap between GAPO and the strongest baseline at each flip rate. Exact numerical values are reported in Appendix C.1 .
Method
AE2 LC (%)
AE2 WR (%)
SimPO
25.1
24.6
Dr.DPO
27.4
23.3
GAPO
27.8
29.9
Table 3: Robustness in the main Mistral-7B alignment setting. Mistral-7B-Instruct trained on UltraFeedback with 20% random preference-label flips. AlpacaEval 2 length-controlled (LC) and raw win rates (WR) are reported in percent.
Figure 3: GAPO’s Anchor-Gap reweighting is statistically biased toward flipped supervision during training (a) Mean GAPO weight wi=βσ(γ−βΓi) separates flipped from clean preferences over training. (b) Stratifying examples directly by wi at each checkpoint. Detailed numerical results are reported in Appendix D .
Figure 4: Batch-wise probe validation. (a) Distribution of the alignment coefficient αi,B . (b) Ranking agreement between the batch-directional first-order proxy and the instance-directional proxy. (c) Top- k% overlap of the most-downweighted samples between the two rankings. Detailed analysis is in Appendix F .
Method
AlpacaEval 2 LC
AlpacaEval 2 WR
GSM8k
ARC-C
SimPO
32.1
34.8
36.6
66.3
GradNorm-Reweight
33.5
34.7
37.0
66.1
GAPO
35.7
38.7
40.3
66.3
Table 4: Gradient-norm reweighting ablation on Mistral-Instruct.
Figure 5: Wall-clock-matched diagnostic on Mistral-7B under the same hardware and training recipe.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Mistral Inst
Llama-3 Base
Llama-3 Instruct
Gemma-2
Learning Rate
5e-7
6e-7
1e-6
8e-7
Global Batch Size
128
128
128
128
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Warmup Ratio
0.1
0.1
0.1
0.1
Max Length
2048
2048
2048
2048
GAPO Specifics
Appendix
Table A : Hyperparameters for GAPO and baselines.
Flip
Method
0%
10%
20%
30%
40%
Random
DPO
63.3
62.3
62.9
58.5
57.0
Dr.DPO
66.2
65.4
64.2
62.7
58.8
GAPO
66.8
65.6
64.9
62.8
59.8
Length- dependent
DPO
63.3
51.7
54.3
54.3
52.0
Dr.DPO
66.2
62.1
49.2
48.3
44.9
GAPO
66.8
63.3
58.3
56.6
52.0
Appendix
Table B : Robustness against label noise: numerical results. Reward accuracy (%) of DPO, Dr.DPO, and GAPO under random and length-dependent label flips at five flip rates. Best per row in bold.
Training seeds
Development seed
Method
Seed 1
Seed 2
Seed 3
Mean ± SD
Seed 42
SimPO
30.44
30.88
33.01
31.44±1.37
32.10
GAPO
33.89
34.34
35.30
34.51±0.72
35.70
Appendix
Table C : Effect of random seeds on Mistral-Instruct (7B). We report AlpacaEval 2 length-controlled win rate (LC, %). Mean ± SD is computed over seeds 1, 2, and 3 using sample standard deviation. The development seed (42) is shown separately. Bold indicates the higher LC in each column.
Sampling (30%)
Metric
AlpacaEval2
GSM8k
ARC-C
Random
-
12.8
54.3
63.6
Unstable
High Γ
9.80
54.8
63.4
Stable
Low Γ
13.8
56.2
64.2
Appendix
Table D : Auxiliary SimPO training on Anchor-Gap-ranked subsets. Comparison of SimPO models trained on 30% data subsets selected by the Anchor Gap ( Γ ). The Stable subset outperforms Random and Unstable.
Figure A : Prospective validation of the Anchor Gap. We compute Γi once at the SFT checkpoint, select low- Γ , random, and high- Γ subsets at three data fractions, and train SimPO on each subset.
Subset
Fraction
Val Acc
Val Margin
KL drift
Stable (low- Γ )
30%
0.621
0.043
0.056
Random
30%
0.610
0.045
0.180
Unstable (high- Γ )
30%
0.618
0.060
0.197
Stable (low- Γ )
50%
0.676
0.072
0.135
Random
50%
0.665
0.084
0.247
Unstable (high- Γ )
50%
0.672
0.096
0.262
Appendix
Table E : Full prospective sweep results. Validation metrics ( n=2,000 preference pairs) and KL drift from the SFT model, measured after training SimPO on Anchor-Gap-selected subsets. Best accuracy and lowest KL drift per fraction shown in bold.
Figure B : Training dynamics and outcomes across the fraction–subset grid. Top row (a–d): training-time mechanism metrics. Bottom row (e–h): post-training outcome metrics. Across all fractions, low- Γ subsets exhibit gentler training dynamics yet achieve higher validation accuracy at lower KL drift.
Figure C : Gradient-norm trajectories across fractions. Smoothed gradient norm (window =3 ) during SimPO training on Anchor-Gap-selected subsets. The dotted red line marks ∥∇∥=100 . High- Γ subsets exhibit systematically elevated gradients across all fractions, while low- Γ subsets remain bounded.
Radius ( ρ )
Mistral-Instruct
Llama-3-Base
0.025
37.9
20.8
0.05
38.7
23.7
0.075
37.3
20.5
0.10
37.0
20.2
Appendix
Table F : Effect of Perturbation Radius ( ρ ). Performance (WR) sensitivity across different perturbation magnitudes on Alpaca-Eval2.
Anchor batch multiplier
AlpacaEval 2 WR
GSM8k
ARC-C
1 ×
35.7
36.7
66.8
4 ×
37.2
39.0
66.2
8 ×
38.7
40.3
66.3
16 ×
36.5
38.6
66.1
Appendix
Table G : Sensitivity to anchor batch multiplier on Mistral-7B. The multiplier scales the per-device micro-batch size used to construct each shared perturbation; gradient-accumulation steps are adjusted inversely so that the effective optimizer batch size remains fixed.
Scope
WR
LM head
20.7
Full Parameters
23.7
Appendix
Table H : Effect of Perturbation Scope. Comparison between perturbing only the last layer versus full parameters in Llama3-Base.
Method
AlpacaEval 2
Arena-Hard
GSM8k
ARC-C
SimPO
32.1
21.0
36.6
66.3
SimPO+SAM
34.9
20.1
38.6
66.1
GAPO
35.7
22.7
40.3
66.3
Appendix
Table I : Comparison with SimPO+SAM. SAM introduces optimizer-level flatness-seeking, whereas GAPO uses Anchor-Gap-based instance-wise reweighting. SimPO+SAM improves over SimPO on some metrics but does not match GAPO’s overall alignment–reasoning trade-off.
Method
Training epochs
Wall-clock (h)
AE2 LC (%)
SimPO
1.00
1.50
32.1
GAPO
0.50
1.50
31.5
α -DPO
≈0.64
1.50
28.4
R-DPO
≈0.64
1.50
18.8
α -DPO
1.00
≈2.33
32.3
R-DPO
1.00
≈2.33
27.3
Appendix
Table J : Wall-clock comparison on Mistral-7B with 2× B200 GPUs. We report AlpacaEval 2 length-controlled win rate (LC, %) at the corresponding training checkpoints. Rows are grouped by runtime; bold indicates the best LC within each group. The 2.33-hour entries correspond to one epoch of α -DPO and R-DPO.
Method
λmax (Sharpness) ↓
Top-5 Avg ↓
λmin
DPO
149.6
103.4
-143.1
SimPO
158.8
98.9
-106.9
GAPO
145.1
94.9
-81.4
Appendix
Table K : Hessian Eigenspectrum Analysis. Comparison of the Hessian eigenvalues ( λ ) averaged over 1,000 samples from the training set.
Although Large Language Models (LLMs) have made remarkable progress, current preference optimization methods still struggle to align directional consistency while preserving reasoning diversity. To address this limitation, we propose Directional-Groupwise Preference Optimization (DGPO), a lightweight framework that aggregates supervision signals at the group level and explicitly models direction-aware alignment through multi-candidate comparisons. DGPO organizes forward and reverse question-answer instances into structured sets and optimizes a margin-based likelihood objective that separates coherent reasoning paths from inconsistent alternatives. This group-wise formulation captures richer relative information than pairwise objectives and reinforces consistency across diverse reasoning pathways. Empirical results show that our constructed reverse data yields a 3.2% average improvement across five benchmarks, while DGPO further delivers consistent gains across multiple datasets and model families, achieving average accuracy improvements of up to 3.6%.
Mengyi Deng, Zhiwei Li, Xin Li +4
1Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China · 2The Hong Kong University of Science and Technology, Hong Kong SAR
Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.96 and BA-DPO removes 81 to 95% of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.
Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.