Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00
Figures & tables
Figure 1: Training-dynamics view of DPO under imperfect preference supervision. Left: A reward-model gap Δr=rRM(yw∣x)−rRM(yl∣x) visualizes three pair-level regimes: preference conflict, low confidence, and reliable update. The reward model is used only for visualization. Right: Standard DPO updates can be dominated by unreliable or high-loss pairs, leading to a distorted optimization trajectory. Compass DPO bounds sample influence along update direction and magnitude, resulting in more stable preference optimization.
Figure 2: Diagnostic training dynamics of DPO and Compass DPO on Pythia-2.8B. DPO is run for 3 epochs only to expose the evolution of gradient dominance, while Compass DPO is shown under its default 1-epoch training setup. (a,d) Tail energy share: the fraction of gradient energy contributed by the top-energy samples within each mini-batch. (b,e) Energy concentration measured by the Herfindahl–Hirschman Index (HHI). (c,f) Preference separation during training, measured by DPO margin and preference accuracy.
Methods
Pythia-2.8B
Llama-3.2-3B
Llama-3-8B
Qwen2.5-7B
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
Vanilla
40.18
73.31
26.94
44.93
66.79
18.42
42.82
67.54
24.69
69.09
32.21
13.16
SFT
41.05
71.30
26.32
35.93
68.67
45.36
23.85
69.92
52.51
25.09
70.55
52.88
DPO
44.23
57.14
19.17
73.70
11.28
6.14
69.19
26.57
14.41
75.81
20.55
7.27
IPO
45.27
61.65
22.68
72.70
11.78
6.14
70.22
26.82
13.53
73.97
21.30
8.02
cDPO
45.27
60.28
16.92
72.91
25.44
12.78
59.48
37.94
23.43
67.65
31.83
13.66
Table 1: Results on PKU-SafeRLHF-30K (test split) across four backbones. Methods include Compass DPO and DPO-family baselines. We report WR (%) and ASR (%), where ASR is measured by two judges (MD and NV).
Methods
Do-Not-Answer
HarmBench
HH-RLHF
Salad Bench
AVG.
MD ↓
NV ↓
MD ↓
NV ↓
MD ↓
NV ↓
MD ↓
NV ↓
Llama-3-8B
54.58
12.37
87.50
23.50
65.10
19.92
73.14
23.26
46.27
SFT
53.52
33.48
93.50
81.50
69.27
47.54
75.10
58.68
64.03
DPO
14.18
7.78
41.00
28.50
23.05
10.30
31.55
19.75
22.85
IPO
14.29
7.36
33.50
17.50
21.70
8.30
30.12
15.69
20.42
cDPO
22.39
13.86
61.00
44.00
30.32
13.34
42.53
26.44
30.7
Table 2: Out-of-distribution ASR on four external safety benchmarks for models trained on PKU-SafeRLHF-30K. Results are shown for Llama-3-8B and Qwen2.5-7B across Compass DPO and DPO-family baselines. ASR is measured by two judges, and AVG reports the mean ASR over the four benchmarks.
Methods
0 Flipped
10 Flipped
20 Flipped
30 Flipped
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
WR ↑
MD ↓
NV ↓
DPO
44.23
57.14
19.17
45.07
60.78
17.42
43.16
66.29
23.93
41.82
69.05
25.19
rDPO
46.54
57.77
23.43
46.3
59.02
20.55
44.10
60.90
21.43
42.96
65.65
22.43
cDPO
45.27
60.28
16.92
44.43
66.17
23.18
43.86
67.04
20.18
42.72
71.3
22.68
IPO
45.27
61.65
22.68
43.61
62.53
23.81
43.36
65.91
23.18
42.23
68.80
24.68
Dr.DPO
57.41
17.04
4.14
56.57
19.92
7.52
54.81
24.56
7.77
52.93
31.95
6.39
Table 3: Results on PKU-SafeRLHF-30K (test split) under synthetic label-flip noise for Pythia-2.8B. A fixed fraction of training preference pairs is randomly flipped (0/10/20/30%).
Figure 3: Hyperparameter sensitivity on Pythia-2.8B, evaluated on PKU-SafeRLHF-30K (test split). Each panel varies one hyperparameter while keeping all others fixed to the default settings in the paper: (a) directional-control budget ρf , (b) cagnitude-control strength ρwmax , and (c) tail threshold quantile q .
Figure 4: Ablation results with Pythia-2.8B. Bars show WR on the PKU-SafeRLHF-30K test split. The line shows average ASR across five benchmarks, measured by the two judges. We compare standard DPO, Compass DPO with directional control only (Dir. only), magnitude control only (Mag. only), and the full method.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Training
Test
PKU-SafeRLHF-30K
26,874
2,989
Do-Not-Answer
–
938
HarmBench
–
200
HH-RLHF
–
8,273
MMLU-Pro
–
1,400
MT-Bench
–
80
Appendix
Table 4: Samples statistic of the benchmarks used in this work.
Methods
Do-Not-Answer
HarmBench
HH-RLHF
Salad Bench
AVG.
MD ↓
NV ↓
MD ↓
NV ↓
MD ↓
NV ↓
MD ↓
NV ↓
Pythia-2.8B
59.70
18.34
86.00
29.50
72.02
24.95
78.95
29.69
52.30
SFT
62.26
18.44
81.00
28.00
72.21
25.75
79.05
30.23
52.68
DPO
46.26
14.50
65.00
15.00
58.82
21.06
63.60
20.79
41.21
IPO
44.99
16.52
73.00
16.50
59.49
22.78
64.55
22.74
42.58
cDPO
47.55
13.33
76.00
14.50
60.33
18.08
67.61
19.24
41.90
Appendix
Table 5: Out-of-distribution ASR on four external safety benchmarks for models trained on PKU-SafeRLHF-30K, reported for Pythia-2.8B and Llama-3.2-3B across Compass DPO and DPO-family baselines.
Figure 5: Sensitivity to the flip warmup ratio α on Pythia-2.8B. WR is measured on the PKU-SafeRLHF test set. ASR is averaged over five benchmarks.
Figure 6: General capability evaluation on Llama-3.2-3B. MMLU-Pro reports accuracy and MT-Bench reports average score; higher is better for both.
Figure 7: OR-Bench evaluation on Llama-3.2-3B. OR-Hard-1K measures benign over-refusal, where lower is better, while OR-Toxic measures harmful-query attack success, where lower is better.
Figure 8: Comparison between Compass DPO and hard truncation (HT) on Llama-3.2-3B.
Figure 9: Effect of batch size on Llama-3.2-3B.
Backbone
DPO
Compass DPO
Llama-3.2-3B
21:36±8 s
21:34±12 s
Llama-3-8B
46:35±15 s
46:47±11 s
Appendix
Table 6: Computational overhead comparison between DPO and Compass DPO.
Figure 10: Robustness to temporally concentrated label flips on Pythia-2.8B.
Method
WR ↑
MMLU ↑
MT ↑
Base
22.32
15.29
3.69
DPO
78.13
23.29
3.28
Dr.DPO
80.75
23.43
3.24
Compass DPO
81.26
23.26
3.37
Appendix
Table 7: Non-safety preference learning results on Llama-3.2-3B using the helpfulness labels from HH.
Safety alignment for large language models relies on preference data, but current pipelines often train on large, redundant datasets. Existing data selection methods typically score each preference pair independently, collapsing directional preference information into scalar quality or diversity scores. This sample-centric view is especially limiting in multi-dataset settings, where shared safety directions coexist with dataset-specific residual risks. We propose DOG-DPO, a training-free data selection framework that treats preference pairs as structured geometric signals. DOG-DPO first represents each preference pair as a direction in model representation space. It then decomposes multi-dataset preference geometry into a global anchor subspace and dataset-specific residual subspaces. Finally, it selects subsets by maximizing diversity-based coverage, encouraging broad, non-redundant coverage of alignment directions before DPO training. Across six safety benchmarks and two model backbones, DOG-DPO achieves a strong utility-robustness trade-off using only 11% of the preference pairs. It recovers most of the safety gains of full-data training while remaining entirely teacher-free, training-free, and substantially faster than representative selection baselines.
Yi Nian, Tiankai Yang, Yudi Zhang +7
University of Southern California · Iowa State University · University of Wisconsin–Madison +4
Direct Preference Optimisation (DPO) is widely used for safety alignment in large language models. However, prior work shows it is brittle and exhibits poor out-of-distribution (OOD) generalisation. In this paper, we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organises preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16% and jailbreak attack success rates by 20%, while preserving general capabilities with near-zero over-refusal. We further show that Staged-Competence (1) matches baseline safety with only 75% of the training data and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the policy optimisation loss and can extend to other DPO variants and alignment domains. Our code and data are available at https://github.com/Sandeep5500/curriculum-learning-for-safety.
Sandeep Kumar, Virginia Smith, Chhavi Yadav
Carnegie Mellon University · Simons Institute, UC Berkeley
With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. However, safety alignment still suffers from severe overfitting, which limits its actual performance. This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pairs, which compromises the model's safety performance. To address this, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on mutual information. A series of experimental results show that B-DPO can enhance the safety capability while maintaining the competitive general capabilities of LLMs on various mainstream benchmarks compared to state-of-the-art methods. \color{red}{Warning: This paper contains examples of harmful texts, and reader discretion is recommended.
Shiji Zhao, Mengyang Wang, Shukun Xiong +7
Institute of Artificial Intelligence, Beihang University, Beijing, China.