CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment

Jilong Liu; Yonghui Yang; Pengyang Shao; Wenjian Tao; Hao Zhan; Haokai Ma; Wei Qin; Richang Hong

cs.LGMar 7, 2026

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment

Authors: Jilong Liu, Yonghui Yang, Pengyang Shao, Wenjian Tao, Hao Zhan, Haokai Ma, Wei Qin, Richang Hong

Organizations: Hefei University of Technology · National University of Singapore

Abstract

Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00

Figures & tables

Figure 1: Training-dynamics view of DPO under imperfect preference supervision. Left: A reward-model gap $\Delta r=r_{\mathrm{RM}}(y_{w}\mid x)-r_{\mathrm{RM}}(y_{l}\mid x)$ visualizes three pair-level regimes: preference conflict, low confidence, and reliable update. The reward model is used only for visualization. Right: Standard DPO updates can be dominated by unreliable or high-loss pairs, leading to a distorted optimization trajectory. Compass DPO bounds sample influence along update direction and magnitude, resulting in more stable preference optimization.

Figure 2: Diagnostic training dynamics of DPO and Compass DPO on Pythia-2.8B. DPO is run for 3 epochs only to expose the evolution of gradient dominance, while Compass DPO is shown under its default 1-epoch training setup. (a,d) Tail energy share: the fraction of gradient energy contributed by the top-energy samples within each mini-batch. (b,e) Energy concentration measured by the Herfindahl–Hirschman Index (HHI). (c,f) Preference separation during training, measured by DPO margin and preference accuracy.

Methods	Pythia-2.8B			Llama-3.2-3B			Llama-3-8B			Qwen2.5-7B
	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$
Vanilla	40.18	73.31	26.94	44.93	66.79	18.42	42.82	67.54	24.69	69.09	32.21	13.16
SFT	41.05	71.30	26.32	35.93	68.67	45.36	23.85	69.92	52.51	25.09	70.55	52.88
DPO	44.23	57.14	19.17	73.70	11.28	6.14	69.19	26.57	14.41	75.81	20.55	7.27
IPO	45.27	61.65	22.68	72.70	11.78	6.14	70.22	26.82	13.53	73.97	21.30	8.02
cDPO	45.27	60.28	16.92	72.91	25.44	12.78	59.48	37.94	23.43	67.65	31.83	13.66

Table 1: Results on PKU-SafeRLHF-30K (test split) across four backbones. Methods include Compass DPO and DPO-family baselines. We report WR (%) and ASR (%), where ASR is measured by two judges (MD and NV).

Methods	Do-Not-Answer		HarmBench		HH-RLHF		Salad Bench		AVG.
	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$
Llama-3-8B	54.58	12.37	87.50	23.50	65.10	19.92	73.14	23.26	46.27
SFT	53.52	33.48	93.50	81.50	69.27	47.54	75.10	58.68	64.03
DPO	14.18	7.78	41.00	28.50	23.05	10.30	31.55	19.75	22.85
IPO	14.29	7.36	33.50	17.50	21.70	8.30	30.12	15.69	20.42
cDPO	22.39	13.86	61.00	44.00	30.32	13.34	42.53	26.44	30.7

Table 2: Out-of-distribution ASR on four external safety benchmarks for models trained on PKU-SafeRLHF-30K. Results are shown for Llama-3-8B and Qwen2.5-7B across Compass DPO and DPO-family baselines. ASR is measured by two judges, and AVG reports the mean ASR over the four benchmarks.

Methods	0 Flipped			10 Flipped			20 Flipped			30 Flipped
	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$	WR $\uparrow$	MD $\downarrow$	NV $\downarrow$
DPO	44.23	57.14	19.17	45.07	60.78	17.42	43.16	66.29	23.93	41.82	69.05	25.19
rDPO	46.54	57.77	23.43	46.3	59.02	20.55	44.10	60.90	21.43	42.96	65.65	22.43
cDPO	45.27	60.28	16.92	44.43	66.17	23.18	43.86	67.04	20.18	42.72	71.3	22.68
IPO	45.27	61.65	22.68	43.61	62.53	23.81	43.36	65.91	23.18	42.23	68.80	24.68
Dr.DPO	57.41	17.04	4.14	56.57	19.92	7.52	54.81	24.56	7.77	52.93	31.95	6.39

Table 3: Results on PKU-SafeRLHF-30K (test split) under synthetic label-flip noise for Pythia-2.8B. A fixed fraction of training preference pairs is randomly flipped (0/10/20/30%).

Figure 3: Hyperparameter sensitivity on Pythia-2.8B, evaluated on PKU-SafeRLHF-30K (test split). Each panel varies one hyperparameter while keeping all others fixed to the default settings in the paper: (a) directional-control budget

\rho_{f}

, (b) cagnitude-control strength

\rho_{w}^{\max}

, and (c) tail threshold quantile

q

Figure 4: Ablation results with Pythia-2.8B. Bars show WR on the PKU-SafeRLHF-30K test split. The line shows average ASR across five benchmarks, measured by the two judges. We compare standard DPO, Compass DPO with directional control only (Dir. only), magnitude control only (Mag. only), and the full method.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Training	Test
PKU-SafeRLHF-30K	26,874	2,989
Do-Not-Answer	–	938
HarmBench	–	200
HH-RLHF	–	8,273
MMLU-Pro	–	1,400
MT-Bench	–	80

Appendix

Table 4: Samples statistic of the benchmarks used in this work.

Methods	Do-Not-Answer		HarmBench		HH-RLHF		Salad Bench		AVG.
	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$	MD $\downarrow$	NV $\downarrow$
Pythia-2.8B	59.70	18.34	86.00	29.50	72.02	24.95	78.95	29.69	52.30
SFT	62.26	18.44	81.00	28.00	72.21	25.75	79.05	30.23	52.68
DPO	46.26	14.50	65.00	15.00	58.82	21.06	63.60	20.79	41.21
IPO	44.99	16.52	73.00	16.50	59.49	22.78	64.55	22.74	42.58
cDPO	47.55	13.33	76.00	14.50	60.33	18.08	67.61	19.24	41.90

Appendix

Table 5: Out-of-distribution ASR on four external safety benchmarks for models trained on PKU-SafeRLHF-30K, reported for Pythia-2.8B and Llama-3.2-3B across Compass DPO and DPO-family baselines.

Figure 5: Sensitivity to the flip warmup ratio

\alpha

on Pythia-2.8B. WR is measured on the PKU-SafeRLHF test set. ASR is averaged over five benchmarks.

Figure 6: General capability evaluation on Llama-3.2-3B. MMLU-Pro reports accuracy and MT-Bench reports average score; higher is better for both.

Figure 7: OR-Bench evaluation on Llama-3.2-3B. OR-Hard-1K measures benign over-refusal, where lower is better, while OR-Toxic measures harmful-query attack success, where lower is better.

Figure 8: Comparison between Compass DPO and hard truncation (HT) on Llama-3.2-3B.

Figure 9: Effect of batch size on Llama-3.2-3B.

Backbone	DPO	Compass DPO
Llama-3.2-3B	$21{:}36\pm 8$ s	$21{:}34\pm 12$ s
Llama-3-8B	$46{:}35\pm 15$ s	$46{:}47\pm 11$ s

Appendix

Table 6: Computational overhead comparison between DPO and Compass DPO.

Figure 10: Robustness to temporally concentrated label flips on Pythia-2.8B.

Method	WR $\uparrow$	MMLU $\uparrow$	MT $\uparrow$
Base	22.32	15.29	3.69
DPO	78.13	23.29	3.28
Dr.DPO	80.75	23.43	3.24
Compass DPO	81.26	23.26	3.37

Appendix

Table 7: Non-safety preference learning results on Llama-3.2-3B using the helpfulness labels from HH.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 162 papers

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
Jun 4, 2026Yi Nian, Tiankai Yang, Yudi Zhang +7Direct Preference Optimization LLM Safety Alignment
University of Southern California · Iowa State University · University of Wisconsin–Madison +4
Curriculum Learning for Safety Alignment
May 25, 2026Sandeep Kumar, Virginia Smith, Chhavi YadavLLM Alignment Direct Preference Optimization
Carnegie Mellon University · Simons Institute, UC Berkeley
Improving Safety Alignment via Balanced Direct Preference Optimization
Mar 24, 2026Shiji Zhao, Mengyang Wang, Shukun Xiong +7LLM Safety Alignment LLM Alignment
Institute of Artificial Intelligence, Beihang University, Beijing, China.

Jun 4, 2026J/K move · Enter open · S save

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

Yi Nian, Tiankai Yang, Yudi Zhang, Qi Pan+6

University of Southern California · Iowa State University · University of Wisconsin–Madison+4

Direct Preference Optimization LLM Safety Alignment Multi-Objective Language Model Alignment Preference Alignment

Open paper AskPDF arXiv

Safety alignment for large language models relies on preference data, but current pipelines often train on large, redundant datasets. Existing data selection methods typically score each preference pair independently, collapsing directional preference information into scalar quality or diversity scores. This sample-centric view is especially limiting in multi-dataset settings, where shared safety directions coexist with dataset-specific residual risks. We propose DOG-DPO, a training-free data selection framework that treats preference pairs as structured geometric signals. DOG-DPO first represents each preference pair as a direction in model representation space. It then decomposes multi-dataset preference geometry into a global anchor subspace and dataset-specific residual subspaces. Finally, it selects subsets by maximizing diversity-based coverage, encouraging broad, non-redundant coverage of alignment directions before DPO training. Across six safety benchmarks and two model backbones, DOG-DPO achieves a strong utility-robustness trade-off using only 11% of the preference pairs. It recovers most of the safety gains of full-data training while remaining entirely teacher-free, training-free, and substantially faster than representative selection baselines.

Abstract

Direct Preference Optimization

Explore similar work

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

Curriculum Learning for Safety Alignment

Improving Safety Alignment via Balanced Direct Preference Optimization