Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment
Organizations: Department of Information Technology Uppsala University Uppsala, 75120
Abstract
Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Values |
|---|---|
| MeZO | |
| Learning rate | |
| DPO coefficient | |
| Perturbation scale | |
| Effective batch size | |
| CA-ZODPO | |
| Judge | Comparison | Win | Tie | Lose |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | MeZO vs. SFT | 25.0 | 39.8 | 35.2 |
| CA-ZODPO vs. SFT | 34.2 | 39.4 | 26.4 | |
| FO-DPO vs. SFT | 21.6 | 61.4 | 17.0 | |
| MeZO vs. FO-DPO | 23.6 | 42.2 | 34.2 | |
| CA-ZODPO vs. FO-DPO | 31.6 | 41.2 | 27.2 | |
| CA-ZODPO vs. MeZO | 36.0 | 38.2 | 25.8 |
| Comparison | Win | Tie | Lose |
|---|---|---|---|
| CA-ZODPO-plain vs. SFT | 30.6 | 37.8 | 31.6 |
| CA-ZODPO-plain vs. FO-DPO | 27.8 | 37.6 | 34.6 |
| CA-ZODPO-plain vs. MeZO | 37.2 | 34.2 | 28.6 |
| CA-ZODPO-plain vs. CA-ZODPO | 31.8 | 32.8 | 35.4 |