cs.LGNov 16, 2026

Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment

Authors: Weiyun Huang, Alessio Galatolo

Organizations: Department of Information Technology Uppsala University Uppsala, 75120

Abstract

Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

    Jun 18, 2026Pranav Bhandari, Nicolas Fay, Amitava Datta +2Continual Learning for LLMsDirect Preference Optimization

  2. A Zeroth-Order Paradigm for LLM Preference Alignment

    Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference LearningLLM Alignment