Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.
Figures & tables
Figure 1: Two key characteristics in aligning individual human preferences. (Left) Human preferences are diverse and heterogeneous. (Right) One person’s preference can be conflict about the same thing and keeps changing due to various reasons.
Figure 2: Illustration of our Preference-Paired Fine-Tuning (PFT) framework. (a) Traditional single-preference training optimizes the model with respect to either the positive or negative side of a preference, (b) Our method leverages preference-paired data to train a mixed model that integrates both sides, and then infer the preference vector to adapt to a specific preference, (c) Example scenario: given one prompt with two contradictory preference responses (risk-taking vs. risk-averse), (d) Rapid customization: with user history data, the model can be steered toward a user’s target preference via ICL, (e) Experimental results show that PFT improves both classification accuracy and text generation alignment compared to single-preference training.
Figure 3: Effect of training dataset size on model performance. Accuracy generally improves as the number of training examples increases, but gains begin to plateau beyond 1000 samples. We therefore use 1000 examples as the standard training size in our main experiments, as it provides a good trade-off between data efficiency and performance stability. The trend suggests that the benefit of additional data diminishes after this point, likely due to the model already capturing the dominant preference signals.
Figure 4: Results across preference dimensions. Upper figure and Lower figure report classification accuracy and open-ended human evaluation scores, respectively. Models trained with paired preferences (Pair) consistently outperform single-preference (Single) and base models (Base), achieving higher accuracy and more balanced alignment across most preference types.
Figure 5: Hyperparameter analysis on the weighting coefficients λp+ (top axis) and λp− (bottom axis) when training Qwen2.5-3B-Instruct. From left to right along the x-axis, λp+ gradually increases while λp− remains fixed. Once λp+ reaches 1.0, λp− then starts to decrease.
Figure 6: Results of training with different numbers of preference pairs across four preference types (risk, social, time, decision-making). Single Pair is trained with 1,000 examples, while Multi ( k ) denotes multi-pair training with k examples per preference pair.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: VCD Behavior Definition
Figure 8: Dataset Generation Template
Figure 9: Detaset Details about Number of multiple-choice options.
Figure 10: BQD Behavior Definition
Figure 11: CAA Layer Selection. For Qwen2.5-3B, Qwen2.5-7B, and Llama-3.1-8B, the models contain 36, 28, and 32 layers, respectively. We observe that layer 16 and nearby layers have the greatest impact on model preference alignment. Therefore, we select layer 16 for all three models.
Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.
Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.
Peng Lai, He Zhu, Zhiwen Ruan +6
Southern University of Science and Technology · Peking University · MSRA +2
Preference alignment is a crucial post-training step for large language models (LLMs) to ensure their outputs align with human values. However, post-training on real human preference data raises privacy concerns, as these datasets often contain sensitive user prompts and human judgments. To address this, we propose DPPrefSyn, a novel algorithm for generating differentially private (DP) synthetic preference data to enable privacy-preserving preference alignment. DPPrefSyn is a principled framework grounded in the Bradley-Terry preference model and the intrinsic geometric structure of pairwise human preference data. It first learns an underlying preference model from private data with formal differential privacy guarantees, and then leverages the learned model together with public prompts to synthesize high-quality preference data. It exploits the shared linear structure of per-cluster reward models to effectively capture heterogeneous human preferences in private datasets, and leverages DP Principal Component Analysis (DP-PCA) to improve learning accuracy. Extensive experimental results demonstrate that DPPrefSyn achieves competitive alignment performance under strong DP guarantees. These findings highlight the potential of synthetic preference data as a practical alternative for privacy-preserving preference alignment across a broad range of applications. To the best of our knowledge, this is the first work to generate DP synthetic preference data for LLM alignment. Our code is available at https://github.com/gfengyu/Differentially-Private-Preference-Data-Synthesis.
Fengyu Gao, Jing Yang
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA.