cs.CLApr 14, 2026

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Authors: Shanyong Wang, Shuhang Lin, Yining Zhao, Xi Zhu, Yongfeng Zhang

Organizations: Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.

Figures & tables

Figure 1: Two key characteristics in aligning individual human preferences. (Left) Human preferences are diverse and heterogeneous. (Right) One person’s preference can be conflict about the same thing and keeps changing due to various reasons.
Figure 2: Illustration of our Preference-Paired Fine-Tuning (PFT) framework. (a) Traditional single-preference training optimizes the model with respect to either the positive or negative side of a preference, (b) Our method leverages preference-paired data to train a mixed model that integrates both sides, and then infer the preference vector to adapt to a specific preference, (c) Example scenario: given one prompt with two contradictory preference responses (risk-taking vs. risk-averse), (d) Rapid customization: with user history data, the model can be steered toward a user’s target preference via ICL, (e) Experimental results show that PFT improves both classification accuracy and text generation alignment compared to single-preference training.
Figure 3: Effect of training dataset size on model performance. Accuracy generally improves as the number of training examples increases, but gains begin to plateau beyond 1000 samples. We therefore use 1000 examples as the standard training size in our main experiments, as it provides a good trade-off between data efficiency and performance stability. The trend suggests that the benefit of additional data diminishes after this point, likely due to the model already capturing the dominant preference signals.
Figure 4: Results across preference dimensions. Upper figure and Lower figure report classification accuracy and open-ended human evaluation scores, respectively. Models trained with paired preferences (Pair) consistently outperform single-preference (Single) and base models (Base), achieving higher accuracy and more balanced alignment across most preference types.
Figure 5: Hyperparameter analysis on the weighting coefficients λp+\lambda_{p_{+}} (top axis) and λp−\lambda_{p_{-}} (bottom axis) when training Qwen2.5-3B-Instruct. From left to right along the x-axis, λp+\lambda_{p_{+}} gradually increases while λp−\lambda_{p_{-}} remains fixed. Once λp+\lambda_{p_{+}} reaches 1.0, λp−\lambda_{p_{-}} then starts to decrease.
Figure 6: Results of training with different numbers of preference pairs across four preference types (risk, social, time, decision-making). Single Pair is trained with 1,000 examples, while Multi ( kk ) denotes multi-pair training with kk examples per preference pair.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: VCD Behavior Definition
Figure 8: Dataset Generation Template
Figure 9: Detaset Details about Number of multiple-choice options.
Figure 10: BQD Behavior Definition
Figure 11: CAA Layer Selection. For Qwen2.5-3B, Qwen2.5-7B, and Llama-3.1-8B, the models contain 36, 28, and 32 layers, respectively. We observe that layer 16 and nearby layers have the greatest impact on model preference alignment. Therefore, we select layer 16 for all three models.
Figure 12: BQD Open-ended Question Evaluation prompt
Figure 13: VCD Open-ended Question Evaluation prompt
More in this topic

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 746 papers

Explore similar work

CardsList

  1. Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

    Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM AlignmentRepresentation Engineering

    University of Technology Sydney, Australia

  2. AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

    Sep 5, 2026Peng Lai, He Zhu, Zhiwen Ruan +6Data SelectionPreference Alignment

    Southern University of Science and Technology · Peking University · MSRA +2

  3. Differentially Private Preference Data Synthesis for Large Language Model Alignment

    May 29, 2026Fengyu Gao, Jing YangPairwise Preference LearningLLM Alignment

    Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA.