VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
Authors: Hefei Xu, Le Wu, Yu Wang, Min Hou, Han Wu, Zhen Zhang, Meng Wang
Organizations: Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, Anhui, China · Innovation School of Artificial Intelligence, Hefei University of Technology
As large language models (LLMs) increasingly shape content generation, interaction, and decision-making across the Web, aligning them with human values has become a central objective in trustworthy AI. This challenge becomes even more pronounced when aligning multiple, potentially conflicting human values. Although recent approaches, such as reward reweighting, prompt-based supervised fine-tuning, and model merging, attempt to tackle multi-value alignment, they still face two major limitations: (1) training separate models for each value combination is prohibitively expensive; (2) value conflicts substantially degrade alignment performance. These limitations make it difficult to achieve favorable trade-offs across diverse human values. To address these challenges, we revisit multi-value alignment from the perspective of value consistency in data and propose VC-soup, a data filtering and parameter merging framework grounded in value-consistent learning. We first design a value consistency metric based on the cosine similarity between the reward-gap vector of each preference pair and an all-ones vector, which quantifies its cross-value coherence. We then filter out low-consistency preference pairs in each value dataset and train on the remaining data to obtain smooth, value-consistent policy models that better preserve linear mode connectivity. Finally, we linearly combine these policies and apply Pareto filtering across values to obtain solutions with balanced multi-value performance. Extensive experiments and theoretical analysis demonstrate that VC-soup effectively mitigates conflicts and consistently outperforms existing multi-value alignment methods.
Figures & tables
Figure 1. Reward-gap distributions of samples in two value dimensions (helpfulness vs. harmlessness) on Anthropic-HH. A large proportion of samples fall into mixed-sign regions, indicating fundamental disagreement among human values.
Figure 2. The framework of VC-soup.
Figure 3. Pareto Frontiers of VC-soup and Baselines on Anthropic-HH and BeaverTails. A curve closer to the upper-right indicates better alignment performance.
Figure 4. Winrates of VC-soup against baselines on Anthropic-HH (a,b) and BeaverTails (c,d).
Method
helpful
harmless
honest
Avg.
Base
-0.269
-0.518
-3.242
-1.343
DPO-Help
2.371
-2.555
-6.533
-2.238
DPO-Harm
-1.619
1.115
-14.386
-4.963
DPO-Honest
1.457
-0.851
3.797
1.465
SeqT
0.334
-0.139
-13.867
-4.557
SOUP
1.277
-1.381
0.678
0.191
Table 1. Three-value alignment performance.
Figure 5. Value consistency improves model merging: Higher τ enables better Pareto trade-offs.
Figure 6. L2 distance and cosine similarity between VS/VC vectors across layers.
Figure 7. Impact of τ and λ .
Method
Filtering
Merging
Helpful
Harmless
Base
×
×
-0.269
-0.518
DPO-Help
×
×
2.371
-2.555
DPO-Harm
×
×
-1.619
1.115
Only merge
×
✓
1.624
-0.01
DPO- Helpτ=0.7
✓
×
2.095
0.116
DPO- Harmτ=0.7
✓
×
1.248
0.71
Table 2. Ablation study on filtering and merging strategies.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Case Study.
Figure 9. Prompt template for GPT-4 response evaluation.
Aligning large language models (LLMs) with human values is a central challenge for ensuring trustworthy and safe deployment. While existing methods such as Reinforcement Learning from Human Feedback (RLHF) and its variants have improved alignment, they often rely on a single evaluator or narrowly defined reward signals, limiting their ability to capture ethical pluralism. In this work, we propose the Value Alignment System using Combinatorial Fusion Analysis (VAS-CFA), a framework that operationalizes multi-agent fusion alignment. It instantiates multiple moral agents, each fine-tuned to represent a distinct normative perspective, and fuses their outputs using CFA with both rank- and score-based aggregation. This design leverages cognitive diversity, between agents, to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect human values. Empirical evaluation demonstrates that VAS-CFA outperforms both single agent baselines and prior aggregation approaches on standard metrics, showing that multi-agent fusion provides a robust and effective mechanism for advancing value alignment in LLMs.
Yuanhong Wu, Djallel Bouneffouf, D. Frank Hsu
Dept. of Computer and Information Science, Fordham University, New York, NY, USA · T. J. Watson Research Center, IBM Research, Yorktown Heights, NY, USA
Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.
Minjae Lee, Kyunghyun Cho, Sangdon Park
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Aligning Large Language Models (LLMs) with nuanced human values remains a critical challenge, as existing methods like Reinforcement Learning from Human Feedback (RLHF) often handle only coarse-grained attributes. In practice, fine-tuning LLMs on task-specific datasets to optimize value alignment inevitably incurs an alignment tax: the model's pre-calibrated value system drifts significantly due to latent bias absorption from training data, while the fine-tuning process also causes severe hallucinations and semantic information loss in generated responses. To address this, we propose VISA (Value Injection via Shielded Adaptation), a closed-loop framework designed to navigate this trade-off. VISA's architecture features a high-precision value detector, a semantic-to-value translator, and a core value-rewriter. The value-rewriter is trained via Group Relative Policy Optimization (GRPO) with a composite reward function that simultaneously optimizes for fine-grained value precision, and the preservation of semantic integrity. By learning an optimal policy to balance these competing objectives, VISA effectively mitigates the alignment tax while staying loyal to the original knowledge. Our experiments demonstrate that this approach enables precise control over a model's value expression while maintaining its factual consistency and general capabilities, significantly outperforming both standard fine-tuning methods and prompting-based baselines, including GPT-4o.