Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion

Yuanhong Wu; Djallel Bouneffouf; D. Frank Hsu

cs.MAMar 11, 2026

Enhancing Value Alignment of LLMs with Multi-agent system and Combinatorial Fusion

Authors: Yuanhong Wu, Djallel Bouneffouf, D. Frank Hsu

Organizations: Dept. of Computer and Information Science, Fordham University, New York, NY, USA · T. J. Watson Research Center, IBM Research, Yorktown Heights, NY, USA

Abstract

Aligning large language models (LLMs) with human values is a central challenge for ensuring trustworthy and safe deployment. While existing methods such as Reinforcement Learning from Human Feedback (RLHF) and its variants have improved alignment, they often rely on a single evaluator or narrowly defined reward signals, limiting their ability to capture ethical pluralism. In this work, we propose the Value Alignment System using Combinatorial Fusion Analysis (VAS-CFA), a framework that operationalizes multi-agent fusion alignment. It instantiates multiple moral agents, each fine-tuned to represent a distinct normative perspective, and fuses their outputs using CFA with both rank- and score-based aggregation. This design leverages cognitive diversity, between agents, to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect human values. Empirical evaluation demonstrates that VAS-CFA outperforms both single agent baselines and prior aggregation approaches on standard metrics, showing that multi-agent fusion provides a robust and effective mechanism for advancing value alignment in LLMs.

Figures & tables

Figure 1: The diagram for Value Alignment System using Combinatorial Fusion Analysis (VAS-CFA).

Panel of Figure 2 — Figure 2: Rank-score function graph for the question $q_{8657}$ : $f_{A},f_{B},f_{C},f_{D}$ and $f_{E}$ refer to agent A, B, C, D and E w.r.t. Authority, Care, Fairness, Loyalty and Sanctity, respectively.

Panel of Figure 3 — Figure 3: F1 BERTScore across 26 combinations under four CFA combination types (ASC, WSCDS, ARC, WRCDS) for the question $q_{8657}$ (ASC sorted in non-decreasing order in each model group).

Model	(a) $F_{1}$ ROUGE-L	(b) $F_{1}$ BERTScore
(i). Individual moral agent
A	0.0925	0.8569
B	0.0821	0.8533
C	0.1249	0.8628
D	0.1376	0.8663
E	0.1343	0.8653

Table 1: Performance summary (ROUGE-L and

F_{1}

BERTScore).

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Contextual Value Alignment via Multilayer Combinatorial Fusion
Aug 7, 2026Yuanhong Wu, Djallel Bouneffouf, D. Frank HsuLLM Alignment Moral Reasoning in Language Models
Dept. of Computer and Information Science, Fordham University, New York, NY, USA · T. J. Watson Research Center, IBM Research, Yorktown Heights, NY, USA
VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
Mar 18, 2026Hefei Xu, Le Wu, Yu Wang +4LLM Alignment Multi-Objective Language Model Alignment
Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, Anhui, China · Innovation School of Artificial Intelligence, Hefei University of Technology
Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs
Mar 11, 2026Panatchakorn Anantaprayoon, Nataliia Babina, Nima Asgharbeygi +1Multi-Agent Deliberation LLM Alignment
Integral AI · The University of Tokyo · MATS

Aug 7, 2026J/K move · Enter open · S save

Contextual Value Alignment via Multilayer Combinatorial Fusion

Yuanhong Wu, Djallel Bouneffouf, D. Frank Hsu

Dept. of Computer and Information Science, Fordham University, New York, NY, USA · T. J. Watson Research Center, IBM Research, Yorktown Heights, NY, USA

LLM Alignment Moral Reasoning in Language Models Multi-Agent LLMs Value Alignment

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values remains a major challenge, especially for trustworthy AI. While existing approaches such as RLHF, CAI, and their variants have achieved promising results, they often rely on a single-agent framework and a unified reward system. This limits their ability to capture ethical pluralism, adapt to diverse moral contexts, and reflect the dynamics of multi-agent moral reasoning. In this work, we propose a framework that utilizes multilayer combinatorial fusion for contextual value alignment (MCF-CVA). At the first layer of the framework, it instantiates multiple moral agents, each fine-tuned to represent a distinctive value. Their outputs are then expanded combinatorially using both score- and rank-combinations as well as average and weighted aggregations. These combined models are then reduced to the same number of initial moral agents. This expansion and reduction (EAR) process continues for multi-layers until a stopping criterion is reached. The MCF-CVA framework leverages cognitive diversity between agents to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect contextual human values. The framework using the EAR algorithm is performed on the dual architecture of Euclidean score space and Kemeny rank space. Empirical evaluations demonstrated that the proposed framework outperforms single-agent baselines, multi-agent single-layer results, and previous aggregation approaches on standard metrics, showing that the MCF-CVA framework provides a robust and effective mechanism for advancing contextual value alignment in LLMs.

Abstract

LLM Alignment

Explore similar work

Contextual Value Alignment via Multilayer Combinatorial Fusion

VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs