Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.
Figures & tables
Figure 1: An illustration of the MAPL framework and methods. (a) Decentralized and centralized MAS; (b) MAPL framework; (c) Procedures of MARLHF and MADPO.
Figure 2: A comparison of MARLHF ( red ) and MADPO ( blue ) under Dec-Dec, Cen-Cen and Dec-Cen on TLDR , CoopHE , BFCL , and Travel across 5 runs. Dotted and dashed lines denote the performance of MAGRPO on decentralized and centralized MAS. Results are normalized to the reward scale w.r.t. each domain.
Method
TLDR
CoopHE
BFCL
Travel
Score
Pass
Call
Correct
Costs
Pass
Success
(a) Single Agent
Raw Model
30.3
56.3
N/A
47.5
2.4
71.3
9.6
GRPO ( Shao et al., 2024 )
91.7
61.8
N/A
50.2
2.5
74.5
25.0
PPO ( Schulman et al., 2017 )
94.5
62.5
N/A
56.3
2.5
73.2
14.5
RLHF ( Christiano et al., 2017 )
79.4
65.0
N/A
48.3
2.3
75.6
5.6
Table 1: Evaluation results of MAPL and baseline methods on TLDR , CoopHE , BFCL , and Travel across 5 runs: Score measures the summary quality; Pass denotes the pass rate; Call denotes the invocation rate of aux( ⋅ ) ; Correct is the functional correctness rate; Costs measures the number of calls per agent; Pass is the commonsense pass rate; Success is the final pass rate with more constraints. Underline and bold indicate the best results across MAPL and all methods. † and ‡ denote pivots for the ablation studies in Table 2 .
Method
TLDR
CoopHE
BFCL
Travel
Score
Pass
Correct
Pass
(a) Replay Buffer
MARLHF ( λ -Decay) †
86.3
61.0
22.7
70.5
MARLHF (Offline)
70.9 ( ↓ 15.4)
46.4 ( ↓ 14.6)
7.5 ( ↓ 15.2)
63.6 ( ↓ 6.9)
MARLHF (Online)
76.0 ( ↓ 10.3)
59.7 ( ↓ 1.3)
14.8 ( ↓ 7.9)
69.8 ( ↓ 0.7)
MADPO ( λ -Decay) ‡
92.9
58.8
15.6
41.5
Table 2: Ablation results on (a) replay buffers, (b) comparators, and (c) MARLHF methods under Dec-Dec. Pivot entries † and ‡ reported in Table 1 use the λ -decay buffer, agent-size comparator, and MAGRPO. Parenthesized values show absolute changes relative to the corresponding pivot, where ↑ denotes an increase and ↓ denotes a decrease. Best results within each ablation are reported in bold , best results for each method within each ablation are underlined .
Figure 3: Evaluation results (a-d) and data distribution shift (e-h) of MARLHF and MADPO on offline ( brown and green ), online ( orange and purple ), λ -decay ( red and blue ) replay buffers. Curves are smoothed by an exponential moving average, and shaded regions indicate the standard deviation. Triangles mark the start of iterations. Data distributions are fitted with a Gaussian. Curves with higher opacity indicate later iterations.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 MARLHF under Dec-Dec
Algorithm 2 MADPO under Dec-Dec
Figure 4: A Venn diagram of different comparator policies.
Configuration
TLDR
CoopHE
BFCL
Travel
Task
Dataset
trl-lib/tldr
OpenMLRL/CoopHumanEval
BFCL
osunlp/TravelPlanner
Filter
None
None
parallel & parallel_multiple
days ⩽ 5 & visiting_city_num ⩽ 2
Training
train [0,250)
train [16:82)
data [0:160)
validation [0:60)
Evaluation
eval [0, 250)
train [0:16)
data [160:200)
validation [60:82)
Model
Appendix
Table 3: Task and model configurations. Unless otherwise specified, the agents, comparators, and reward model in MARLHF share the same model configuration. The filtered subsets preserve the original order.
Hyperparameter
TLDR
CoopHE
BFCL
Travel
Comparator
Generation mode (Dec)
decentralized
decentralized
decentralized
decentralized
Generation mode (Cen)
centralized
centralized
centralized
centralized
Model
current
current
current_copy
current_copy
Preference dataset pairs
20
20
20
20
Replay Buffer
Appendix
Table 4: Comparator, replay buffer, and optimization hyperparameters.
Figure 5: Ablation on different sizes of D for offline MARLHF and MADPO ( brown and green ) across 5 runs.
Figure 6: Ablation on the different replay buffers for MARLHF ( red ) and MADPO ( blue ) across 5 runs. We consider the decentralized MAS. † and ‡ denote the pivot settings for MARLHF and MADPO, respectively.
Figure 7: Results of MARLHF and MADPO on TLDR , CoopHE , BFCL , and Travel across 5 runs.
Figure 8: Ablation on the different comparators used in MARLHF and MADPO across 5 runs, ϖdec in red and blue , ϖcen in blue-green and orange , respectively. We consider decentralized MAS. † denotes the pivot settings.
Figure 9: Results of MARLHF and MADPO on TLDR, CoopHE, BFCL, and Travel across 5 runs.
Figure 10: Ablation on different MAS configurations. Green curves present the performance before training, red and blue show results via MARLHF and MADPO training. Hyperparameters are the same as in Table 4 .
Method
BFCL
Travel
Two-Agent †
Three-Agent
Two-Agent †
Three-Agent
Raw MAS
12.8
17.5
66.2
63.1
MARLHF
22.7
18.4
70.5
72.6
MADPO
15.6
15.1
41.5
44.3
Appendix
Table 5: Performance of two-agent and three-agent MAS on BFCL and Travel. We report Correct for BFCL and Pass for Travel. Raw MAS denotes parallel inference with raw LLMs.
Method
GPUs
Duration
Model Placement
Peak GPU VRAM
Agents
Reward
Comparator
cuda:0
cuda:1
cuda:2
Total
TLDR
MAGRPO (Dec)
1 × RTX
12 h
cuda:0
N/A
N/A
40
N/A
N/A
40
MAGRPO (Cen)
1 × B200
10 h
cuda:0
N/A
N/A
50
N/A
N/A
50
MARLHF (Offline)
1 × B200
12 h
cuda:0
cuda:0
N/A
55
N/A
N/A
55
MARLHF (Dec-Dec)
1 × RTX
15 h
cuda:0
cuda:0
cuda:0
75
N/A
N/A
75
Appendix
Table 6: Experimental devices and the peak GPU memory usage (GB) during training. RTX and H100 represent the RTX 6000 Pro Blackwell and H100 PCIe editions, respectively. The VRAM on each device is rounded to the nearest 5 GB, total VRAM sums the rounded values.