cs.MAOct 26, 2026

Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning

Authors: Qiyue Xia, J. Michael Herrmann

Organizations: School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.

Figures & tables

Figure 1. Team-level and individual-level preference modelling in MOMARL. An externally specified team preference defines the desired system-level trade-off, while internally developed individual preferences give agents flexibility in how they contribute to the cooperative task. This figure illustrates the main features of the proposed approach including a symbolic visualisation of a possible set of objectives and the representation of the agent-related preferences in the full MOMARL setting.
Figure 2. System overview for MIRL -S and MIRL -D. This figure highlights the differences of \algoname{}-S and \algoname{}-D as also described in Sect.\ref{subsec:static_meta} and Sect.\ref{subsec:dynamic_meta}.
Figure 3. Illustration of training update in MIRL -S. This figure shows an illustration of training update in~\algoname{}-S which is also described in Sect.~\ref{subsec:static_meta}
EnvironmentMetricMethod
DTDE-PPOCTDE-PPO-VCTDE-PPO-QMO-COMAMO-MIXMIRL -SMIRL -D
SpreadHV1222.55 ±\pm 15.431434.46 ±\pm 52.341512.17 ±\pm 19.411570.08 ±\pm 8.611213.48 ±\pm 131.151581.40 ±\pm 30.051594.55 ±\pm 20.34
EU36.57 ±\pm 0.2438.16 ±\pm 0.5439.32 ±\pm 0.1239.85 ±\pm 0.0836.44 ±\pm 2.2640.02 ±\pm 0.3240.14 ±\pm 0.25
SP60.58 ±\pm 76.7714.03 ±\pm 3.082.86 ±\pm 2.321.20 ±\pm 0.820.07 ±\pm 0.071.39 ±\pm 1.0612.08 ±\pm 19.77
FormationHV1359.53 ±\pm 10.201815.27 ±\pm 40.361975.36 ±\pm 40.192178.79 ±\pm 23.311525.67 ±\pm 333.532187.33 ±\pm 7.912196.13 ±\pm 15.76
EU37.79 ±\pm 0.2542.59 ±\pm 0.3444.02 ±\pm 0.3145.76 ±\pm 0.2039.21 ±\pm 4.7745.85 ±\pm 0.1946.03 ±\pm 0.25
Table 1. Evaluation of the Pareto front quality.
Figure 4. Pareto front comparison of the MIRL variants with the baselines across four benchmark problems. Pareto front comparison shows that the proposed algorithm variant MIRL-S is competitive in comparison with the listed algorithms on four different benchmark problems, while the dynamic variant MIRL-D has a small advantage as also clear from the information in Tabl.~\ref{tab:eval_results}.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterDTDE-PPOCTDE-PPO-VCTDE-PPO-Q
Actor learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}3×10−43\times 10^{-4}
Critic learning rate3×10−43\times 10^{-4}3×10−43\times 10^{-4}3×10−43\times 10^{-4}
PPO clipping coefficient0.20.20.20.20.20.2
Entropy coefficient0.010.010.010.010.010.01
Optimisation epochs444444
Minibatch size256256256256256256
Appendix
Table 2. Hyperparameters used by the PPO-based baselines.
HyperparameterMO-COMAMIRL-SMIRL-D
Actor learning rate5×10−45\times 10^{-4}5×10−45\times 10^{-4}5×10−45\times 10^{-4}
Critic learning rate5×10−45\times 10^{-4}5×10−45\times 10^{-4}5×10−45\times 10^{-4}
Episode batch size888888
λ\lambda0.80.80.80.80.80.8
Actor recurrent hidden size646464646464
Critic hidden size128128128128128128
Appendix
Table 3. Hyperparameters used by MO-COMA , MIRL-S , and MIRL-D .
HyperparameterValue
Learning rate5×10−65\times 10^{-6}
Episode batch size6464
Replay capacity1200012000 episodes
Agent recurrent hidden size6464
Mixing hidden size3232
Training preference interval0.050.05
Appendix
Table 4. Hyperparameters used by MO-MIX .
Figure 5. Individual task preferences learned by MIRL -S during training on Line .
Figure 6. Individual task preferences learned by MIRL -S and MIRL -D on Spread , with ωteam,task=0.35\omega_{\mathrm{team,task}}=0.35 .
Figure 7. Individual task preferences generated by MIRL -D during two example evaluation episodes on Spread under the fixed team preference ωteam,task=0.35\omega_{\mathrm{team,task}}=0.35 .
More in this topic

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 144 papers

Explore similar work

CardsList

  1. Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning

    Jun 12, 2026Pengxin Wang, Lihao Guo, Yi Xie +3Multi-Objective Reinforcement LearningMulti-Agent System Optimization

    Department of Electrical and Computer Engineering, University of Arizona

  2. MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning

    Feb 28, 2026Tianmeng Hu, Biao Luo, Chunhua Yang +1Multi-Agent Reinforcement LearningMulti-Objective Reinforcement Learning

    School of Automation, Central South University, Changsha, 410012, China. · Texas A&M University at Qatar, Doha, 23874, Qatar.

  3. Coordination Graphs for Constrained Multi-Agent Reinforcement Learning

    Jun 1, 2026Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders JonssonMulti-Objective Reinforcement LearningMulti-Agent System Optimization

    Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain