Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.
Figures & tables
Figure 1. Team-level and individual-level preference modelling in MOMARL. An externally specified team preference defines the desired system-level trade-off, while internally developed individual preferences give agents flexibility in how they contribute to the cooperative task. This figure illustrates the main features of the proposed approach including a symbolic visualisation of a possible set of objectives and the representation of the agent-related preferences in the full MOMARL setting.
Figure 2. System overview for MIRL -S and MIRL -D. This figure highlights the differences of \algoname{}-S and \algoname{}-D as also described in Sect.\ref{subsec:static_meta} and Sect.\ref{subsec:dynamic_meta}.
Figure 3. Illustration of training update in MIRL -S. This figure shows an illustration of training update in~\algoname{}-S which is also described in Sect.~\ref{subsec:static_meta}
Environment
Metric
Method
DTDE-PPO
CTDE-PPO-V
CTDE-PPO-Q
MO-COMA
MO-MIX
MIRL -S
MIRL -D
Spread
HV
1222.55 ± 15.43
1434.46 ± 52.34
1512.17 ± 19.41
1570.08 ± 8.61
1213.48 ± 131.15
1581.40 ± 30.05
1594.55 ± 20.34
EU
36.57 ± 0.24
38.16 ± 0.54
39.32 ± 0.12
39.85 ± 0.08
36.44 ± 2.26
40.02 ± 0.32
40.14 ± 0.25
SP
60.58 ± 76.77
14.03 ± 3.08
2.86 ± 2.32
1.20 ± 0.82
0.07 ± 0.07
1.39 ± 1.06
12.08 ± 19.77
Formation
HV
1359.53 ± 10.20
1815.27 ± 40.36
1975.36 ± 40.19
2178.79 ± 23.31
1525.67 ± 333.53
2187.33 ± 7.91
2196.13 ± 15.76
EU
37.79 ± 0.25
42.59 ± 0.34
44.02 ± 0.31
45.76 ± 0.20
39.21 ± 4.77
45.85 ± 0.19
46.03 ± 0.25
Table 1. Evaluation of the Pareto front quality.
Figure 4. Pareto front comparison of the MIRL variants with the baselines across four benchmark problems. Pareto front comparison shows that the proposed algorithm variant MIRL-S is competitive in comparison with the listed algorithms on four different benchmark problems, while the dynamic variant MIRL-D has a small advantage as also clear from the information in Tabl.~\ref{tab:eval_results}.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
DTDE-PPO
CTDE-PPO-V
CTDE-PPO-Q
Actor learning rate
3×10−4
3×10−4
3×10−4
Critic learning rate
3×10−4
3×10−4
3×10−4
PPO clipping coefficient
0.2
0.2
0.2
Entropy coefficient
0.01
0.01
0.01
Optimisation epochs
4
4
4
Minibatch size
256
256
256
Appendix
Table 2. Hyperparameters used by the PPO-based baselines.
Hyperparameter
MO-COMA
MIRL-S
MIRL-D
Actor learning rate
5×10−4
5×10−4
5×10−4
Critic learning rate
5×10−4
5×10−4
5×10−4
Episode batch size
8
8
8
λ
0.8
0.8
0.8
Actor recurrent hidden size
64
64
64
Critic hidden size
128
128
128
Appendix
Table 3. Hyperparameters used by MO-COMA , MIRL-S , and MIRL-D .
Hyperparameter
Value
Learning rate
5×10−6
Episode batch size
64
Replay capacity
12000 episodes
Agent recurrent hidden size
64
Mixing hidden size
32
Training preference interval
0.05
Appendix
Table 4. Hyperparameters used by MO-MIX .
Figure 5. Individual task preferences learned by MIRL -S during training on Line .
Figure 6. Individual task preferences learned by MIRL -S and MIRL -D on Spread , with ωteam,task=0.35 .
Figure 7. Individual task preferences generated by MIRL -D during two example evaluation episodes on Spread under the fixed team preference ωteam,task=0.35 .
Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal equilibrium problem, and show that, under suitable conditions, preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.
Pengxin Wang, Lihao Guo, Yi Xie +3
Department of Electrical and Computer Engineering, University of Arizona
Deep reinforcement learning (RL) has been applied extensively to solve complex decision-making problems. In many real-world scenarios, tasks often have several conflicting objectives and may require multiple agents to cooperate, which are the multi-objective multi-agent decision-making problems. However, only few works have been conducted on this intersection. Existing approaches are limited to separate fields and can only handle multi-agent decision-making with a single objective, or multi-objective decision-making with a single agent. In this paper, we propose MO-MIX to solve the multi-objective multi-agent reinforcement learning (MOMARL) problem. Our approach is based on the centralized training with decentralized execution (CTDE) framework. A weight vector representing preference over the objectives is fed into the decentralized agent network as a condition for local action-value function estimation, while a mixing network with parallel architecture is used to estimate the joint action-value function. In addition, an exploration guide approach is applied to improve the uniformity of the final non-dominated solutions. Experiments demonstrate that the proposed method can effectively solve the multi-objective multi-agent cooperative decision-making problem and generate an approximation of the Pareto set. Our approach not only significantly outperforms the baseline method in all four kinds of evaluation metrics, but also requires less computational cost.
Tianmeng Hu, Biao Luo, Chunhua Yang +1
School of Automation, Central South University, Changsha, 410012, China. · Texas A&M University at Qatar, Doha, 23874, Qatar.
Constrained Multi-agent reinforcement learning (CMARL) faces two intertwined challenges: the joint action space grows exponentially with the number of agents, and additional requirements couple agents in ways that reward structure alone does not capture. We introduce Coordination Graphs for Constrained Multi-Agent Reinforcement Learning (CG-CMARL), a framework that addresses both challenges by combining coordination graphs with Lagrangian duality. The system decomposes the joint problem into pairwise regions, each served by a set of shared Q-functions, one for the primary objective and one for each of the constraints, so that the number of learned models is independent of the number of agents. At execution time, Max-Sum message passing coordinates actions across the factor graph, while a Lagrangian multiplier controls the objective--constraint tradeoff, allowing a single trained model to trace a Pareto front without retraining. We provide convergence guarantees under mild conditions, together with a compositional error bound that decomposes into separate interpretable sources, each traceable to a specific design choice and independently controllable. Experiments on cooperative navigation tasks (where teams of up to 10 agents must coordinate to reach target positions while satisfying pairwise constraints) show that our method produces Pareto fronts dominating established baselines trained at fixed reward-shaping ratios, while scaling to team sizes where centralized approaches become intractable.
Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders Jonsson
Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain