Deep reinforcement learning (RL) has been applied extensively to solve complex decision-making problems. In many real-world scenarios, tasks often have several conflicting objectives and may require multiple agents to cooperate, which are the multi-objective multi-agent decision-making problems. However, only few works have been conducted on this intersection. Existing approaches are limited to separate fields and can only handle multi-agent decision-making with a single objective, or multi-objective decision-making with a single agent. In this paper, we propose MO-MIX to solve the multi-objective multi-agent reinforcement learning (MOMARL) problem. Our approach is based on the centralized training with decentralized execution (CTDE) framework. A weight vector representing preference over the objectives is fed into the decentralized agent network as a condition for local action-value function estimation, while a mixing network with parallel architecture is used to estimate the joint action-value function. In addition, an exploration guide approach is applied to improve the uniformity of the final non-dominated solutions. Experiments demonstrate that the proposed method can effectively solve the multi-objective multi-agent cooperative decision-making problem and generate an approximation of the Pareto set. Our approach not only significantly outperforms the baseline method in all four kinds of evaluation metrics, but also requires less computational cost.
Figures & tables
Fig. 1: The architecture of the CAN. Each agent uses its respective CAN to estimate the partial multi-objective Q-function and to select the action. The inputs to a CAN are the partial observation of agent i , the action of the previous time step and the current preference w . That is, we use the preference w as a condition for estimating the value function. A GRU layer is used to take advantage of the partial observation history. The output of the CAN is multi-objective Q-vectors for each optional action, each vector containing Q-values for m objectives. The agents select actions independently based on the ϵ -greedy policy.
Fig. 2: The architecture of the MOMN. The MOMN takes outputs of all CANs and reorganizes them according to different objectives. The MOMN is internally divided into m parallel tracks corresponding to m objectives. Each track has two MLP layers. Multiple track outputs are then concatenated as the outputs of the entire network, which are the multi-objective joint Q-values. The global state st at time step t is fed to several hypernetworks and used to generate weights and biases for the mixing network.
Fig. 3: The connection between hypernetworks and one MOMN track. For each neural network layer of each MOMN track, two hypernetworks are used to generate its parameters. One is used for the weights and another for the biases.
Fig. 4: The average utility curve of 75000 episodes.The data are based on five independent runs of the MO-MIX algorithm and local means were estimated using a sliding average algorithm with a parameter of 0.85. The light-colored part shows the standard deviation.
Fig. 5: The hypervolume variation curve. Data are based on five independent runs of the MO-MIX algorithm, tested once every 5000 episodes. The light-colored part shows the standard deviation.
Fig. 6: The final non-dominated set of the two methods. The left figure shows the results of five independent runs of the MO-MIX algorithm. Different runs are distinguished by different colors. The right figure shows the results of the outer-loop QMIX method after 41 rounds of training. All dominated solutions have been removed.
Fig. 7: The movement of the non-dominated set during the entire training process. The figure shows the non-dominated set for episodes 5000, 10000, 20000, 40000, and 75000 respectively.
Metrics
MO-MIX
Outer-loop QMIX (baseline v1)
Outer-loop QMIX (baseline v2)
Hypervolume ↑
11080.9224
9448.3380
9791.7632
Diversity ↑
40.40
17.00
21.00
Spacing ↓
4.2273
15.8387
12.2401
Sparsity ↓
43.6463
467.3278
236.0842
TABLE I: Comparison of Different Algorithms on Four Evaluation Metrics
Fig. 8: Results of the MO-MIX algorithm using different preference sampling intervals. The subgraphs show the variation curves of hypervolume, diversity, spacing and sparsity metrics, respectively. We plot the average performance over five experiments. The light-colored part shows the standard deviation.
Fig. 9: Results of the complete MO-MIX algorithm and an ablated version of MO-MIX in which the exploration guide part is removed. We plot the average performance over five experiments. The light-colored part shows the standard deviation.
Metrics
MO-MIX
Outer-loop QMIX
Hypervolume ↑
7371.7852
6226.9744
Diversity ↑
18.40
10.00
Spacing ↓
7.8697
33.5992
Sparsity ↓
203.7040
1509.6655
TABLE II: Comparison of Different Algorithms on Four Evaluation Metrics with the SMAC environment
Fig. 10: The final non-dominated set of the two methods, based on the SMAC-2s3z environment. The left figure shows the results of five independent runs of the MO-MIX algorithm. Different runs are distinguished by different colors. The right figure shows the results of the outer-loop QMIX method after 41 rounds of training. All dominated solutions have been removed.
Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.
Qiyue Xia, J. Michael Herrmann
School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal equilibrium problem, and show that, under suitable conditions, preference diversity yields a first-order improvement in the team objective. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.
Pengxin Wang, Lihao Guo, Yi Xie +3
Department of Electrical and Computer Engineering, University of Arizona
Many sequential decision-making tasks involve optimizing multiple conflicting objectives, requiring policies that adapt to different user preferences. In multi-objective reinforcement learning (MORL), one widely studied approach} addresses this by training a single policy network conditioned on preference-weighted rewards. In this paper, we explore a novel algorithmic perspective: leveraging reward-free reinforcement learning (RFRL) for MORL. While RFRL has historically been studied independently of MORL, it learns optimal policies for any possible reward function, making it a natural fit for MORL's challenge of handling unknown user preferences. We propose using the RFRL's training objective as an auxiliary task to enhance MORL, enabling more effective knowledge sharing beyond the multi-objective reward function given at training time. To this end, we adapt a state-of-the-art RFRL algorithm to the MORL setting and introduce a preference-guided exploration strategy that focuses learning on relevant parts of the environment. Through extensive experiments and ablation studies, we demonstrate that our approach significantly outperforms the state-of-the-art MORL methods across diverse MO-Gymnasium tasks, achieving superior performance and data efficiency. This work provides the first systematic adaptation of RFRL to MORL, demonstrating its potential as a scalable and empirically effective solution to multi-objective policy learning.
Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology