MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning

Tianmeng Hu; Biao Luo; Chunhua Yang; Tingwen Huang

cs.AIFeb 28, 2026

MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning

Authors: Tianmeng Hu, Biao Luo, Chunhua Yang, Tingwen Huang

Organizations: School of Automation, Central South University, Changsha, 410012, China. · Texas A&M University at Qatar, Doha, 23874, Qatar.

Abstract

Deep reinforcement learning (RL) has been applied extensively to solve complex decision-making problems. In many real-world scenarios, tasks often have several conflicting objectives and may require multiple agents to cooperate, which are the multi-objective multi-agent decision-making problems. However, only few works have been conducted on this intersection. Existing approaches are limited to separate fields and can only handle multi-agent decision-making with a single objective, or multi-objective decision-making with a single agent. In this paper, we propose MO-MIX to solve the multi-objective multi-agent reinforcement learning (MOMARL) problem. Our approach is based on the centralized training with decentralized execution (CTDE) framework. A weight vector representing preference over the objectives is fed into the decentralized agent network as a condition for local action-value function estimation, while a mixing network with parallel architecture is used to estimate the joint action-value function. In addition, an exploration guide approach is applied to improve the uniformity of the final non-dominated solutions. Experiments demonstrate that the proposed method can effectively solve the multi-objective multi-agent cooperative decision-making problem and generate an approximation of the Pareto set. Our approach not only significantly outperforms the baseline method in all four kinds of evaluation metrics, but also requires less computational cost.

Figures & tables

Fig. 1: The architecture of the CAN. Each agent uses its respective CAN to estimate the partial multi-objective Q-function and to select the action. The inputs to a CAN are the partial observation of agent $i$ , the action of the previous time step and the current preference $\boldsymbol{w}$ . That is, we use the preference $\boldsymbol{w}$ as a condition for estimating the value function. A GRU layer is used to take advantage of the partial observation history. The output of the CAN is multi-objective Q-vectors for each optional action, each vector containing Q-values for $m$ objectives. The agents select actions independently based on the $\epsilon$ -greedy policy.

Fig. 2: The architecture of the MOMN. The MOMN takes outputs of all CANs and reorganizes them according to different objectives. The MOMN is internally divided into $m$ parallel tracks corresponding to $m$ objectives. Each track has two MLP layers. Multiple track outputs are then concatenated as the outputs of the entire network, which are the multi-objective joint Q-values. The global state $s_{t}$ at time step $t$ is fed to several hypernetworks and used to generate weights and biases for the mixing network.

Fig. 3: The connection between hypernetworks and one MOMN track. For each neural network layer of each MOMN track, two hypernetworks are used to generate its parameters. One is used for the weights and another for the biases.

Fig. 4: The average utility curve of 75000 episodes.The data are based on five independent runs of the MO-MIX algorithm and local means were estimated using a sliding average algorithm with a parameter of 0.85. The light-colored part shows the standard deviation.

Fig. 5: The hypervolume variation curve. Data are based on five independent runs of the MO-MIX algorithm, tested once every 5000 episodes. The light-colored part shows the standard deviation.

Fig. 6: The final non-dominated set of the two methods. The left figure shows the results of five independent runs of the MO-MIX algorithm. Different runs are distinguished by different colors. The right figure shows the results of the outer-loop QMIX method after 41 rounds of training. All dominated solutions have been removed.

Fig. 7: The movement of the non-dominated set during the entire training process. The figure shows the non-dominated set for episodes 5000, 10000, 20000, 40000, and 75000 respectively.

Metrics	MO-MIX	Outer-loop QMIX (baseline v1)	Outer-loop QMIX (baseline v2)
Hypervolume $\uparrow$	11080.9224	9448.3380	9791.7632
Diversity $\uparrow$	40.40	17.00	21.00
Spacing $\downarrow$	4.2273	15.8387	12.2401
Sparsity $\downarrow$	43.6463	467.3278	236.0842

TABLE I: Comparison of Different Algorithms on Four Evaluation Metrics

Fig. 8: Results of the MO-MIX algorithm using different preference sampling intervals. The subgraphs show the variation curves of hypervolume, diversity, spacing and sparsity metrics, respectively. We plot the average performance over five experiments. The light-colored part shows the standard deviation.

Fig. 9: Results of the complete MO-MIX algorithm and an ablated version of MO-MIX in which the exploration guide part is removed. We plot the average performance over five experiments. The light-colored part shows the standard deviation.

Metrics	MO-MIX	Outer-loop QMIX
Hypervolume $\uparrow$	7371.7852	6226.9744
Diversity $\uparrow$	18.40	10.00
Spacing $\downarrow$	7.8697	33.5992
Sparsity $\downarrow$	203.7040	1509.6655

TABLE II: Comparison of Different Algorithms on Four Evaluation Metrics with the SMAC environment

Fig. 10: The final non-dominated set of the two methods, based on the SMAC-2s3z environment. The left figure shows the results of five independent runs of the MO-MIX algorithm. Different runs are distinguished by different colors. The right figure shows the results of the outer-loop QMIX method after 41 rounds of training. All dominated solutions have been removed.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning
Oct 26, 2026Qiyue Xia, J. Michael HerrmannMulti-Objective Reinforcement Learning Multi-Agent Reinforcement Learning
School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning
Jun 12, 2026Pengxin Wang, Lihao Guo, Yi Xie +3Multi-Objective Reinforcement Learning Multi-Agent System Optimization
Department of Electrical and Computer Engineering, University of Arizona
A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning
Apr 27, 2026Ying-Tu Chen, Wei Hung, Bing-Shu Wu +2Multi-Objective Reinforcement Learning Reinforcement Learning
National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Massachusetts Institute of Technology

Oct 26, 2026J/K move · Enter open · S save

Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning

Qiyue Xia, J. Michael Herrmann

School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

Multi-Objective Reinforcement Learning Multi-Agent Reinforcement Learning Cooperative MARL

Open paper AskPDF arXiv

Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning

Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning

A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning