The price of decentralization in managing engineering systems through multi-agent reinforcement learning

Prateek Bhustali; Pablo G. Morato; Konstantinos G. Papakonstantinou; Charalampos P. Andriotis

cs.MAMar 12, 2026

The price of decentralization in managing engineering systems through multi-agent reinforcement learning

Authors: Prateek Bhustali, Pablo G. Morato, Konstantinos G. Papakonstantinou, Charalampos P. Andriotis

Organizations: Faculty of Architecture and the Built Environment, Delft University of Technology, Delft, 2628 BL, The Netherlands · Engineering Risk Analysis Group, Technical University of Munich, Munich, 80333, Germany · Department of Civil & Environmental Engineering, The Pennsylvania State University, University Park, PA, 16802, USA

Abstract

Inspection and maintenance (I&M) planning involves sequential decision making under uncertainties and incomplete information, and can be modeled as a partially observable Markov decision process (POMDP). While single-agent deep reinforcement learning provides approximate solutions to POMDPs, it does not scale well in multi-component systems. Scalability can be achieved through multi-agent deep reinforcement learning (MADRL), which decentralizes decision-making across multiple agents, locally controlling individual components. However, this decentralization can induce cooperation pathologies that degrade the optimality of the learned policies. To examine these effects in I&M planning, we introduce a set of deteriorating systems in which redundancy is varied systematically. These benchmark environments are designed such that computation of centralized (near-)optimal policies remains tractable, enabling direct comparison of solution methods. We implement and benchmark a broad set of MADRL algorithms spanning fully centralized and decentralized training paradigms, from value-factorization to actor-critic methods. Our results show a clear effect of redundancy on coordination: MADRL algorithms achieve near-optimal performance in series-like settings, whereas increasing redundancy amplifies coordination challenges and can lead to optimality losses. Nonetheless, decentralized agents learn structured policies that consistently outperform optimized heuristic baselines, highlighting both the promise and current limitations of decentralized learning for scalable maintenance planning.

Figures & tables

Figure 1: In multi-component systems, joint state, action, and observation spaces grow exponentially with the number of components, rendering single-agent approaches intractable. Dec-POMDPs address this curse of dimensionality by decentralizing control and are commonly solved using multi-agent DRL methods. Left : Scalability of solution approaches. The curved boundaries illustrate the approximate, relative scalability of POMDP and Dec-POMDP solution approaches as these spaces increase. The ( ) indicates the approximate size of the systems studied in this work, where single-agent methods remain tractable and allow direct comparison with multi-agent approaches. Right : Agent-centric paradigms. Single-agent approaches are intrinsically centralized because a single agent possesses all information during training and execution. In contrast, multi-agent approaches, such as CTDE, relax decentralization during training by allowing agents to share information, but agents must learn to execute actions independently at execution/inference.

Figure 2: A Dec-POMDP unrolled as a dynamic decision network over three time steps. At each time step

t

, the environment state

s^{t}

evolves according to the transition model

s^{t+1}\sim T(\cdot\mid s^{t},\mathbf{a}^{t})

under the joint action

\mathbf{a}^{t}

, yielding a global reward

r^{t}

and individual observations

\mathbf{o}^{t+1}\sim O(\cdot\mid s^{t+1},\mathbf{a}^{t})

for each agent

m\in\mathcal{M}

. Each agent selects its action based on its local action–observation history

h_{m}^{t}

Figure 3: An overview of MADRL algorithms for managing a (four-component) reliability system. Algorithms are classified by training–execution paradigms: centralized training centralized execution (CTCE), centralized training decentralized execution (CTDE), and decentralized training decentralized execution (DTDE), reflecting increasing decentralization and information constraints. Bottom labels indicate the underlying formulation framework (POMDP, MPOMDP, Dec-POMDP). Suffix PS denotes parameter sharing across agents.

Paradigm	Formulation	Algorithm			Actor (policy-based only)		Critic / Value Function
	Framework	Name	Type	On/Off-policy	Input	Output	Input	Output
CTCE	POMDP (SADRL)	DDQN	V	Off-policy	-	-	Centralized	Centralized
	POMDP (SADRL)	JAC	A+C	Off-policy	Centralized	Centralized	Centralized	Centralized
	MPOMDP	DCMAC	A+C	Off-policy	Centralized	Local	Centralized	Centralized
CTDE	Dec-POMDP	IACC-PS	A+C	Off-policy	Local	Local	Centralized	Centralized
		MAPPO-PS	A+C	On-policy	Local	Local	Centralized	Centralized

Table 1: Summary of single- and multi-agent DRL algorithms shown in Figure 3 . The Actor columns describe the policy, when present, with Input indicating the information available to the policy and Output indicating whether actions are produced jointly,

\mathbf{a}

, or per agent,

a_{m}

. The Critic / Value Function columns analogously report the information available to the value estimator and whether it outputs a single centralized value (e.g.,

Q(\mathbf{h},\mathbf{a})

V(\mathbf{h})

) or per-agent values (e.g.,

Q_{m}(h_{m},a_{m})

). Centralized denotes access to global state or joint observations and actions, whereas Local denotes access only to an agent’s own observation or action–observation history.

Figure 4: Left: Payoff matrix for the Climb Game with two agents, each choosing from three actions. The joint action

(\mathfrak{a}^{1},\mathfrak{a}^{1})

yields the optimal reward of 11 [ Claus and Boutilier, 1998 ] . Right: Best performance of MADRL algorithms aggregated over 30 training instances in the Climb Game played repeatedly for 25 time steps [ Papoudakis et al., 2021 ] . The optimal return is

11\times 25=275

Figure 5: Box plots summarize algorithm performance across

k

-out-of-4 systems, aggregated over 10 training instances. The results are normalized using the expected cost of the (near-)optimal SARSOP policy (Table 2 ), allowing comparison across varying redundancy levels. Most decentralized algorithms perform near-optimally in series configurations

(k=n)

, but deviate from the (near-)optimal SARSOP baseline as redundancy increases, with the largest performance drop observed in the parallel configurations

(k=1)

$k$ -out-of-4 systems		1-out-of-4 system	2-out-of-4 system	3-out-of-4 system	4-out-of-4 system
SARSOP	(LBound, UBound)	(43.67, 51.82)	(113.19, 141.76)	(272.84, 309.06)	(727.36, 755.52)
SARSOP	Expected return	50.41 (50.31, 50.51)	138.59 (138.25, 138.93)	303.93 (303.18, 304.68)	747.54 (745.92, 749.15)
Heuristic	Expected return	70.19 (70.12, 70.25)	159.32 (159.01, 159.63)	358.51 (357.44, 359.57)	913.03 (911.36, 914.70)
CTCE	DDQN	50.81 (50.71, 50.90)	141.70 (141.48, 141.92)	312.61 (311.86, 313.36)	750.20 (748.62, 751.77)
	JAC	50.89 (50.81, 50.98)	143.66 (143.38, 143.94)	311.95 (311.20, 312.70)	749.90 (748.34, 751.46)
	DCMAC	53.83 (53.75, 53.92)	144.45 (144.22, 144.69)	311.37 (310.57, 312.17)	749.57 (747.99, 751.16)

Table 2: Best performance of single and multi-agent DRL algorithms across

k

-out-of-4 systems over 10 training instances. Reported values are expected costs, computed as discounted costs via Monte Carlo rollouts, with 95% confidence intervals in brackets. SARSOP, using one-step look-ahead, serves as the near-optimal baseline. SARSOP expected costs shown in bold are also used to normalize results in Figure 5 . Entries marked with (

\ddagger

) correspond to deterministic periodic repair policies and exhibit negligible variance.

Figure 6: Belief space plots. Using the best-performing training instance of each algorithm, we visualize the learned policies over the belief space. Specifically, we show the actions taken over 100 Monte Carlo rollouts of 20 time steps in the 1-out-of-4 system. Each rollout begins from the initial belief $b_{0}=(0.6,0.4,0)$ indicated with a blue dot ( • ). The SARSOP baseline, JAC, and DDQN coordinate repairs across components 1 and 2 and use inspections to keep beliefs updated; they let components 3 and 4 drift toward failure to optimally exploit redundancy. Decentralized algorithms, such as DCMAC, IACC-PS, MAPPO-PS, IPPO-PS, learn a structured coordination of components 1 and 2 (e.g., periodic repairs to eliminate non-stationarity), yet these policies remain suboptimal as detailed in Table 2 .

Figure 7: A sample rollout of the best IACC-PS policy in the 1-out-of-4 system for 40 time steps. Interestingly, agent 1 chooses a repair action every 5 and 7 time steps, and agent 2 chooses repairs every 5 time steps, even though neither observes the time step, and both are purely based on their individual components’ beliefs.

Climb Game Agent 1

\mathfrak{a}^{1}

\mathfrak{a}^{2}

\mathfrak{a}^{3}

Agent 2

\mathfrak{a}^{1}

11^{*}

-30 0

\mathfrak{a}^{2}

-30 7 6

\mathfrak{a}^{3}

0 0 5

VDN-PS Decomposition

Q_{1}(\mathfrak{a}^{1})

Q_{1}(\mathfrak{a}^{2})

Q_{1}(\mathfrak{a}^{3})

-4.18 -7.35 6.31

Q_{2}(\mathfrak{a}^{1})

-4.76 -8.95 -12.11 1.55

Q_{2}(\mathfrak{a}^{2})

-0.33 -4.51 -7.67 5.99

Q_{2}(\mathfrak{a}^{3})

-0.77 -4.95 -8.11 5.55

QMIX-PS Decomposition

Q_{1}(\mathfrak{a}^{1})

Q_{1}(\mathfrak{a}^{2})

Q_{1}(\mathfrak{a}^{3})

-0.13 -5153.65 -2.88

Q_{2}(\mathfrak{a}^{1})

1.19 11.10 -9.54 1.65

Q_{2}(\mathfrak{a}^{2})

-5412.35 -9.54 -9.54 -9.54

Q_{2}(\mathfrak{a}^{3})

-1.66 1.73 -9.54 1.59

Figure 8: Value decompositions for the Climb Game. Left: Climb Game rewards. Middle and right: value decompositions learned by the best-performing instances of VDN-PS and QMIX-PS, respectively. Tables show estimated joint action-values, with rows and columns corresponding to the agents’ local actions; orange entries denote the greedy joint action. While VDN-PS assigns the correct maximum to the optimal joint action

(a_{3},a_{2})

, its linear factorization misrepresents other action utilities. QMIX-PS, despite its monotonic mixing constraint, also misrepresents joint utilities, with optimal performance occurring only in outlier cases.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Sub-optimal JAC policy in the 1-out-of-4 system. While the optimal JAC and SARSOP policies intervene on components 1 and 2 to maintain system functionality, this learned policy only involves agent 2 taking intervention actions, and all other agents always choosing do-nothing.

Figure 10: Learning curves of CTCE, CTDE, and DTDE algorithms on the Climb Game. Performance is evaluated periodically during training and shown as median cost across seeds, with shaded regions indicating the interquartile range. In this one-step environment, the initial episode is excluded from best-checkpoint selection, since random initializations can yield the optimal action before any learning occurs.

1-out-of-4 system (parallel configuration)
		Training instances
Paradigm	Algorithm	1	2	3	4	5	6	7	8	9	10
Baseline	SARSOP (MC)	50.41 (50.31, 50.51)
Baseline	Heuristic	70.19 (70.12, 70.25)
CTCE	DDQN	50.81	50.91	50.99	51.04	51.04	51.06	51.10	51.15	51.23	51.25
	JAC	50.89	51.20	51.24	51.30	51.36	51.78	53.78	53.79	53.86	53.87

Appendix

Table 3: The expected costs corresponding to the best checkpoint for each of the 10 training instances, reported in ascending order, across all algorithms and

k

-out-of-4 systems. These values correspond to the data summarized in the box plots shown in Figure 5 .

Figure 11: Learning curves of nine algorithms in all infinite-horizon

k

-out-of-4 systems summarized using 10 training instances. For off-policy algorithms, the performance is evaluated every 5,000 episodes during training, and for on-policy algorithms, we evaluate performance every 1M timesteps. Costs are normalised with respect to the SARSOP Monte Carlo mean, and the curves show the median performance across seeds, with shaded regions indicating the interquartile range (IQR).

Figure 12: SARSOP convergence plots. All settings timeout after 48 hours, before upper and lower bounds converge. We observe sharp drops in the number of alpha vectors in the 1-out-of-4 and 2-out-of-4 systems, indicating significant pruning.

Paradigm		CTCE			CTDE				DTDE
Algorithm		DDQN	JAC	DCMAC	IACC-PS	MAPPO-PS	VDN-PS	QMIX-PS	IAC-PS	IPPO-PS
	Episodes (E)	50,000				-	50,000			-
	Timesteps (T)	$E\times t_{h}=50,000\times 50=2.5$ M				20M	2.5M			20M
Architecture	Actor	-	[12, 64, 64, 81]	[12, 32, 32, 12]	[7, 32, 32, 3]	[7, 64, 64, 3]	-	-	[7, 32, 32, 3]	[7, 64, 64, 3]
	# parameters	-	10,257	1,868	1,411	4,867	-	-	1,411	4,867
	Critic	[12, 64, 64, 81]	[12, 64, 64, 1]	[12, 64, 64, 1]	[12, 64, 64, 1]	[12, 64, 64, 1]	[7, 64, 64, 3]	[7, 64, 64, 3]	[7, 64, 64, 1]	[7, 64, 64, 1]

Appendix

Table 4: Hyperparameters of single- and multi-agent DRL algorithms used across all

k

-out-of-

n

systems, and the Climb Game. However, in the Climb Game, we train off-policy algorithms for 100,000 episodes.

Figure 13: Empirical validation of our MAPPO-PS, QMIX-PS, and IPPO-PS implementations using EPyMARL Papoudakis et al. [2021] across all

k

-out-of-4 systems. Here, we juxtapose the performance of our implementation with the EPyMARL equivalent. Similar to Figure 5 , we summarize the performance over 10 training runs, with costs normalized using the SARSOP Monte Carlo mean from Table 2 . Note that we also used 100,000 Monte Carlo rollouts for the EPyMARL evaluations.

Figure 14: Ablation studies . Performance of MADRL algorithms and heuristic baselines are reported relative to the (near-)optimal SARSOP baseline. The

n=4

system with

\kappa=3

and mobilization cost

=4

(main text, Figure 5 ) serves as the base case. In the following ablations, only the specified environment parameters are varied, while algorithm hyperparameters and evaluation procedures remain unchanged. Blue arrows (

\uparrow

) indicate entries that exceed the plotted

y

-axis limits. Panel A : We test robustness by ablating the reward model, reducing the failure penalty factor from

\kappa=3

\kappa=1.5

and setting mobilization costs to 0, on the

n=4

system across all

k

. Panel B We ablate the number of components by considering an

n=3

system (with

\kappa=3

and mobilization cost 0) across all

k

. Panel C We further ablate the number of components by considering an

n=2

system (with

\kappa=3

and mobilization cost 0) across both values of

k

. Results confirm consistent performance trends, showing that our findings are not limited to a specific reward model.

Agent 2 $\mathtt{DN}$ $\mathtt{Rep}$ . Agent 1 $\mathtt{DN}$ $r_{\mathtt{DN},\mathtt{DN}}$ $r_{\mathtt{DN},\mathtt{Rep}.}$ $\mathtt{Rep}$ . $r_{\mathtt{Rep}.,\mathtt{DN}}$ $r_{\mathtt{Rep}.,\mathtt{Rep}.}$ (a) Generic joint reward matrix	Agent 2 $Q_{2}(\mathtt{DN})$ $Q_{2}(\mathtt{Rep}.)$ Agent 1 $Q_{1}(\mathtt{DN})$ $Q(\mathtt{DN},\mathtt{DN})$ $Q(\mathtt{DN},\mathtt{Rep}.)$ $Q_{1}(\mathtt{Rep}.)$ $Q(\mathtt{Rep}.,\mathtt{DN})$ $Q(\mathtt{Rep}.,\mathtt{Rep}.)$ (b) Value decomposition form
Agent 2 $\mathtt{DN}$ $\mathtt{Rep}$ . Agent 1 $\mathtt{DN}$ $-c_{f}$ $-c_{2}$ $\mathtt{Rep}$ . $-c_{1}$ $-(c_{1}+c_{2})$ (c) Parallel system	Agent 2 $\mathtt{DN}$ $\mathtt{Rep}$ . Agent 1 $\mathtt{DN}$ $-c_{f}$ $-(c_{f}+c_{2})$ $\mathtt{Rep}$ . $-(c_{f}+c_{1})$ $-(c_{1}+c_{2})$ (d) Series system

Appendix

Table 5: Two-agent matrix game used to analyze value decomposition. (a) shows a generic two-agent reward matrix. (b) shows its representation under value decomposition, where each agent chooses between doing nothing (

\mathtt{DN}

) and repairing (

\mathtt{Rep}

.). (c) and (d) specify reward structures for parallel and series systems respectively, parameterized by repair costs

c_{1},c_{2}

and a large failure penalty

c_{f}

, under the assumption that all components are initially in a failed state. These cases highlight how redundancy alters optimal coordination and exposes limitations of value factorization methods.

Decentralized MARL

Decentralized multi-agent reinforcement learning studies how autonomous agents learn coordinated behavior from local information, with or without communication.

See all 122 papers

Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning
Sep 22, 2025Chuhao Qin, Evangelos PournarasMulti-Agent System Optimization Multi-Agent Coordination
School of Computer Science, University of Leeds, UK
Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning
Feb 23, 2026Shan Yang, Yang LiuCooperative MARL Multi-Agent Reinforcement Learning
National University of Singapore
Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics
May 28, 2026Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders JonssonMulti-Agent System Optimization Multi-Agent Coordination
Department of Engineering, University Pompeu Fabra

Sep 22, 2025J/K move · Enter open · S save

Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning

Chuhao Qin, Evangelos Pournaras

School of Computer Science, University of Leeds, UK

Multi-Agent System Optimization Multi-Agent Coordination Hierarchical RL Multi-Agent Resource Allocation Multi-Robot Coordination Multi-Agent Systems+1

Open paper AskPDF arXiv

Efficient resource allocation in multi-agent systems requires autonomous agents to coordinate their decisions while balancing system-wide objectives with individual costs. This becomes increasingly challenging over long time horizons, where decisions that improve the current allocation may compromise future resource allocation, while decentralized agents have limited observations of the overall system. Multi-agent reinforcement learning (MARL) can learn such long-term dependencies via local observations, but directly applying it to large-scale coordination leads to rapidly growing decision spaces and inefficient training. To this end, we propose Hierarchical Reinforcement and Collective Learning (HRCL), a hierarchical framework that uses MARL to guide, rather than replace, decentralized multi-agent coordination. At the high level, MARL learns strategies that restrict the alternatives considered during coordination and guide agents in balancing system-wide and individual objectives. At the low level, agents perform efficient decentralized coordination under this strategic guidance. This separation reduces the learning space and allows short-term coordination trade-offs to be evaluated according to their long-term effects. Experiments on a synthetic benchmark show that HRCL converges substantially faster than standalone MARL and reduces system-wide and individual costs by 35.53% and 27.05%, respectively. Evaluations on energy self-management and drone swarm sensing further show improved resource allocation, power-peak regulation, and sensing efficiency. These results show that learning strategic guidance for an existing coordination process can retain scalable decentralized coordination without letting short-term decisions compromise future resource allocation.

Abstract

Decentralized MARL

Explore similar work

Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning

Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning

Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics