The price of decentralization in managing engineering systems through multi-agent reinforcement learning
Authors: Prateek Bhustali, Pablo G. Morato, Konstantinos G. Papakonstantinou, Charalampos P. Andriotis
Organizations: Faculty of Architecture and the Built Environment, Delft University of Technology, Delft, 2628 BL, The Netherlands · Engineering Risk Analysis Group, Technical University of Munich, Munich, 80333, Germany · Department of Civil & Environmental Engineering, The Pennsylvania State University, University Park, PA, 16802, USA
Inspection and maintenance (I&M) planning involves sequential decision making under uncertainties and incomplete information, and can be modeled as a partially observable Markov decision process (POMDP). While single-agent deep reinforcement learning provides approximate solutions to POMDPs, it does not scale well in multi-component systems. Scalability can be achieved through multi-agent deep reinforcement learning (MADRL), which decentralizes decision-making across multiple agents, locally controlling individual components. However, this decentralization can induce cooperation pathologies that degrade the optimality of the learned policies. To examine these effects in I&M planning, we introduce a set of deteriorating systems in which redundancy is varied systematically. These benchmark environments are designed such that computation of centralized (near-)optimal policies remains tractable, enabling direct comparison of solution methods. We implement and benchmark a broad set of MADRL algorithms spanning fully centralized and decentralized training paradigms, from value-factorization to actor-critic methods. Our results show a clear effect of redundancy on coordination: MADRL algorithms achieve near-optimal performance in series-like settings, whereas increasing redundancy amplifies coordination challenges and can lead to optimality losses. Nonetheless, decentralized agents learn structured policies that consistently outperform optimized heuristic baselines, highlighting both the promise and current limitations of decentralized learning for scalable maintenance planning.
Figures & tables
Figure 1: In multi-component systems, joint state, action, and observation spaces grow exponentially with the number of components, rendering single-agent approaches intractable. Dec-POMDPs address this curse of dimensionality by decentralizing control and are commonly solved using multi-agent DRL methods. Left : Scalability of solution approaches. The curved boundaries illustrate the approximate, relative scalability of POMDP and Dec-POMDP solution approaches as these spaces increase. The ( ) indicates the approximate size of the systems studied in this work, where single-agent methods remain tractable and allow direct comparison with multi-agent approaches. Right : Agent-centric paradigms. Single-agent approaches are intrinsically centralized because a single agent possesses all information during training and execution. In contrast, multi-agent approaches, such as CTDE, relax decentralization during training by allowing agents to share information, but agents must learn to execute actions independently at execution/inference.
Figure 2: A Dec-POMDP unrolled as a dynamic decision network over three time steps. At each time step t , the environment state st evolves according to the transition model st+1∼T(⋅∣st,at) under the joint action at , yielding a global reward rt and individual observations ot+1∼O(⋅∣st+1,at) for each agent m∈M . Each agent selects its action based on its local action–observation history hmt .
Figure 3: An overview of MADRL algorithms for managing a (four-component) reliability system. Algorithms are classified by training–execution paradigms: centralized training centralized execution (CTCE), centralized training decentralized execution (CTDE), and decentralized training decentralized execution (DTDE), reflecting increasing decentralization and information constraints. Bottom labels indicate the underlying formulation framework (POMDP, MPOMDP, Dec-POMDP). Suffix PS denotes parameter sharing across agents.
Paradigm
Formulation
Algorithm
Actor (policy-based only)
Critic / Value Function
Framework
Name
Type
On/Off-policy
Input
Output
Input
Output
CTCE
POMDP (SADRL)
DDQN
V
Off-policy
-
-
Centralized
Centralized
JAC
A+C
Off-policy
Centralized
Centralized
Centralized
Centralized
MPOMDP
DCMAC
A+C
Off-policy
Centralized
Local
Centralized
Centralized
CTDE
Dec-POMDP
IACC-PS
A+C
Off-policy
Local
Local
Centralized
Centralized
MAPPO-PS
A+C
On-policy
Local
Local
Centralized
Centralized
Table 1: Summary of single- and multi-agent DRL algorithms shown in Figure 3 . The Actor columns describe the policy, when present, with Input indicating the information available to the policy and Output indicating whether actions are produced jointly, a , or per agent, am . The Critic / Value Function columns analogously report the information available to the value estimator and whether it outputs a single centralized value (e.g., Q(h,a) , V(h) ) or per-agent values (e.g., Qm(hm,am) ). Centralized denotes access to global state or joint observations and actions, whereas Local denotes access only to an agent’s own observation or action–observation history.
Figure 4: Left: Payoff matrix for the Climb Game with two agents, each choosing from three actions. The joint action (a1,a1) yields the optimal reward of 11 [ Claus and Boutilier, 1998 ] . Right: Best performance of MADRL algorithms aggregated over 30 training instances in the Climb Game played repeatedly for 25 time steps [ Papoudakis et al., 2021 ] . The optimal return is 11×25=275 .
Figure 5: Box plots summarize algorithm performance across k -out-of-4 systems, aggregated over 10 training instances. The results are normalized using the expected cost of the (near-)optimal SARSOP policy (Table 2 ), allowing comparison across varying redundancy levels. Most decentralized algorithms perform near-optimally in series configurations (k=n) , but deviate from the (near-)optimal SARSOP baseline as redundancy increases, with the largest performance drop observed in the parallel configurations (k=1) .
k -out-of-4 systems
1-out-of-4 system
2-out-of-4 system
3-out-of-4 system
4-out-of-4 system
SARSOP
(LBound, UBound)
(43.67, 51.82)
(113.19, 141.76)
(272.84, 309.06)
(727.36, 755.52)
Expected return
50.41 (50.31, 50.51)
138.59 (138.25, 138.93)
303.93 (303.18, 304.68)
747.54 (745.92, 749.15)
Heuristic
Expected return
70.19 (70.12, 70.25)
159.32 (159.01, 159.63)
358.51 (357.44, 359.57)
913.03 (911.36, 914.70)
CTCE
DDQN
50.81 (50.71, 50.90)
141.70 (141.48, 141.92)
312.61 (311.86, 313.36)
750.20 (748.62, 751.77)
JAC
50.89 (50.81, 50.98)
143.66 (143.38, 143.94)
311.95 (311.20, 312.70)
749.90 (748.34, 751.46)
DCMAC
53.83 (53.75, 53.92)
144.45 (144.22, 144.69)
311.37 (310.57, 312.17)
749.57 (747.99, 751.16)
Table 2: Best performance of single and multi-agent DRL algorithms across k -out-of-4 systems over 10 training instances. Reported values are expected costs, computed as discounted costs via Monte Carlo rollouts, with 95% confidence intervals in brackets. SARSOP, using one-step look-ahead, serves as the near-optimal baseline. SARSOP expected costs shown in bold are also used to normalize results in Figure 5 . Entries marked with ( ‡ ) correspond to deterministic periodic repair policies and exhibit negligible variance.
Figure 6: Belief space plots. Using the best-performing training instance of each algorithm, we visualize the learned policies over the belief space. Specifically, we show the actions taken over 100 Monte Carlo rollouts of 20 time steps in the 1-out-of-4 system. Each rollout begins from the initial belief b0=(0.6,0.4,0) indicated with a blue dot ( • ). The SARSOP baseline, JAC, and DDQN coordinate repairs across components 1 and 2 and use inspections to keep beliefs updated; they let components 3 and 4 drift toward failure to optimally exploit redundancy. Decentralized algorithms, such as DCMAC, IACC-PS, MAPPO-PS, IPPO-PS, learn a structured coordination of components 1 and 2 (e.g., periodic repairs to eliminate non-stationarity), yet these policies remain suboptimal as detailed in Table 2 .
Figure 7: A sample rollout of the best IACC-PS policy in the 1-out-of-4 system for 40 time steps. Interestingly, agent 1 chooses a repair action every 5 and 7 time steps, and agent 2 chooses repairs every 5 time steps, even though neither observes the time step, and both are purely based on their individual components’ beliefs.
Figure 8: Value decompositions for the Climb Game. Left: Climb Game rewards. Middle and right: value decompositions learned by the best-performing instances of VDN-PS and QMIX-PS, respectively. Tables show estimated joint action-values, with rows and columns corresponding to the agents’ local actions; orange entries denote the greedy joint action. While VDN-PS assigns the correct maximum to the optimal joint action (a3,a2) , its linear factorization misrepresents other action utilities. QMIX-PS, despite its monotonic mixing constraint, also misrepresents joint utilities, with optimal performance occurring only in outlier cases.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Sub-optimal JAC policy in the 1-out-of-4 system. While the optimal JAC and SARSOP policies intervene on components 1 and 2 to maintain system functionality, this learned policy only involves agent 2 taking intervention actions, and all other agents always choosing do-nothing.
Figure 10: Learning curves of CTCE, CTDE, and DTDE algorithms on the Climb Game. Performance is evaluated periodically during training and shown as median cost across seeds, with shaded regions indicating the interquartile range. In this one-step environment, the initial episode is excluded from best-checkpoint selection, since random initializations can yield the optimal action before any learning occurs.
Training instances
Paradigm
Algorithm
1
2
3
4
5
6
7
8
9
10
1-out-of-4 system (parallel configuration)
Baseline
SARSOP (MC)
50.41 (50.31, 50.51)
Heuristic
70.19 (70.12, 70.25)
CTCE
DDQN
50.81
50.91
50.99
51.04
51.04
51.06
51.10
51.15
51.23
51.25
JAC
50.89
51.20
51.24
51.30
51.36
51.78
53.78
53.79
53.86
53.87
Appendix
Table 3: The expected costs corresponding to the best checkpoint for each of the 10 training instances, reported in ascending order, across all algorithms and k -out-of-4 systems. These values correspond to the data summarized in the box plots shown in Figure 5 .
Figure 11: Learning curves of nine algorithms in all infinite-horizon k -out-of-4 systems summarized using 10 training instances. For off-policy algorithms, the performance is evaluated every 5,000 episodes during training, and for on-policy algorithms, we evaluate performance every 1M timesteps. Costs are normalised with respect to the SARSOP Monte Carlo mean, and the curves show the median performance across seeds, with shaded regions indicating the interquartile range (IQR).
Figure 12: SARSOP convergence plots. All settings timeout after 48 hours, before upper and lower bounds converge. We observe sharp drops in the number of alpha vectors in the 1-out-of-4 and 2-out-of-4 systems, indicating significant pruning.
Paradigm
CTCE
CTDE
DTDE
Algorithm
DDQN
JAC
DCMAC
IACC-PS
MAPPO-PS
VDN-PS
QMIX-PS
IAC-PS
IPPO-PS
Episodes (E)
50,000
-
50,000
-
Timesteps (T)
E×th=50,000×50=2.5 M
20M
2.5M
20M
Architecture
Actor
-
[12, 64, 64, 81]
[12, 32, 32, 12]
[7, 32, 32, 3]
[7, 64, 64, 3]
-
-
[7, 32, 32, 3]
[7, 64, 64, 3]
# parameters
-
10,257
1,868
1,411
4,867
-
-
1,411
4,867
Critic
[12, 64, 64, 81]
[12, 64, 64, 1]
[12, 64, 64, 1]
[12, 64, 64, 1]
[12, 64, 64, 1]
[7, 64, 64, 3]
[7, 64, 64, 3]
[7, 64, 64, 1]
[7, 64, 64, 1]
Appendix
Table 4: Hyperparameters of single- and multi-agent DRL algorithms used across all k -out-of- n systems, and the Climb Game. However, in the Climb Game, we train off-policy algorithms for 100,000 episodes.
Figure 13: Empirical validation of our MAPPO-PS, QMIX-PS, and IPPO-PS implementations using EPyMARL Papoudakis et al. [2021] across all k -out-of-4 systems. Here, we juxtapose the performance of our implementation with the EPyMARL equivalent. Similar to Figure 5 , we summarize the performance over 10 training runs, with costs normalized using the SARSOP Monte Carlo mean from Table 2 . Note that we also used 100,000 Monte Carlo rollouts for the EPyMARL evaluations.
Figure 14: Ablation studies . Performance of MADRL algorithms and heuristic baselines are reported relative to the (near-)optimal SARSOP baseline. The n=4 system with κ=3 and mobilization cost =4 (main text, Figure 5 ) serves as the base case. In the following ablations, only the specified environment parameters are varied, while algorithm hyperparameters and evaluation procedures remain unchanged. Blue arrows ( ↑ ) indicate entries that exceed the plotted y -axis limits. Panel A : We test robustness by ablating the reward model, reducing the failure penalty factor from κ=3 to κ=1.5 and setting mobilization costs to 0, on the n=4 system across all k . Panel B We ablate the number of components by considering an n=3 system (with κ=3 and mobilization cost 0) across all k . Panel C We further ablate the number of components by considering an n=2 system (with κ=3 and mobilization cost 0) across both values of k . Results confirm consistent performance trends, showing that our findings are not limited to a specific reward model.
Agent 2 DNRep . Agent 1 DN−cf−(cf+c2)Rep . −(cf+c1)−(c1+c2) (d) Series system
Appendix
Table 5: Two-agent matrix game used to analyze value decomposition. (a) shows a generic two-agent reward matrix. (b) shows its representation under value decomposition, where each agent chooses between doing nothing ( DN ) and repairing ( Rep .). (c) and (d) specify reward structures for parallel and series systems respectively, parameterized by repair costs c1,c2 and a large failure penalty cf , under the assumption that all components are initially in a failed state. These cases highlight how redundancy alters optimal coordination and exposes limitations of value factorization methods.
Decentralized multi-agent reinforcement learning studies how autonomous agents learn coordinated behavior from local information, with or without communication.
Efficient resource allocation in multi-agent systems requires autonomous agents to coordinate their decisions while balancing system-wide objectives with individual costs. This becomes increasingly challenging over long time horizons, where decisions that improve the current allocation may compromise future resource allocation, while decentralized agents have limited observations of the overall system. Multi-agent reinforcement learning (MARL) can learn such long-term dependencies via local observations, but directly applying it to large-scale coordination leads to rapidly growing decision spaces and inefficient training. To this end, we propose Hierarchical Reinforcement and Collective Learning (HRCL), a hierarchical framework that uses MARL to guide, rather than replace, decentralized multi-agent coordination. At the high level, MARL learns strategies that restrict the alternatives considered during coordination and guide agents in balancing system-wide and individual objectives. At the low level, agents perform efficient decentralized coordination under this strategic guidance. This separation reduces the learning space and allows short-term coordination trade-offs to be evaluated according to their long-term effects. Experiments on a synthetic benchmark show that HRCL converges substantially faster than standalone MARL and reduces system-wide and individual costs by 35.53% and 27.05%, respectively. Evaluations on energy self-management and drone swarm sensing further show improved resource allocation, power-peak regulation, and sensing efficiency. These results show that learning strategic guidance for an existing coordination process can retain scalable decentralized coordination without letting short-term decisions compromise future resource allocation.
Chuhao Qin, Evangelos Pournaras
School of Computer Science, University of Leeds, UK
Scaling cooperative multi-agent reinforcement learning (MARL) is fundamentally limited by cross-agent noise. When agents share a common reward, each agent's learning signal is computed from a shared return that depends on all agents, so the stochasticity of the other agents enters the signal as cross-agent noise that grows with N. Many engineering systems, such as cloud computing and power systems, have differentiable analytical models that prescribe efficient system states, providing a new reference beyond noisy shared returns. In this work, we propose Descent-Guided Policy Gradient (DG-PG), a framework that augments policy-gradient updates with a low-noise descent signal toward this reference. We prove that DG-PG reduces policy-gradient estimator variance from O(N) to O(1), preserves the stationary points of the cooperative objective, and achieves agent-independent sample complexity O(1/ε3). We evaluate DG-PG on cloud resource scheduling and power dispatch tasks, which involve discrete and continuous decisions, respectively. In both tasks, DG-PG converges within 10 episodes on average at every scale up to 1500 heterogeneous agents.
We present a distributed approach for constrained Multi-Agent Reinforcement Learning (MARL) that combines state-augmented policy learning with distributed consensus over dual variables. Our method targets systems where agents have separable dynamics but must coordinate to satisfy global resource constraints, a setting in which, as we demonstrate empirically, independent learning fails to produce feasible solutions because agents cannot determine appropriate individual contributions toward collective constraint satisfaction. The key technical contribution is showing that lightweight neighbor-to-neighbor consensus over Lagrange multipliers suffices for globally coordinated constraint enforcement while preserving the scalability of independent training. Each agent learns a single augmented policy offline, conditioned on both its local state and a dual variable encoding constraint feedback. During execution, agents reach agreement on this dual variable through local communication alone. We prove that under mild connectivity assumptions, the consensus error among agents' multipliers is bounded, and show that this translates to a bounded constraint violation that decreases with graph connectivity and the number of consensus rounds. Unlike centralized training with decentralized execution (CTDE) approaches, whose complexity grows at least quadratically with agent count, our method scales linearly in both training and execution. Experiments on smart grid demand response demonstrate that consensus coordination is essential for feasibility: without it, agents satisfy grid capacity constraints only by indefinitely postponing demand, a degenerate non-solution. With consensus, agents converge to a shared dual variable and satisfy both grid constraints and demand fulfillment, scaling to thousands of agents while CTDE baselines are limited to dozens.
Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders Jonsson
Department of Engineering, University Pompeu Fabra