Fully Decentralized and Safety-Aware Multi-Agent Reinforcement Learning for Control on Networks

Theodore Rogalski; Shirantha Welikala

eess.SYSep 28, 2026

Fully Decentralized and Safety-Aware Multi-Agent Reinforcement Learning for Control on Networks

Authors: Theodore Rogalski, Shirantha Welikala

Organizations: Electrical & Computer Engineering Department Stevens Institute of Technology Hoboken, USA

Abstract

This paper develops a safe and fully decentralized multi-agent reinforcement learning (MARL) algorithm to solve a class of discrete-time control problems on networks, including the persistent monitoring problem. Fully decentralized control of agents, while offering numerous benefits, faces issues such as exponentially increasing sample complexity, lack of global information about the system, and challenges in coordinating between agents. To address these issues, this paper introduces a fully decentralized multi-agent reinforcement learning algorithm that integrates deep reinforcement learning with safety considerations. This method feeds a history of local observations of the network's state into two parallel neural-network branches: the graph encoder, which adds structural information and correlations among nodes, and a state estimator, which predicts the uncertainty at each node in the graph. Additionally, the result of feeding that input into an actor-critic network is passed through a discrete-time control barrier heuristic to reduce the likelihood that any node will be neglected. This approach enables teams of fully decentralized agents to solve challenging problems by increasing system awareness and incorporating built-in safety measures to prevent the adoption of potentially harmful control policies. Numerical results from a custom simulation environment demonstrate that the proposed algorithm achieves 26.3 percent lower average uncertainty than a centralized control policy and is within 1 percent of the uncertainty performance of a more computationally complex algorithm with added attention layers.

Figures & tables

Fig. 1: A persistent monitoring system with six nodes and two agents at discrete time steps $k$ and $k+1$ .

Fig. 2: At time-step $k$ , the agent $\mathcal{A}_{i}$ receives $\mathcal{M}_{i,k}$ , which is processed in the following manner: concatenation with $\mathcal{L}_{i}$ , transformation by Graph Attention (GAT), concatenation with $\mathcal{L}_{i}(\mathcal{M}_{i,k})$ and then transformation by the Actor network. The result of the processed observation is passed through a safety layer and then sent to $\mathcal{A}_{i}$ as a probability mass function over possible actions.

Fig. 3: Network with 50 nodes, eight targets (highlighted in peach) and connections between 20% of nodes.

Fig. 4: Various reward (above) and reward (below) curves for different parts of the training process, with curves (a) and (e) representing episode one, curves (b) and (d) representing episode seven, and curves (c) and (f) representing episode 15, demonstrating the process of moving away from a random control policy to a more effective one over time

Fig. 5: An exponentially increasing pattern for $\mathcal{A}_{3}$ for one episode (a), a pattern from $\mathcal{A}_{3}$ demonstrating a low-loss pattern for one episode (b) & state estimator loss at one early episode with the proposed model architecture on a graph with 100 nodes for four agents (c)

Algorithm	Avg. Uncertainty	Std. Dev. ( Uncertainty )	Avg. Neglect Time	Std. Dev. ( Neglect )
Proposed Algorithm	1736.3	895	1176.6	561.6
Grazing	1128.1	1078.4	276.7	195.2
Random	3006	1147.9	5001 *	0
Sit on Nodes	3006	1147.9	5001 *	0
Centralized Algorithm	2356.7	1120	322	213

TABLE I: Simulation Performance Comparison

Algorithm	Avg. Uncertainty	Std. Dev. ( Uncertainty )	Avg. Neglect Time	Std. Dev. ( Neglect )
Proposed Algorithm	1736.3	895	1176.6	561.6
No DCBF	3006.0	994	10001 *	0
No State Est.	1770.0	831	1241.3	667.4
No Collisions	1750.4	847	1139.3	564.3
Extra Attention	1734.6	830	1103.1	439.5

TABLE II: Ablation and Variant Analysis

Explore similar work

Jun 10, 2026cs.LG

Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning

Offline reinforcement learning allows control policies to be learned directly from data without online interaction, making it suitable for safety-critical tasks. Recent studies have applied diffusion models to offline reinforcement learning to leverage their strong capacity for modeling complex data distributions. However, existing approaches primarily focus on single-agent settings, leaving the safety challenges in multi-agent environments largely unexplored. In this work, we propose a safe offline multi-agent reinforcement learning algorithm that embeds neural individual control barrier functions into the diffusion model to enhance safety during trajectory generation, with control policies recovered through inverse dynamics. We evaluate our algorithm across diverse benchmarks, demonstrating substantial safety improvements while maintaining competitive rewards.

Qingyun Guo, Junyi Shi, Jianuo Huang +1

Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland · School of Computing and Data Science, Xiamen University Malaysia, Sepang, Malaysia · Department of Computer Science, University of Toronto, Toronto, Canada

Multi-Agent Reinforcement Learning Model-Based Reinforcement Learning Offline Reinforcement Learning Control Barrier Functions Inverse Dynamics Barriers+1

Ask PDF

Sep 28, 2025cs.RO

MAD-PINN: A Decentralized Physics-Informed Machine Learning Framework for Safe and Optimal Multi-Agent Control

Co-optimizing safety and performance in large-scale multi-agent systems remains a fundamental challenge. Existing approaches based on multi-agent reinforcement learning (MARL), safety filtering, or Model Predictive Control (MPC) either lack strict safety guarantees, suffer from conservatism, or fail to scale effectively. We propose MAD-PINN, a decentralized physics-informed machine learning framework for solving the multi-agent state-constrained optimal control problem (MASC-OCP). Our method leverages an epigraph-based reformulation of SC-OCP to simultaneously capture performance and safety, and approximates its solution via a physics-informed neural network. Scalability is achieved by training the SC-OCP value function on reduced-agent systems and deploying them in a decentralized fashion, where each agent relies only on local observations of its neighbours for decision-making. To further enhance safety and efficiency, we introduce an Hamilton-Jacobi (HJ) reachability-based neighbour selection strategy to prioritize safety-critical interactions, and a receding-horizon policy execution scheme that adapts to dynamic interactions while reducing computational burden. Experiments on multi-agent navigation tasks demonstrate that MAD-PINN achieves superior safety-performance trade-offs, maintains scalability as the number of agents grows, and consistently outperforms state-of-the-art baselines.

Manan Tayal, Aditya Singh, Shishir Kolathaya +1

Center for Cyber-Physical Systems, Indian Institute of Science, Bangalore, India · Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, USA · Department of Aeronautics and Astronautics, Stanford University, USA

Artificial Intelligence Control Physics-Informed Learning

Ask PDF

Jun 22, 2026cs.AI

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.

Zihao Guo, Jianing Zhao, Ling Li +3

1King’s College London · University of California, Berkeley · 3The Alan Turing Institute

Multi-Agent Reinforcement Learning Safety Constraints Learning Policy Global Coordination Hierarchical

Ask PDF