Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form

Xuefeng Wang; Lei Zhang; Henglin Pu; Husheng Li; Ahmed H. Qureshi

cs.MAFeb 19, 2026

Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form

Authors: Xuefeng Wang, Lei Zhang, Henglin Pu, Husheng Li, Ahmed H. Qureshi

Organizations: Purdue University

Abstract

Multi-agent reinforcement learning (MARL) has made significant progress in recent years, but most algorithms still rely on a discrete-time Markov Decision Process (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the development of continuous-time MARL (CT-MARL). Existing CT-MARL methods are mainly built on Hamilton-Jacobi-Bellman (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these introduce discontinuities that make HJB-based learning difficult. To address this challenge, we propose a continuous-time constrained MDP (CT-CMDP) formulation and a novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. We then solve this by proposing a novel physics-informed neural network (PINN)-based actor-critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi-particle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method.

Figures & tables

Figure 1: Overview of the proposed epigraph-based CT-MARL framework. The pipeline begins with data collection, where individual agent rollouts are aggregated into a centralized rollout $\mathcal{X}_{R}$ for the training; the outer optimization computes optimal $z^{*}$ to balance discounted cumulative cost and safety constraints; the inner optimization corresponds to critic learning, where return networks $V^{\text{ret}}_{\psi}(x)$ and constraint value networks $V^{\text{cons}}_{\phi}(x)$ are optimized jointly with the optimal auxiliary state $z^{*}$ ; and actor learning leverages the advantage function to improve policies.

Figure 2: Overall results for adapted MPE environments.

Figure 3: Performance of constraints and cost over MPE settings.

Figure 4: Overall results for adapted multi-agent MuJoCo environments.

Figure 5: Ablation study of different loss terms in critic network over MPE.

Figure 6: Performance of constraints and cost over MuJoCo settings.

Figure 7: Weighted loss performance.

Figure 8: Performance with and without epigraph reformulation.

Figure 9: Sensitivity test of different z choices.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Trajectory demonstrations (key frames) across methods in Formation. Row 1: EPI results, Row 2: EPPO results, Row 3: MACPO results.

Figure 11: Performance under Different Noise Levels.

Figure 12: Average Distance to the Target under Different

\Delta t

Figure 13: $z^{*}$ Trajectory through the Training in Target.

Figure 14: Performance of EPI and Traditional Epigraph under MPE settings.

Figure 15: Performance of EPI and Discrete-time Baselines under MPE settings.

Parameter	Value
Episode length for MPE	50
Episode length for MuJoCo	100
Episode length for Didactic	50
Total number of episode for MPE	30000
Total number of episode for MuJoCo	30000
Total number of episode for Didactic	3000

Appendix

Table 1: Hyperparameter settings used.

Network	Input Dimension	Architecture and Activation
Return Value Network	State ( $d$ )	FC(128) → FC(128) → FC(1), ReLU or Tanh
Constraint Value Network	State ( $d$ )	FC(128) → FC(128) → FC(1), ReLU or Tanh
Dynamics Network	State + Joint Action ( $d+na$ )	FC(128) → FC(128) → FC( $d$ ), ReLU
Reward Network	State + Joint Action ( $d+na$ )	FC(128) → FC(128) → FC(1), ReLU
PolicyNet	Observation + Time Interval ( $o+1$ )	FC(128) → FC(128) → FC(64) → FC( $a$ ), ReLU

Appendix

Table 2: Summary of neural network architectures used in our framework.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control
Jun 22, 2026Zihao Guo, Jianing Zhao, Ling Li +3Robotic Control Multi-Agent Coordination
1King’s College London · University of California, Berkeley · 3The Alan Turing Institute
TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning
Jun 16, 2026Zijie Meng, Ziwei Li, Yufei Liu +5Cyber-Physical Systems Reinforcement Learning
Peking University · National Taiwan University · WHU +1
Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology
Sep 8, 2026Xiao Sizhe, Dong Lijing, Bai Rui +1Multi-Robot Coordination Safe RL
School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, Beijing, China

Jun 22, 2026J/K move · Enter open · S save

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Zihao Guo, Jianing Zhao, Ling Li, Hao Liang+2

1King’s College London · University of California, Berkeley · 3The Alan Turing Institute

Robotic Control Multi-Agent Coordination Hierarchical RL Cooperative MARL Multi-Agent Reinforcement Learning Constrained MARL

Open paper AskPDF arXiv

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology