SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning

Hongrui Zheng; Aditya Singh; Rahul Mangharam

cs.LGOct 9, 2026

SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning

Authors: Hongrui Zheng, Aditya Singh, Rahul Mangharam

Organizations: University of Pennsylvania, Philadelphia, USA

Abstract

Cooperative multi-agent tasks often impose constraints on a shared objective or collective safety. A team level violation cost function can measure this constraint outcome, but does not distinguish which agents' actions most strongly influence that violation. We present SafeDiSTL, a constrained multi-agent reinforcement learning framework that allows for credit assignment even with joint constraint measurements. We use Signal Temporal Logic (STL) robustness to measure episode-level violation through running-maximum cost increments. Smooth full-episode robustness and differentiable dynamics provide action sensitivities, which allocate each cost increment across agents at the same decision time. These sensitivities account for delayed effects when weighting agents' shares of each cost increment. We study the framework in cooperative payload transport, connected navigation, and networked traffic control tasks. SafeDiSTL achieves 1.6-5.1x the safe-success rate in transport, improves navigation safe-success by 11.1%, and delivers 50-68% more throughput in traffic when compared to constrained baselines. Results demonstrate a strong improvement of the learned policy in dealing with shared constraint while retaining the performance of unconstrained policies.

Figures & tables

Figure 1: An illustration of the responsibility attribution mechanism. (a) A shared payload approaches an obstacle and incurs a constraint violation. (b) Uniform responsibility allocation produces conflicting corrections and agents stay deadlocked. (c) Agent-specific attribution enables coordinated obstacle avoidance.

Figure 2: Constraint–performance tradeoffs across benchmarks. Points and error bars show means and sample standard deviations across six training seeds. Traffic reports overload-free episode percentages versus completed trips. Others report safe-success rates where goal was reached without constraint violation versus task return.

x

-axes metrics are normalized by MAPPO values. Note that the sub-plots zooms into parts of the

y

-range for better visibility and comparison, and hence have different

y

-axes ranges.

Figure 3: Transport results across obstacle counts (mean

\pm

SD over six training seeds). Labels are SafeDiSTL mean values.

Figure 4: Example rollout comparison with baselines in the 1-obstacle transport scenario with common initial conditions.

Figure 5: Example rollout comparison with baselines in the connected navigation benchmark with common initial conditions.

Figure 6: Navigation results (mean

\pm

SD over six training seeds). Labels are SafeDiSTL mean values.

Figure 7: Seven region traffic network with a shared core occupancy constraint. Controls for each agent is a 6-D vector that controls the flow rate for all 6 destinations (0 always for self).

Figure 8: Traffic results (mean

\pm

SD over six training seeds). Labels are SafeDiSTL mean values.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Obstacles	Algorithm	Return	Goal success	Constraint satisfaction	Safe success
1	MAPPO	$133.68\pm 7.96$	$75.7\pm 13.7$	$10.4\pm 3.8$	$8.8\pm 4.4$
1	MACPO	$49.25\pm 2.23$	$0.0\pm 0.0$	$9.1\pm 3.9$	$0.0\pm 0.0$
1	MAPPO-L	$105.49\pm 5.40$	$33.9\pm 4.8$	$33.4\pm 2.5$	$7.0\pm 3.7$
1	Ours	$125.83\pm 10.24$	$42.2\pm 9.7$	$82.0\pm 5.9$	$35.6\pm 9.6$
2	MAPPO	$111.98\pm 4.19$	$38.6\pm 8.6$	$4.5\pm 1.9$	$2.5\pm 1.3$
2	MACPO	$49.65\pm 0.91$	$0.0\pm 0.0$	$8.0\pm 3.4$	$0.0\pm 0.0$

Appendix

Table 1: Supplementary Transport results compared to baselines. Entries are mean

\pm

sample SD across six seeds and 500 common resets per seed. Rates are percentages. Safe success denotes joint occurrence of goal reaching and shared-constraint satisfaction.

Algorithm	Return	Goal success (%)	Constraint satisfaction (%)	Safe success (%)	Safe return	Final 20 Step distance
MAPPO	$130.43\pm 3.38$	$51.17\pm 3.31$	$0.91\pm 1.04$	$0.78\pm 0.99$	$1.49\pm 1.72$	$2.14\pm 0.16$
MACPO	$90.32\pm 9.70$	$11.85\pm 5.79$	$31.25\pm 4.39$	$4.43\pm 1.96$	$29.24\pm 3.29$	$2.59\pm 0.23$
MAPPO-L	$114.14\pm 8.71$	$52.99\pm 12.67$	$56.64\pm 10.61$	$32.81\pm 8.62$	$69.80\pm 10.19$	$1.84\pm 0.27$
Ours	$125.11\pm 8.80$	$71.88\pm 6.13$	$54.69\pm 8.33$	$43.88\pm 4.40$	$75.31\pm 10.50$	$1.83\pm 0.30$

Appendix

Table 2: Supplementary Navigation results compared to baselines. Entries are mean

\pm

sample SD across six seeds and 128 common resets per seed. Goal success, constraint satisfaction, and safe success are percentages; safe success requires both goal achievement and whole-episode connectivity. Safe return assigns zero to unsafe episodes; final 20 step distance is in meters.

Method	Completed trips	Network vehicle-hours	Core vehicle-hours	Episode-max core cost	Time above core limit
MAPPO	$1.000\pm 0.012$	$1.000\pm 0.001$	$1.000\pm 0.041$	$1.000\pm 0.081$	$1.000\pm 0.010$
MACPO	$0.476\pm 0.007$	$1.053\pm 0.001$	$0.091\pm 0.002$	$0.000\pm 0.000$	$0.000\pm 0.000$
MAPPO-L	$0.425\pm 0.016$	$1.058\pm 0.002$	$0.104\pm 0.007$	$0.00074\pm 0.00180$	$0.009\pm 0.023$
Ours	$0.714\pm 0.066$	$1.029\pm 0.007$	$0.088\pm 0.018$	$0.000061\pm 0.000053$	$0.003\pm 0.004$

Appendix

Table 3: Supplementary Traffic results relative to MAPPO (column-wise MAPPO mean

=1

). Mean

\pm

SD across six training seeds; 1,000 common reset scenarios per policy. Higher completed trips and lower values in all other columns are preferable.

Obstacles	Algorithm	Return	Goal success	Shared constraint sat.	Safe success
1	Uniform attribution Max-incremental cost	$58.91\pm 8.30$	$0.5\pm 0.8$	$74.4\pm 6.3$	$0.2\pm 0.4$
1	Uniform attribution Max-incremental cost + Adam	$70.00\pm 26.21$	$8.4\pm 8.7$	$75.3\pm 6.5$	$5.6\pm 7.7$
1	Uniform attribution Step cost	$46.81\pm 10.14$	$0.0\pm 0.0$	$57.7\pm 9.1$	$0.0\pm 0.0$
1	Uniform attribution Step cost + Adam	$44.26\pm 5.41$	$0.0\pm 0.0$	$58.6\pm 11.7$	$0.0\pm 0.0$
1	Ours	$125.83\pm 10.24$	$42.2\pm 9.7$	$82.0\pm 5.9$	$35.6\pm 9.6$
2	Uniform attribution Max-incremental cost	$59.40\pm 6.96$	$0.7\pm 1.4$	$61.8\pm 8.9$	$0.2\pm 0.3$

Appendix

Table 4: Two-dimensional transport ablations for one through three obstacles. Rates are percentages; entries are mean

\pm

sample SD across six training seeds, with 500 common resets per seed. All three Ours rows are copied unchanged from Table 1 .

Algorithm	Return	Goal (%)	Safe (%)	Safe goal (%)	Safe return	Final-20 distance
Uniform attribution Max-incremental cost	$126.77\pm 9.41$	$54.30\pm 15.19$	$2.60\pm 2.55$	$2.60\pm 2.55$	$4.21\pm 4.19$	$2.13\pm 0.34$
Uniform attribution Max-incremental cost + Adam	$128.58\pm 5.08$	$50.52\pm 13.01$	$0.39\pm 0.65$	$0.39\pm 0.65$	$0.65\pm 1.10$	$2.17\pm 0.29$
Uniform attribution Step cost	$39.60\pm 22.40$	$12.63\pm 9.34$	$90.36\pm 4.57$	$11.98\pm 9.11$	$36.42\pm 20.95$	$5.29\pm 1.66$
Uniform attribution Step cost + Adam	$129.38\pm 15.43$	$61.33\pm 20.97$	$11.98\pm 8.81$	$10.16\pm 9.40$	$19.28\pm 14.75$	$1.94\pm 0.43$
Ours	$125.11\pm 8.80$	$71.88\pm 6.13$	$54.69\pm 8.33$	$43.88\pm 4.40$	$75.31\pm 10.50$	$1.83\pm 0.30$

Appendix

Table 5: Navigation ablations. Metrics and evaluation resets match Table 2 ; Ours is copied unchanged. Entries are mean

\pm

sample SD across six training seeds. Goal, constraint-satisfaction, and safe-goal rates are percentages; final-20 distance is in meters.

Method	Completed trips	Network vehicle-hours	Core vehicle-hours	Episode-max core cost	Time above core limit
Uniform attribution Max-incremental cost	$0.786\pm 0.018$	$1.021\pm 0.002$	$0.079\pm 0.017$	$0.00031\pm 0.00058$	$0.005\pm 0.009$
Uniform attribution Max-incremental cost + Adam	$1.002\pm 0.010$	$1.000\pm 0.001$	$1.011\pm 0.076$	$1.002\pm 0.111$	$1.000\pm 0.010$
Uniform attribution Step cost	$0.388\pm 0.235$	$1.063\pm 0.025$	$0.065\pm 0.038$	$0.00000049\pm 0.00000076$	$0.000041\pm 0.000063$
Uniform attribution Step cost + Adam	$0.881\pm 0.033$	$1.011\pm 0.003$	$0.837\pm 0.050$	$0.788\pm 0.082$	$0.985\pm 0.012$
Ours	$0.714\pm 0.066$	$1.029\pm 0.007$	$0.088\pm 0.018$	$0.000061\pm 0.000053$	$0.003\pm 0.004$

Appendix

Table 6: Traffic ablations. Values use the same fixed, column-wise MAPPO mean as Table 3 ; Ours is copied unchanged. Mean

\pm

sample SD across six training seeds and 1,000 common resets per policy. Higher completed trips and lower values in all other columns are preferable.

Constrained MARL

See all 13 papers

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control
Jun 22, 2026Zihao Guo, Jianing Zhao, Ling Li +3Robotic Control Multi-Agent Coordination
1King’s College London · University of California, Berkeley · 3The Alan Turing Institute
Contract-Based Compositional Shielding for Safe Multi-Agent Reinforcement Learning
Jun 12, 2026Omar Adalat, Edwin Hamel-De le Court, Francesco BelardinelliMulti-Agent Coordination Formal Verification
Imperial College London, London SW7 2AZ, UK · University of Manchester, Manchester M13 9PL, UK
Who Bears the Burden? Learning Responsibility for Shared Constraints in Multi-Agent Reinforcement Learning
Oct 5, 2026Xiaoyang Cao, Jingqi Li, Zhe Fu +1Game Theory Constrained MARL
Massachusetts Institute of Technology · The University of Texas at Austin · Stanford University +1

Jun 22, 2026J/K move · Enter open · S save

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Zihao Guo, Jianing Zhao, Ling Li, Hao Liang+2

1King’s College London · University of California, Berkeley · 3The Alan Turing Institute

Robotic Control Multi-Agent Coordination Hierarchical RL Cooperative MARL Multi-Agent Reinforcement Learning Constrained MARL

Open paper AskPDF arXiv

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.

Abstract

Constrained MARL

Explore similar work

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Contract-Based Compositional Shielding for Safe Multi-Agent Reinforcement Learning

Who Bears the Burden? Learning Responsibility for Shared Constraints in Multi-Agent Reinforcement Learning