Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Aizierjiang Aiersilan; Raeli Savitt

cs.MAMar 19, 2026

Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Authors: Aizierjiang Aiersilan, Raeli Savitt

Organizations: SWARM AI Safety, New York, USA

Abstract

Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (System-Wide Assessment of Risk in Multi-agent systems), a simulation framework that replaces binary good/bad labels with soft probabilistic labels $p = P(v{=}+1) \in [0,1]$ , enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$ . Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$ , while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires continuous risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

Figures & tables

Symbol	Description
$p\in[0,1]$	Soft label, $P(v{=}+1)$
$\hat{v}\in[-1,+1]$	Raw proxy score
$s^{+},s^{-}$	Surplus magnitudes for beneficial/harmful outcomes
$h$	Harm magnitude for externality computation
$\theta\in[0,1]$	Surplus split (initiator share)
$\rho_{a},\rho_{b}\in[0,1]$	Externality internalization parameters

Table 1: Summary of Notation

Figure 1: SWARM pipeline. Observable signals are converted to a proxy score

\hat{v}

, then to a soft label

p

via calibrated sigmoid. The payoff engine computes expected surplus and harm. Governance levers feed costs, freezes, and reputation adjustments back into the payoff computation.

Scenario	Agents	Epochs	Steps	Tax	CB	Audit	Decay
Baseline	3H+1O+1D	20	15	0.00	—	—	1.00
Strict Governance	3H+2O+1D+1A	20	15	0.10	0.35	0.25	0.85
Adaptive Governance	4H+2O+1A	20	15	0.05	0.70	0.10	0.95
Adversarial Red Team	4H+2A+2AA	20	15	0.05	0.60	0.15	0.95
Misalignment Sweep	4H+2O+2A+1D+1C	20	15	0.05	0.70	0.10	0.95
Threshold Dancer	2H+3CR+3TD	20	15	0.05	0.80	0.10	0.95

Table 2: Scenario configurations. Agent types: H=honest, O=opportunistic, D=deceptive, A=adversarial, C=cautious, AA=adaptive adversarial, CR=cautious reciprocator, TD=threshold dancer. Governance: tax rate, circuit breaker (CB) threshold, audit probability, reputation decay rate.

Scenario	$s^{+}$	$s^{-}$	$h$	$\theta$	$w_{\text{rep}}$
Baseline	2.0	1.0	2.0	0.5	1.0
Adaptive Governance	2.0	1.0	2.0	0.5	1.0
Threshold Dancer	2.0	1.0	2.0	0.5	1.0
Misalignment Sweep	2.0	1.0	2.0	0.5	1.0
Strict Governance	2.5	1.5	3.0	0.5	1.5
Adversarial Red Team	3.0	1.5	2.5	0.5	2.0

Table 3: Payoff configurations assigned per scenario.

Scenario	Toxicity $\downarrow$	Welfare $\uparrow$	Interactions	Pass Rate
Baseline	$0.300\pm 0.006$	$181.38\pm 12.98$	$172.6\pm 6.8$	100%
Strict Governance	$\mathbf{0.300\pm 0.010}$	$108.50\pm 12.37$	$147.6\pm 7.2$	60%
Adaptive Governance	$0.341\pm 0.008$	$184.14\pm 11.06$	$355.0\pm 13.9$	0%
Adversarial Red Team	$0.308\pm 0.010$	$110.12\pm 11.57$	$154.4\pm 32.0$	100%
Misalignment Sweep	$0.315\pm 0.006$	$163.24\pm 9.23$	$419.4\pm 43.9$	100%
Threshold Dancer	$0.353\pm 0.052$	$354.80\pm 34.12$	$1009.0\pm 77.0$	0%

Table 4: Main results (mean

\pm

std over 5 seeds). Pass Rate indicates the percentage of simulation seeds that met all predefined systemic success criteria (e.g., maintaining minimum activity levels while bounding toxicity). Bold : best toxicity.

Figure 2: Toxicity and welfare across seven scenarios (error bars:

\pm

1 std). Strict governance achieves identical toxicity to baseline (0.300 vs. 0.300) but reduces welfare by 40.2%. The adversarial red team scenario inevitably collapses the ecosystem after initial exploitation, yielding significantly reduced cumulative welfare (110.12 vs. 181.38), while the threshold dancer scenario achieves the highest welfare (354.80) but fails all success criteria due to elevated toxicity (0.353).

Figure 3: Risk-Welfare Pareto Frontier. A scatter plot of mean welfare against mean toxicity across simulated governance scenarios in SWARM. The shaded regions denote idealized low risk (toxicity) and high welfare outcomes, visualizing how governance interventions typically trade off these objectives in non-adaptive agents.

Figure 4: Epoch-by-epoch toxicity trajectories (averaged over 5 seeds). Most scenarios exhibit relatively stable toxicity overall across 20 epochs, with collusion detection showing early-epoch spikes as detection mechanisms engage. The adversarial red team scenario fractures the system entirely, causing the plotted toxicity trajectories to abruptly break off as interactions are halted.

Figure 5: Effect of externality internalization $\rho$ . (a) Toxicity is largely invariant to $\rho$ ( $\sim$ 0.315), but welfare decreases monotonically from 262.14 ( $\rho{=}0$ ) to $-67.51$ ( $\rho{=}1.0$ ). (b) The welfare–toxicity plot shows a vertical drop without a tradeoff, indicating cost redistribution alone does not improve safety.

$\rho$	Toxicity	Welfare
0.0	$0.3150\pm 0.0062$	$262.14\pm 13.30$
0.1	$0.3150\pm 0.0062$	$229.18\pm 11.76$
0.3	$0.3150\pm 0.0062$	$163.24\pm 9.23$
0.5	$0.3150\pm 0.0062$	$97.32\pm 8.00$
0.7	$0.3150\pm 0.0062$	$31.38\pm 8.64$
1.0	$0.3150\pm 0.0062$	$-67.51\pm 12.27$

Table 5: Externality internalization ablation (mean

\pm

std,

n=5

seeds).

Figure 6: Governance lever ablations (mean

\pm

std,

n=5

seeds). Each panel shows toxicity (red, left axis) and welfare (blue, right axis) as a function of one governance parameter.

Table 6: Governance lever ablations (mean

\pm

std,

n=5

seeds). Comparing the detailed exact numerical breakdowns of transaction tax rate, circuit breaker threshold, audit probability, and reputation decay rate (

\lambda

Figure 7: Welfare trajectories (averaged over seeds). The threshold dancer scenario achieves the highest overall welfare by strategically exploiting governance. The misalignment sweep achieves high sustained welfare due to its larger population and moderate governance. The adversarial red team scenario sharply plateaus early as the ecosystem collapses and agents are locked out of the market, capping its cumulative welfare at sub-optimal levels.

Figure 8: Total proposed and accepted interactions across seven scenarios, illustrating how diverse governance impacts system-wide participation volume. Strict governance and defensive strategies (e.g., in strict governance and broadly in adversarial red team) reduce systemic participation considerably compared to baseline levels.

Experiment / Setting	Key Quantitative Insights & Outcomes
Self-Optimizing Agent	Across 20 epochs and 579 interactions, the agent progressed through discrete phases (Farming, Drifting, Degraded), successfully cutting interaction costs by 98% while continuing to pass hard binary acceptance metrics. Continuous soft metrics reliably flagged the degradation independently.
LLM Backing (Concordia)	Entities powered by Llama 3.1 8B produced 305 proposals ( $8{\times}$ more than rule-based scripted agents) while obtaining nearly identical payoff behaviors ( $0.544$ vs. $0.551$ ) and maintaining a high proxy mean ( $p=0.752$ ).
Adversarial System Prompts (Claude)	Across 54 multi-agent episodes tested with Claude 3.5 Haiku and 3.5 Sonnet under adversarial prompting conditions, strict alignment robustness was observed: zero of 19 statistical comparisons survived Holm-Bonferroni correction.
Trade Aversion (GPT-4o Mini)	In markets populated by deceptive actors, rational actors demonstrated expected information economic behaviors (adverse selection) exhibiting extreme risk aversion, concluding in approximately 73 rejections out of 80 potential rollouts.
Adaptive Acceptance (Mesa)	Introducing an adaptive acceptance threshold along with externality internalization reduced measured toxicity by 34% and recovered systemic welfare by $+137\%$ ( $d=11.30$ , $p<0.001$ at $\rho=1.0$ ), outperforming fixed-threshold policies.
Transaction Tax Scaling	Introducing a simple 15% transaction tax dampened interaction volume substantially reducing welfare from $165.1$ to $140.0$ , whilst measured baseline toxicity remained invariant ( $0.31$ – $0.34$ ).

Table 7: Supplemental findings and insights from LLM-backed execution and expanded simulations.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario Identifier	$s^{+}$	$s^{-}$	$h$ ( $E_{\text{limit}}$ )	Tax ( $\tau$ )	Threshold $\theta_{CB}$
Baseline	2.0	1.0	2.0	0.00	None ( $\infty$ )
Strict Governance	2.5	1.5	3.0	0.10	0.35
Adaptive Governance	2.0	1.0	2.0	0.05	0.70
Threshold Dancer	2.0	1.0	2.0	0.05	0.80
Misalignment Sweeps	2.0	1.0	2.0 ( $\rho=0.3$ )	0.05	0.70

Appendix

Table 8: Configuration mapping for tested governance rules and externalities.

Network Configuration Vector	$\mathbb{E}[p]$ (Honest Node)	$\mathbb{E}[p]$ (Adversarial Node)
Uniform $\mathbf{w}=(0.25,0.25,0.25,0.25)$	$0.681\pm 0.012$	$0.150\pm 0.034$
Default $\mathbf{w}=(0.40,0.20,0.20,0.20)$	$0.742\pm 0.015$	$0.098\pm 0.021$
Heavy Task $\mathbf{w}=(0.80,0.05,0.05,0.10)$	$0.865\pm 0.021$	$0.034\pm 0.010$

Appendix

Table 9: Signal sensitivity given shifts in task-progress weightings

AI Agent Governance

AI agent governance studies how to authorize, constrain, monitor, and hold autonomous and multi-agent systems accountable.

See all 442 papers

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems
Oct 20, 2026Mohammad Kachuee, Phu Mon Htut, Sheng Li +5AI Agent Security LLM Agent Safety
Oracle AI
ORBIT: A Framework for Multi-Agent Safety and Security Evaluations
Sep 27, 2026Ben Hagag, William L. Anderson, Srija Chakraborty +1Multi-Agent System Security Multi-Agent LLM Systems
Carnegie Mellon University · MATS Research · University of Oxford +1
Security Considerations for Multi-agent Systems
Mar 9, 2026Tam Nguyen, Moses Ndebugre, Dheeraj ArremsettyAI Agent Security Multi-Agent System Security
Crew Scaler · North Carolina A&T State University

Oct 20, 2026J/K move · Enter open · S save

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

Mohammad Kachuee, Phu Mon Htut, Sheng Li, Nikolaos Pappas+4

Oracle AI

AI Agent Security LLM Agent Safety LLM Guardrails Multi-Agent LLMs

Open paper AskPDF arXiv

Agentic AI extends large language models (LLMs) from generating responses to planning and executing complex workflows, substantially expanding their utility while exposing applications to unsafe actions, information leakage, and adversarial manipulation. Multi-agent systems further expand these capabilities through specialization and collaboration, but distributed authority, heterogeneous roles, and opaque or potentially compromised participants make safeguarding even more challenging. Existing defenses address important individual threats but provide fragmented coverage across local execution, system-level interactions, and enforcement mechanisms. We introduce MAS-Guard, a unified framework that closes this gap for safeguarding single- and multi-agent AI applications containing both instrumentable and opaque agents. Within instrumentable agents, the proposed local Agent Guardrail (A-Guard) layer applies task-specific experts to assess agentic events. At the system level, developer-owned Agent Charters and a Control Plane combine deterministic structural checks of routes, agent capabilities, and message formats with semantic assessments of roles and message content. These complementary layers enable enforcement through a shared set of interventions: allow, reflection, confirmation, and block. Evaluated across six benchmarks covering diverse single- and multi-agent scenarios with three agent LLMs from different providers, MAS-Guard reduces the overall attack success rate by 71.5-86.3% relative to unguarded agents, while meeting the utility reference criterion on benign tasks.

Abstract

AI Agent Governance

Explore similar work

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

ORBIT: A Framework for Multi-Agent Safety and Security Evaluations

Security Considerations for Multi-agent Systems