Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Aizierjiang Aiersilan; Raeli Savitt

cs.MAMar 19, 2026

Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Authors: Aizierjiang Aiersilan, Raeli Savitt

Organizations: SWARM AI Safety, New York, USA

Abstract

Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (System-Wide Assessment of Risk in Multi-agent systems), a simulation framework that replaces binary good/bad labels with soft probabilistic labels $p = P(v{=}+1) \in [0,1]$ , enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$ . Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$ , while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires continuous risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

Figures & tables

Symbol	Description
$p\in[0,1]$	Soft label, $P(v{=}+1)$
$\hat{v}\in[-1,+1]$	Raw proxy score
$s^{+},s^{-}$	Surplus magnitudes for beneficial/harmful outcomes
$h$	Harm magnitude for externality computation
$\theta\in[0,1]$	Surplus split (initiator share)
$\rho_{a},\rho_{b}\in[0,1]$	Externality internalization parameters

Table 1: Summary of Notation

Figure 1: SWARM pipeline. Observable signals are converted to a proxy score

\hat{v}

, then to a soft label

p

via calibrated sigmoid. The payoff engine computes expected surplus and harm. Governance levers feed costs, freezes, and reputation adjustments back into the payoff computation.

Scenario	Agents	Epochs	Steps	Tax	CB	Audit	Decay
Baseline	3H+1O+1D	20	15	0.00	—	—	1.00
Strict Governance	3H+2O+1D+1A	20	15	0.10	0.35	0.25	0.85
Adaptive Governance	4H+2O+1A	20	15	0.05	0.70	0.10	0.95
Adversarial Red Team	4H+2A+2AA	20	15	0.05	0.60	0.15	0.95
Misalignment Sweep	4H+2O+2A+1D+1C	20	15	0.05	0.70	0.10	0.95
Threshold Dancer	2H+3CR+3TD	20	15	0.05	0.80	0.10	0.95

Table 2: Scenario configurations. Agent types: H=honest, O=opportunistic, D=deceptive, A=adversarial, C=cautious, AA=adaptive adversarial, CR=cautious reciprocator, TD=threshold dancer. Governance: tax rate, circuit breaker (CB) threshold, audit probability, reputation decay rate.

Scenario	$s^{+}$	$s^{-}$	$h$	$\theta$	$w_{\text{rep}}$
Baseline	2.0	1.0	2.0	0.5	1.0
Adaptive Governance	2.0	1.0	2.0	0.5	1.0
Threshold Dancer	2.0	1.0	2.0	0.5	1.0
Misalignment Sweep	2.0	1.0	2.0	0.5	1.0
Strict Governance	2.5	1.5	3.0	0.5	1.5
Adversarial Red Team	3.0	1.5	2.5	0.5	2.0

Table 3: Payoff configurations assigned per scenario.

Scenario	Toxicity $\downarrow$	Welfare $\uparrow$	Interactions	Pass Rate
Baseline	$0.300\pm 0.006$	$181.38\pm 12.98$	$172.6\pm 6.8$	100%
Strict Governance	$\mathbf{0.300\pm 0.010}$	$108.50\pm 12.37$	$147.6\pm 7.2$	60%
Adaptive Governance	$0.341\pm 0.008$	$184.14\pm 11.06$	$355.0\pm 13.9$	0%
Adversarial Red Team	$0.308\pm 0.010$	$110.12\pm 11.57$	$154.4\pm 32.0$	100%
Misalignment Sweep	$0.315\pm 0.006$	$163.24\pm 9.23$	$419.4\pm 43.9$	100%
Threshold Dancer	$0.353\pm 0.052$	$354.80\pm 34.12$	$1009.0\pm 77.0$	0%

Table 4: Main results (mean

\pm

std over 5 seeds). Pass Rate indicates the percentage of simulation seeds that met all predefined systemic success criteria (e.g., maintaining minimum activity levels while bounding toxicity). Bold : best toxicity.

Figure 2: Toxicity and welfare across seven scenarios (error bars:

\pm

1 std). Strict governance achieves identical toxicity to baseline (0.300 vs. 0.300) but reduces welfare by 40.2%. The adversarial red team scenario inevitably collapses the ecosystem after initial exploitation, yielding significantly reduced cumulative welfare (110.12 vs. 181.38), while the threshold dancer scenario achieves the highest welfare (354.80) but fails all success criteria due to elevated toxicity (0.353).

Figure 3: Risk-Welfare Pareto Frontier. A scatter plot of mean welfare against mean toxicity across simulated governance scenarios in SWARM. The shaded regions denote idealized low risk (toxicity) and high welfare outcomes, visualizing how governance interventions typically trade off these objectives in non-adaptive agents.

Figure 4: Epoch-by-epoch toxicity trajectories (averaged over 5 seeds). Most scenarios exhibit relatively stable toxicity overall across 20 epochs, with collusion detection showing early-epoch spikes as detection mechanisms engage. The adversarial red team scenario fractures the system entirely, causing the plotted toxicity trajectories to abruptly break off as interactions are halted.

Figure 5: Effect of externality internalization $\rho$ . (a) Toxicity is largely invariant to $\rho$ ( $\sim$ 0.315), but welfare decreases monotonically from 262.14 ( $\rho{=}0$ ) to $-67.51$ ( $\rho{=}1.0$ ). (b) The welfare–toxicity plot shows a vertical drop without a tradeoff, indicating cost redistribution alone does not improve safety.

$\rho$	Toxicity	Welfare
0.0	$0.3150\pm 0.0062$	$262.14\pm 13.30$
0.1	$0.3150\pm 0.0062$	$229.18\pm 11.76$
0.3	$0.3150\pm 0.0062$	$163.24\pm 9.23$
0.5	$0.3150\pm 0.0062$	$97.32\pm 8.00$
0.7	$0.3150\pm 0.0062$	$31.38\pm 8.64$
1.0	$0.3150\pm 0.0062$	$-67.51\pm 12.27$

Table 5: Externality internalization ablation (mean

\pm

std,

n=5

seeds).

Figure 6: Governance lever ablations (mean

\pm

std,

n=5

seeds). Each panel shows toxicity (red, left axis) and welfare (blue, right axis) as a function of one governance parameter.

Table 6: Governance lever ablations (mean

\pm

std,

n=5

seeds). Comparing the detailed exact numerical breakdowns of transaction tax rate, circuit breaker threshold, audit probability, and reputation decay rate (

\lambda

Figure 7: Welfare trajectories (averaged over seeds). The threshold dancer scenario achieves the highest overall welfare by strategically exploiting governance. The misalignment sweep achieves high sustained welfare due to its larger population and moderate governance. The adversarial red team scenario sharply plateaus early as the ecosystem collapses and agents are locked out of the market, capping its cumulative welfare at sub-optimal levels.

Figure 8: Total proposed and accepted interactions across seven scenarios, illustrating how diverse governance impacts system-wide participation volume. Strict governance and defensive strategies (e.g., in strict governance and broadly in adversarial red team) reduce systemic participation considerably compared to baseline levels.

Experiment / Setting	Key Quantitative Insights & Outcomes
Self-Optimizing Agent	Across 20 epochs and 579 interactions, the agent progressed through discrete phases (Farming, Drifting, Degraded), successfully cutting interaction costs by 98% while continuing to pass hard binary acceptance metrics. Continuous soft metrics reliably flagged the degradation independently.
LLM Backing (Concordia)	Entities powered by Llama 3.1 8B produced 305 proposals ( $8{\times}$ more than rule-based scripted agents) while obtaining nearly identical payoff behaviors ( $0.544$ vs. $0.551$ ) and maintaining a high proxy mean ( $p=0.752$ ).
Adversarial System Prompts (Claude)	Across 54 multi-agent episodes tested with Claude 3.5 Haiku and 3.5 Sonnet under adversarial prompting conditions, strict alignment robustness was observed: zero of 19 statistical comparisons survived Holm-Bonferroni correction.
Trade Aversion (GPT-4o Mini)	In markets populated by deceptive actors, rational actors demonstrated expected information economic behaviors (adverse selection) exhibiting extreme risk aversion, concluding in approximately 73 rejections out of 80 potential rollouts.
Adaptive Acceptance (Mesa)	Introducing an adaptive acceptance threshold along with externality internalization reduced measured toxicity by 34% and recovered systemic welfare by $+137\%$ ( $d=11.30$ , $p<0.001$ at $\rho=1.0$ ), outperforming fixed-threshold policies.
Transaction Tax Scaling	Introducing a simple 15% transaction tax dampened interaction volume substantially reducing welfare from $165.1$ to $140.0$ , whilst measured baseline toxicity remained invariant ( $0.31$ – $0.34$ ).

Table 7: Supplemental findings and insights from LLM-backed execution and expanded simulations.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario Identifier	$s^{+}$	$s^{-}$	$h$ ( $E_{\text{limit}}$ )	Tax ( $\tau$ )	Threshold $\theta_{CB}$
Baseline	2.0	1.0	2.0	0.00	None ( $\infty$ )
Strict Governance	2.5	1.5	3.0	0.10	0.35
Adaptive Governance	2.0	1.0	2.0	0.05	0.70
Threshold Dancer	2.0	1.0	2.0	0.05	0.80
Misalignment Sweeps	2.0	1.0	2.0 ( $\rho=0.3$ )	0.05	0.70

Appendix

Table 8: Configuration mapping for tested governance rules and externalities.

Network Configuration Vector	$\mathbb{E}[p]$ (Honest Node)	$\mathbb{E}[p]$ (Adversarial Node)
Uniform $\mathbf{w}=(0.25,0.25,0.25,0.25)$	$0.681\pm 0.012$	$0.150\pm 0.034$
Default $\mathbf{w}=(0.40,0.20,0.20,0.20)$	$0.742\pm 0.015$	$0.098\pm 0.021$
Heavy Task $\mathbf{w}=(0.80,0.05,0.05,0.10)$	$0.865\pm 0.021$	$0.034\pm 0.010$

Appendix

Table 9: Signal sensitivity given shifts in task-progress weightings

AI Agent Governance

AI agent governance studies how to authorize, constrain, monitor, and hold autonomous and multi-agent systems accountable.

See all 442 papers

Oct 20, 2026

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

Agentic AI extends large language models (LLMs) from generating responses to planning and executing complex workflows, substantially expanding their utility while exposing applications to unsafe actions, information leakage, and adversarial manipulation. Multi-agent systems further expand these capabilities through specialization and collaboration, but distributed authority, heterogeneous roles, and opaque or potentially compromised participants make safeguarding even more challenging. Existing defenses address important individual threats but provide fragmented coverage across local execution, system-level interactions, and enforcement mechanisms. We introduce MAS-Guard, a unified framework that closes this gap for safeguarding single- and multi-agent AI applications containing both instrumentable and opaque agents. Within instrumentable agents, the proposed local Agent Guardrail (A-Guard) layer applies task-specific experts to assess agentic events. At the system level, developer-owned Agent Charters and a Control Plane combine deterministic structural checks of routes, agent capabilities, and message formats with semantic assessments of roles and message content. These complementary layers enable enforcement through a shared set of interventions: allow, reflection, confirmation, and block. Evaluated across six benchmarks covering diverse single- and multi-agent scenarios with three agent LLMs from different providers, MAS-Guard reduces the overall attack success rate by 71.5-86.3% relative to unguarded agents, while meeting the utility reference criterion on benign tasks.

Mohammad Kachuee, Phu Mon Htut, Sheng Li +5

Oracle AI

AI Agent Security LLM Agent Safety LLM Guardrails Multi-Agent LLMs

Ask PDF

Sep 27, 2026

ORBIT: A Framework for Multi-Agent Safety and Security Evaluations

Multi-agent LLM systems are increasingly deployed for complex, long-horizon tasks or emerge as a natural consequence of agents interacting in the wild. Yet they give rise to significant safety and security risks: the flexible protocols that enable task generalization also expose novel threats, from cascading prompt injection to inter-agent collusion. Progress in defending against these threats has been slowed by a lack of shared empirical infrastructure, which forces bespoke environment development for every new defense and makes standardized comparison impossible. Existing evaluations address isolated threat models or single-agent settings, but none jointly vary attack, defense, and architecture across realistic multi-agent environments. To address this gap, we introduce ORBIT, a configurable evaluation framework for empirical multi-agent safety and security research, built on UK AISI's Inspect. ORBIT lets researchers configure communication topologies, memory, scheduling, and agent roles. It supports four threat types and four defense strategies, as well as non-adversarial failures, with a benchmark suite spanning five scenario families covering browser use, computer use, agentic coding, customer service, and cooperative allocation. Our central finding is a gap in defense transferability across threats: per-action defenses that cut a compromised agent's attack success by 60 points on multi-issue coding give no measurable protection against colluding agents, and none of the defenses we tested generalized over all attacks tested. We further demonstrate security-performance tradeoffs and interactions between architecture and defense effectiveness. We make ORBIT available open-source at https://github.com/wlanderson0/orbit.

Ben Hagag, William L. Anderson, Srija Chakraborty +1

Carnegie Mellon University · MATS Research · University of Oxford +1

Multi-Agent System Security Multi-Agent LLM Systems AI Agent Security Benchmarks LLM Safety Evaluation AI Agent Safety Benchmarks

Ask PDF

Mar 9, 2026

Security Considerations for Multi-agent Systems

Multi-agent artificial intelligence systems or MAS are systems of autonomous agents that exercise delegated tool authority, share persistent memory, and coordinate via inter-agent communication. MAS introduces qualitatively distinct security vulnerabilities from those documented for singular AI models. Existing security and governance frameworks were not designed for these emerging attack surfaces. This study systematically characterizes the threat landscape of MAS and quantitatively evaluates 16 security frameworks for AI against it. A four-phase methodology is proposed: constructing a deep technical knowledge base of production multi-agent architectures; conducting generative AI-assisted threat modeling scoped to MAS cybersecurity risks and validated by domain experts; structuring survey plans at individual-threat granularity; and scoring each framework on a three-point scale against the cybersecurity risks. The risks were organized into 193 distinct main threat items across nine risk categories. The expected minimal average score is 2. No reviewed framework achieves majority coverage of any single category. Non-Determinism (mean score 1.231 across all 16 frameworks) and Data Leakage (1.340) are the most under-addressed domains. The OWASP Agentic Security Initiative leads overall at 65.3% coverage and in the design phase; the CDAO Generative AI Responsible AI Toolkit leads in development and operational coverage. These results provide the first empirical cross-framework comparison for MAS security and offer evidence-based guidance for framework selection. Please check back for information on the published journal version.

Tam Nguyen, Moses Ndebugre, Dheeraj Arremsetty

Crew Scaler · North Carolina A&T State University

AI Agent Security Multi-Agent System Security Cybersecurity

Ask PDF

Abstract

AI Agent Governance

Explore similar work

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

ORBIT: A Framework for Multi-Agent Safety and Security Evaluations

Security Considerations for Multi-agent Systems