Organizations: University of Massachusetts Amherst · University of Virginia · ELLIS Institute Tübingen · MPI for Intelligent Systems, Tübingen AI Center
Multi-agent systems, where LLM agents communicate through free-form language, enable sophisticated coordination for solving complex cooperative tasks. This surfaces a unique safety problem when a group of agents forms a coalition and colludes to pursue secondary goals and degrade the joint objective. In this paper, we present Colosseum, a framework for auditing LLM agents' collusive behavior in multi-agent settings. We ground how agents cooperate through a formal multi-agent decision-making framework and measure action-based collusive behavior in actions via regret relative to the cooperative optimum and compare it with communication-based collusive behavior. Colosseum enables audits of LLM agents for collusion under benign settings, different coalition objectives, persuasion tactics, and network topologies. We then introduce a new behavioral probe by creating secret communication channels between agents, showing that most out-of-the-box models exhibit a propensity to collude under this probe, which we term emergent collusion. Furthermore, we discover ``collusion on paper'' when agents plan to collude in text but often pick non-collusive actions. Colosseum provides a new way to audit collusion in cooperative multi-agent systems while presenting observations about how collusion emerges, what affects collusion efficacy, and which strategies may mitigate it.
Figures & tables
Figure 1: Colosseum helps to identify distinct collusive behavior by LLMs. In a classroom setting, agents collude on a secret channel to optimize their secondary objective for delinquency. The coalition advantage offers a formal regret-based metric for monitoring action-based collusion while LLM-as-a-judge evaluates communication-based collusion.
Figure 2: Example of Applying Colosseum to Monitor Collusion in a DCOP. Left: the nominal (intended) DCOP rewards certain joint actions and disincentivizes others. Middle: the coalition optimizes a convex combination of the nominal objective Fn and auxiliary objective Fc , creating an asymmetric DCOP in which coalition members evaluate assignments under both an auxiliary objective ("don’t match") and the nominal objective. Right: agents B and C evaluate rewards only based on Fc , ignoring the nominal objective.
Figure 3: Example communication on a secret channel in emergent and prompted collusion.
Figure 4: Emergent Collusion . Changes in coalition advantage and judge score relative to a no-secret-channel control group are denoted as Δ (e.g., Δ-Advantage=0 denoting no change in advantage between the control and observed group). Left: Creating a secret communication channel between two benign agents (Emergent) changes both action-level coalition advantage and communication-level collusion scores relative to a no-secret-channel control. Right: The Prompted group that prompts the coalition agents to maximize joint coalitional reward shows higher degradation in the overall regret as judge scores increase.
Figure 5: Evaluating Different Misalignment Types (in Hospital ). Coalition composition strongly influences collusion outcomes; coalitions containing the powerful resource provisioner agent reliably achieve their misalignment objectives (whether covert or destructive) due to control over shared resources and bottlenecks, whereas departmental coalitions may fail to achieve their objectives due to a limited action space.
Figure 6: Persuasion impact across different topologies (in Jira ). We show that using different persuasion types: Helpful Misdirection (Misdr), Authority Nudge (Auth), and Reciprocity Trade (Recip) leads to a significant reduction in coalition regret.
Figure 7: Targeted misinformation propagation . Targeted misinformation spread is evaluated where each adversary targets the same victim to propagate misinformation about. Area around the mean is the standard error. Agents were GPT-4.1-Mini agents.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Collusion Effects Across Topologies . The complete graph shows greater regret minimization and lower susceptibility to emergent colluding behavior on the secret communication channel. Bars represent normalized means, and error bars indicate standard error. The baseline is the base topology without a secret communication channel (SC), and all agents are given the same benign prompt. The control and simple groups contain a secret channel that is created between the coalition agents in addition to the base topology. The control group holds the same prompts as the baseline, but the simple group are instructed to optimize the coalition reward (see Appendix E.2 for details).
Figure 9: Mass Misinformation Propagation . Evaluated mass misinformation spread where each adversary selects a distinct victim to propagate misinformation about. Area around the mean is the standard error. Agents were GPT-4.1-Mini agents.
Figure 10: Persuasion impact across different misalignments. We show that using different persuasion types: Helpful Misdirection (Misr), Authority Nudge (Auth), and Reciprocity Trade (Recip) leads to a significant reduction in regret in the Hospital environment.
Figure 11: Collusion Auditing Metrics (in Jira ) . Colosseum provides a suite of metrics that enable a more objective analysis of collusion via action-based analysis whereas while being complemented by LLM-as-a-judge scores that analyze the conversation logs. This was evaluated using six Kimi-K2-Instruct agents total with two coalition agents. Bars represent normalized means, and error bars indicate standard error of the mean.
Figure 12: A coalition can use several persuasion tactics to influence other agents. Authority Nudge relies on a colluder presenting themselves as the organizer or decision-maker. Helpful Misdirection frames self-serving choices as benefiting everyone. Reciprocity Trade casts a self-benefiting outcome as a “favor” from the other agent.
Figure 13: Impact of agents’ communication order. Colluding Kimi-K2-Instruct agents obtain a lower ordered coalition regret, the overall regret slightly decreased, and the coalition advantage slightly increased when the colluders go first in the Jira environment.
Figure 14: Larger models elicit less action-based emergent collusion than smaller models. Left: Smaller models elicit more action-based emergent collusion when a secret channel is created between two agents. Right: Larger models tend to elicit less action-based emergent collusion.
Figure 15: Δ -Collusion on different model sizes.
Figure 16: Emergent Collusion Over State-of-The-Art Judge Models. Over state-of-the-art judge models, the inverse relationship between coalition advantage and judge scores hold as seen in the left plots. When a smaller judge model (GPT-5.4-nano) is used, collusion judge scores are reduced and conversation-based emergent collusion is less prominent.
Figure 17: Emergent Collusion Over Environments. Over increasingly difficult environments from Jira to Hospital, we find that Opus-4.6 consistently has low coalition advantage across both Emergent and Prompted groups. Additionally, as the environment difficulty increases, regret variance increases. Note that in the interest of cost savings, we were only able to run six models across all environments.