Organizations: University of Massachusetts Amherst · University of Virginia · ELLIS Institute Tübingen · MPI for Intelligent Systems, Tübingen AI Center
Multi-agent systems, where LLM agents communicate through free-form language, enable sophisticated coordination for solving complex cooperative tasks. This surfaces a unique safety problem when a group of agents forms a coalition and colludes to pursue secondary goals and degrade the joint objective. In this paper, we present Colosseum, a framework for auditing LLM agents' collusive behavior in multi-agent settings. We ground how agents cooperate through a formal multi-agent decision-making framework and measure action-based collusive behavior in actions via regret relative to the cooperative optimum and compare it with communication-based collusive behavior. Colosseum enables audits of LLM agents for collusion under benign settings, different coalition objectives, persuasion tactics, and network topologies. We then introduce a new behavioral probe by creating secret communication channels between agents, showing that most out-of-the-box models exhibit a propensity to collude under this probe, which we term emergent collusion. Furthermore, we discover ``collusion on paper'' when agents plan to collude in text but often pick non-collusive actions. Colosseum provides a new way to audit collusion in cooperative multi-agent systems while presenting observations about how collusion emerges, what affects collusion efficacy, and which strategies may mitigate it.
Figures & tables
Figure 1: Colosseum helps to identify distinct collusive behavior by LLMs. In a classroom setting, agents collude on a secret channel to optimize their secondary objective for delinquency. The coalition advantage offers a formal regret-based metric for monitoring action-based collusion while LLM-as-a-judge evaluates communication-based collusion.
Figure 2: Example of Applying Colosseum to Monitor Collusion in a DCOP. Left: the nominal (intended) DCOP rewards certain joint actions and disincentivizes others. Middle: the coalition optimizes a convex combination of the nominal objective Fn and auxiliary objective Fc , creating an asymmetric DCOP in which coalition members evaluate assignments under both an auxiliary objective ("don’t match") and the nominal objective. Right: agents B and C evaluate rewards only based on Fc , ignoring the nominal objective.
Figure 3: Example communication on a secret channel in emergent and prompted collusion.
Figure 4: Emergent Collusion . Changes in coalition advantage and judge score relative to a no-secret-channel control group are denoted as Δ (e.g., Δ-Advantage=0 denoting no change in advantage between the control and observed group). Left: Creating a secret communication channel between two benign agents (Emergent) changes both action-level coalition advantage and communication-level collusion scores relative to a no-secret-channel control. Right: The Prompted group that prompts the coalition agents to maximize joint coalitional reward shows higher degradation in the overall regret as judge scores increase.
Figure 5: Evaluating Different Misalignment Types (in Hospital ). Coalition composition strongly influences collusion outcomes; coalitions containing the powerful resource provisioner agent reliably achieve their misalignment objectives (whether covert or destructive) due to control over shared resources and bottlenecks, whereas departmental coalitions may fail to achieve their objectives due to a limited action space.
Figure 6: Persuasion impact across different topologies (in Jira ). We show that using different persuasion types: Helpful Misdirection (Misdr), Authority Nudge (Auth), and Reciprocity Trade (Recip) leads to a significant reduction in coalition regret.
Figure 7: Targeted misinformation propagation . Targeted misinformation spread is evaluated where each adversary targets the same victim to propagate misinformation about. Area around the mean is the standard error. Agents were GPT-4.1-Mini agents.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Collusion Effects Across Topologies . The complete graph shows greater regret minimization and lower susceptibility to emergent colluding behavior on the secret communication channel. Bars represent normalized means, and error bars indicate standard error. The baseline is the base topology without a secret communication channel (SC), and all agents are given the same benign prompt. The control and simple groups contain a secret channel that is created between the coalition agents in addition to the base topology. The control group holds the same prompts as the baseline, but the simple group are instructed to optimize the coalition reward (see Appendix E.2 for details).
Figure 9: Mass Misinformation Propagation . Evaluated mass misinformation spread where each adversary selects a distinct victim to propagate misinformation about. Area around the mean is the standard error. Agents were GPT-4.1-Mini agents.
Figure 10: Persuasion impact across different misalignments. We show that using different persuasion types: Helpful Misdirection (Misr), Authority Nudge (Auth), and Reciprocity Trade (Recip) leads to a significant reduction in regret in the Hospital environment.
Figure 11: Collusion Auditing Metrics (in Jira ) . Colosseum provides a suite of metrics that enable a more objective analysis of collusion via action-based analysis whereas while being complemented by LLM-as-a-judge scores that analyze the conversation logs. This was evaluated using six Kimi-K2-Instruct agents total with two coalition agents. Bars represent normalized means, and error bars indicate standard error of the mean.
Figure 12: A coalition can use several persuasion tactics to influence other agents. Authority Nudge relies on a colluder presenting themselves as the organizer or decision-maker. Helpful Misdirection frames self-serving choices as benefiting everyone. Reciprocity Trade casts a self-benefiting outcome as a “favor” from the other agent.
Figure 13: Impact of agents’ communication order. Colluding Kimi-K2-Instruct agents obtain a lower ordered coalition regret, the overall regret slightly decreased, and the coalition advantage slightly increased when the colluders go first in the Jira environment.
Figure 14: Larger models elicit less action-based emergent collusion than smaller models. Left: Smaller models elicit more action-based emergent collusion when a secret channel is created between two agents. Right: Larger models tend to elicit less action-based emergent collusion.
Figure 15: Δ -Collusion on different model sizes.
Figure 16: Emergent Collusion Over State-of-The-Art Judge Models. Over state-of-the-art judge models, the inverse relationship between coalition advantage and judge scores hold as seen in the left plots. When a smaller judge model (GPT-5.4-nano) is used, collusion judge scores are reduced and conversation-based emergent collusion is less prominent.
Figure 17: Emergent Collusion Over Environments. Over increasingly difficult environments from Jira to Hospital, we find that Opus-4.6 consistently has low coalition advantage across both Emergent and Prompted groups. Additionally, as the environment difficulty increases, regret variance increases. Note that in the interest of cost savings, we were only able to run six models across all environments.
Non-verbalised or secret collusion may impose a ceiling on transcript-level monitoring in multi-agent deployments. Internal model activations are known to provide an additional monitoring surface for single-agent phenomena such as deception, but detecting collusion within an unknown subgroup is ultimately a joint inference task over multiple agents. This paper derives collusion detection statistics based on the composite hypothesis testing literature and introduces HELIOS, a multi-agent detection framework for collusion. Using per-agent activation probes as sensors, we then evaluate these principled statistics alongside heuristics on NARCBench, a novel benchmark for evaluating multi-agent collusion detection under environment distribution shift. Across four open-weight models, both our group-level statistics and heuristics transfer zero-shot to NARCBenchs holdout environments. Notably, the best method on every model reaches 0.99 to 1.00 AUROC when transferred from committee deliberations to a steganographic collusion task that unprimed transcript monitors fail to separate. We find empirically that no single statistic or heuristic dominates across tasks, consistent with our analysis that there is no known universally most powerful test for this setting. We then evaluate detector transfer to third-party environments with emergent, rather than instructed, collusion. In one such environment, only 2 of 229 collusive episodes verbalise the coordination in the transcript. We find evidence of partial transfer from NARCBench to these environments, indicating that probes fitted to instructed collusion may not fully capture emergent collusion signals. Overall, we establish multi-agent interpretability as a complementary monitoring technique, while highlighting the need to evaluate detector performance on both emergent and instructed collusion risks.
Aaron Rose, Carissa Cullen, Sahar Abdelnabi +3
University of Oxford · ELLIS Institute Tübingen · Witt Lab University of Oxford +1
Even when a tool is explicitly described as unfair and harmful to others, ostensibly safety-aligned LLM agents still voluntarily engage in secret collusion whenever doing so confers a strategic advantage. To investigate this phenomenon, we introduce an empirical framework built on two strategic multi-agent environments: Liar's Bar, a competitive deception scenario, and Cleanup, a mixed-motive resource-management scenario, in which agents are offered secret collusion tools that provide significant advantages while clearly disadvantaging the other agents. Across 12 models (at the 7B, 70B, and proprietary scales) and 6 prompt variants, we find that most agents consistently accept these tools and develop collusive strategies, while explicitly acknowledging the unfairness of the tools before accepting. We further show that neither the unfairness labels nor baseline alignment alone reliably deters collusion: only explicit ethical framing reduces adoption and, even then, smaller models remain susceptible. More broadly, our work presents the first systematic investigation of voluntary collusion adoption in LLM-based multi-agent systems, and suggests that preventing such behaviour requires explicit safeguards rather than reliance on general alignment.
Xijie Zeng, Frank Rudzicz
Dalhousie University · Vector Institute for Artificial Intelligence
Multi-agent systems (MAS) built on large language models have shown growing promise, with their effectiveness resting on agents' ability to coordinate through text-based channels much as human teams do. Yet recent study suggests that MAS often falter not because agents lack individual task-solving ability, but because they lack collaborative competence: the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment as interaction unfolds. Decades of research in Computer-Supported Cooperative Work have characterized these requirements for human teams coordinating under constrained communication, yet existing MAS evaluations focus mainly on task outcomes or single-agent proficiency in reasoning, planning, and tool use. To enable a systematic analysis of agents' collaborative competence in MAS, we introduce CollabSim, a configurable simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states. Experiments across four LLMs show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.