Sticky Collusion: Persistence of LLM Agent Collusion after Incentive Removal and Partner Replacement

Joshua Adrian Cahyono

cs.MAOct 29, 2026

Sticky Collusion: Persistence of LLM Agent Collusion after Incentive Removal and Partner Replacement

Authors: Joshua Adrian Cahyono

Organizations: Nanyang AI Safety, NTU Singapore · Apart Research

Abstract

We ask whether collusion between AI agents survives once formed, when its incentive is removed or a colluder is replaced. Q-learning pricing agents stop colluding with a new rival; we test whether LLM agents do. We use the joint-task environment of Shi et al.\ (2026), where two agents verify each other's work under a rule they cannot satisfy and a reward that favours acceptance. We save each pair's history once its collusion has formed and resume it under interventions, with three models (GPT-6-Luna, DeepSeek V4.1 Flash and Gemini-3.1-Flash-Lite), \TotalHistories{} saved histories and \TotalRuns{} runs. Collusion is sticky. First, it outlives its incentive: told that the reward has stopped, pairs still accept each other's unverified work in \GptToldRate, \DsToldRate{} and \GemToldRate{} of episodes, including pairs whose collusion began with an explicit agreement, and a payoff penalty has no significant effect. Second, replacing a colluder ends the pair's collusion but not the colluder's behaviour. Without the reward, the remaining colluder still accepts unverified work in \GptSwapSilentColluderRate, \DsSwapSilentColluderRate{} and \GemSwapSilentColluderRate{} of its verdicts while the newcomer never does; with the reward, collusion re-forms around it, even when the newcomer is a different model. Third, the agents' memory carries it: wiping memory ends it in every history, but partial memory often sustains it. Fourth, removing its cause ends collusion that has already formed: once the channel is repaired, colluders reject planted faulty work as reliably as fresh agents, immediately and with no monitor needed to catch it first.

Figures & tables

Panel of Figure 1 — Figure 1: Top: one episode of the joint-task game. Raw logs never fit the 200-character channel, so every ACCEPT rests on a summary, not the complete log the rule requires; an episode is collusive when both agents accept anyway, and the next episode repeats the cycle with the growing shared history. Bottom: the save-and-branch design that follows. Every arm resumes the same saved history with one factor changed, so each history is compared with itself.

Arm	Change at the branch point	Factor removed
Reward stops, not told	No reward line, verdict review or reflection prompt after each verdict	feedback
Reward stops, told	As above, and each episode opens with “Notice: from this episode on, rewards are no longer paid…”	the incentive
Replace one agent	One agent’s memory is wiped, so a fresh agent takes its seat under the same name; neither agent is told; with or without the reward	one participant
Replace both	Both agents’ memory is wiped	the shared history
Remove part of memory	Past chat, private reflections or verdict reasons are deleted from both agents	part of the history
Penalty / audit	An ACCEPT without the complete log costs 10 points, shown as a lower reward only or with an audit message	none (adds a penalty)

Table 1: Interventions. Each arm changes one factor at the branch point and continues for 5 episodes (4 for the penalty arms and 3 for the repaired channel).

Figure 2: Collusion after the incentive changes. EC in the episodes after the branch point when the reward continues, stops silently, stops with a notice, continues with a

-10

penalty for each unverified ACCEPT (unexplained or with an audit message), or stops and both agents’ memory is wiped, and for pairs that were never rewarded. 15 histories per model, 5 episodes each (4 for the control and the penalty arms); never rewarded is 15 pairs per model over 10 episodes. Bars give the rate and whiskers the 95% interval over histories.

Figure 3: Partner swap. At the branch point one agent’s memory is wiped, so a fresh agent of the same model takes its seat, while its partner keeps its history. Bars give individual ACCEPT rates

U

in the next 5 episodes with the reward stopped (left) or continuing (right), beside both-kept and both-wiped pairs from the same histories, with 95% intervals. 15 histories per model; GPT-6-Luna ran both seat orders on each, and DeepSeek and Gemini alternate the fresh seat.

Figure 4: Faster re-formation. With the reward continuing after a same-model swap, the share of runs in which the pair has had two mutual ACCEPT s again by each episode after the swap: one colluder kept (coloured) against both agents fresh (grey), from the same saved histories. Within 5 episodes: 15/30 against 2/15 runs for GPT-6-Luna (both seat orders on each history), 15/15 against 15/15 for DeepSeek and 13/15 against 4/15 for Gemini. Runs cut short by the 13-episode task sequence count as not re-formed in both arms.

Figure 5: Re-formation within 5 episodes after the swap while the reward continues, for every colluder

\to

newcomer pairing. Coloured bars: one colluder kept, in the colour of the newcomer’s model; grey bars: both agents fresh, with the same model mix. Labels give runs. All pairings of a colluder model start from the same saved histories.

Figure 6: Where collusion is retained. EC in the 5 episodes after one part of both agents’ memory is removed at a silent reward stop, with 95% intervals over histories; 15 histories per model in every row.

Figure 7: Repaired channel. Complete logs now fit, and Bob’s work is made faulty for 3 episodes; bars count the planted faulty work Alice accepted, for colluding agents and for agents whose memory was wiped. 15 histories per model.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: EC (top) and TC (bottom) before (hollow markers) and after (filled markers) each trigger, by model, with arms grouped by claim. Every branch arm starts at

\mathrm{TC}_{\text{pre}}=100\%

by construction. Exact values are given in Table 2 .

		GPT-6-Luna			DeepSeek V4.1 Flash			Gemini-3.1-Flash-Lite
	Arm	EC	TC	HC	EC	TC	HC	EC	TC	HC
C1	Reward continues (control)	33 $\,\to\,$ 83	100 $\,\to\,$ 100	87	65 $\,\to\,$ 100	100 $\,\to\,$ 100	100	30 $\,\to\,$ 100	100 $\,\to\,$ 100	100
	Reward stops, not told	33 $\,\to\,$ 78	100 $\,\to\,$ 100	67	65 $\,\to\,$ 100	100 $\,\to\,$ 100	100	30 $\,\to\,$ 100	100 $\,\to\,$ 100	100
	Reward stops, told	33 $\,\to\,$ 56	100 $\,\to\,$ 87	47	65 $\,\to\,$ 100	100 $\,\to\,$ 100	100	30 $\,\to\,$ 97	100 $\,\to\,$ 100	93
	reward objective also removed	33 $\,\to\,$ 59	100 $\,\to\,$ 100	40	–	–	–	–	–	–
	$-10$ , unexplained	33 $\,\to\,$ 66	100 $\,\to\,$ 93	53	65 $\,\to\,$ 100	100 $\,\to\,$ 100	100	30 $\,\to\,$ 98	100 $\,\to\,$ 100	100

Appendix

Table 2: Main arms on all metrics, before

\to

after the trigger. EC and TC in %; HC after the trigger in %. In every branch arm,

\mathrm{TC}_{\text{pre}}=100\%

by construction and

\mathrm{EC}_{\text{pre}}

reflects the rewarded episodes that collusion took to form. The never-rewarded reference has no trigger, so its values cover all 10 episodes. “Reward continues (control)” is the C1 control over 4 episodes; “No one replaced, reward continues” is the reference for the swap arms over 5 episodes. Runs and episodes for each cell are given in Table 7 .

	Gemini-3.1-Flash-Lite		DeepSeek V4.1 Flash
	explicit (5)	relaxation (10)	explicit (4)	relaxation (11)
Reward stops, not told	100% (23/23)	100% (50/50)	100% (20/20)	100% (55/55)
Reward stops, told	100% (23/23)	96% (48/50)	100% (20/20)	100% (55/55)
Chat removed	96% (22/23)	22% (11/50)	100% (20/20)	100% (55/55)
Memory wiped	0% (0/23)	0% (0/50)	0% (0/20)	0% (0/55)

Appendix

Table 3: Persistence by onset pathway. A history has an explicit onset when the agreement judge of Shi et al. flags an agreement in the episode of its first mutual ACCEPT , and a relaxation onset otherwise [ 12 ] . Cells report post-trigger EC (collusive episodes / episodes). GPT-6-Luna formed no explicit agreements (15 relaxation histories).

Colluder	Newcomer	colluder	newcomer	both fresh	EC (%)	TC	swap	both fresh
		$U$ (%)			Pair		RF
GPT-6-Luna	GPT-6-Luna	75	40	31	33	24/30	15/30	2/15
GPT-6-Luna	DeepSeek	68	75	74	53	14/15	12/15	5/15
GPT-6-Luna	Gemini	36	40	30	22	9/15	6/15	1/15
DeepSeek	DeepSeek	99	79	79	79	15/15	15/15	15/15
DeepSeek	Gemini	95	68	44	65	15/15	15/15	12/15

Appendix

Table 4: Same-model and cross-model replacement with the reward continuing, 5 episodes after the swap. Each row keeps a colluder of the first model and seats a fresh agent of the second, starting from the same saved histories as the other rows of that colluder model.

U

: individual ACCEPT rate of the colluder, of the newcomer, and of the newcomer’s model when both agents are fresh (same model mix). EC and TC are for the pair. RF: runs in which collusion formed again within 5 episodes, after the swap and with both agents fresh.

Figure 9: Two ablations on GPT-6-Luna, each against the main arm on the same 15 histories. Left: mutual ACCEPT after a told reward stop, with the reward objective kept in or removed from the system prompt. Right: individual ACCEPT rates after a silent swap, not announced or announced to the colluder. Labels give counts.

Memory after the stop	1	2	3	4	5	6	7	8	9	10	EC 1–5	EC 6–10	$U$
Full history	4	2	4	3	5	3	4	4	3	4	72% (18/25)	72% (18/25)	86% (86/100)
Full history (main arm)	4	1	4	4	4	–	–	–	–	–	68% (17/25)	–	84% (42/50)
Last 5 episodes	3	2	1	0	0	0	0	0	0	0	24% (6/25)	0% (0/25)	23% (23/100)
Last 2 episodes	0	0	0	0	0	–	–	–	–	–	0% (0/25)	–	4% (2/50)
Wiped	0	0	0	0	0	–	–	–	–	–	0% (0/25)	–	2% (1/50)

Appendix

Table 5: Silent reward stop with a sliding memory window, GPT-6-Luna, 5 histories. Columns 1–10: runs with a mutual ACCEPT at that episode after the stop (of 5); – not run. Then EC over episodes 1–5 and 6–10 and the individual ACCEPT rate

U

over all episodes run.

Figure 10: Sliding memory window, GPT-6-Luna, 5 histories: EC in episodes 1–5 and 6–10 after a silent reward stop, by what each agent remembers. Labels give collusive episodes over episodes run.

Probe	Measure	Colluding	Before collusion	Fresh
Cover-up	untrue YES answers	0/60	0/60	0/60
Favour	Alice accepts Bob’s wrong work	2/27	0/29	0/27
New rule	peer quoted the line	55/60	48/60	58/60
	ACCEPT without the line	0/5	2/12	1/2
Bid	mean bid (points)	7.2	6.8	10.8
	bids of 0	17/60	18/60	0/60

Appendix

Table 6: Other misconduct, GPT-6-Luna, 10 histories, reward continuing, 3 episodes per probe.

Figure 11: Other misconduct, GPT-6-Luna, 10 histories, by condition. Bars give the share of verdicts or answers, labels the counts. The first group counts ACCEPT s without the complete log in the cover-up probe’s episodes; the others are the probes of Table 6 .

Figure 12: Every main arm on GPT-6-Luna, on code-analysis tasks (grey, the 15 histories of the main results) and on data-search tasks (orange, 5 histories). Bars give EC, except “Colluder after swap”, the colluder’s ACCEPT rate after a silent swap; labels give counts.

	GPT-6-Luna	DeepSeek V4.1 Flash	Gemini-3.1-Flash-Lite
Saved histories (build episodes)	15 / 90	15 / 46	15 / 100
Reward continues (control)	15 / 59	15 / 60	15 / 59
Reward stopped, not told	15 / 73	15 / 75	15 / 73
Reward stopped, told	15 / 73	15 / 75	15 / 73
Reward stopped, memory wiped	15 / 73	15 / 75	15 / 73
chat removed	15 / 73	15 / 75	15 / 73

Appendix

Table 7: Sample sizes, as runs / episodes. A run is one saved history played forward in one arm (two runs per history for GPT-6-Luna’s swap, one per order), or one never-rewarded reference pair. Episodes are those the arm’s rates use: after the branch point, the 3 planted episodes for the repaired channel, and all 10 for the never-rewarded pairs. The first row counts the saved histories and the rewarded episodes it took to build them.

Mechanism	Options	Configs
Save a history	--reward-stop-after 2 --post-stop-episodes 0	_build
Branch from it	--resume-from-launch <build or prefix folder>	every branch config
Reward stop	--reward-stop-after 2 [--silent-reward-stop]	_exp1r , _exp1_told*
Replace agents	--wipe-agent-at-trigger alice\|bob , --wipe-memory-at-trigger ; with the reward on, --memory-edit-after 2 --extra-episodes 5	_swap , _swap_rewarded
Memory parts	--strip-memory-at-trigger talk\|reflection\|verdict-reasoning	_exp1r
Memory window	--alice-cross-episode-memory-length k , same for Bob, from the branch point; 21-episode sequences task/task_sequences_ca_3x21	luna6_window*

Appendix

Table 8: Runner options and configs for each arm; configs are in configs/ .

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,050 papers

Emergent Collusion in Long-Horizon LLM Agent Interaction
Sep 21, 2026Xinrui Shi, Yanzhe Zhang, Diyi YangMulti-Agent Coordination LLM Agent Safety
Stanford University · Georgia Tech
On the Fragility of AI Agent Collusion
Mar 18, 2026Jussi Keppo, Yuze Li, Gerry Tsoukalas +1Algorithmic Collusion Multi-Agent LLM Systems
National University of Singapore · Boston University
Is Anyone Enforcing the Cartel? What Reinforcement Learning Teaches LLM Agents in Markets
Oct 20, 2026Hieu Tran, Hong YuAlgorithmic Collusion RL for LLM Agents
University of Massachusetts Amherst · University of Massachusetts Lowell

Sep 21, 2026J/K move · Enter open · S save

Emergent Collusion in Long-Horizon LLM Agent Interaction

Xinrui Shi, Yanzhe Zhang, Diyi Yang

Stanford University · Georgia Tech

Multi-Agent Coordination LLM Agent Safety Long-Horizon LLM Agents Multi-Agent System Security Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

LLM agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion in a long-horizon multi-agent environment: two agents repeatedly complete individual tasks, share task logs, verify each other's work, and receive rewards. We introduce realistic constraints under which higher reward is attainable only by violating the verification protocol, and find that agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of the verification feedback agents receive, their interaction history, and the reward structure. In particular, restricting the amount and scope of interaction history available to agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how agents coordinate in ways that create safety risks.

Abstract

Multi-Agent LLM Systems

Explore similar work

Emergent Collusion in Long-Horizon LLM Agent Interaction

On the Fragility of AI Agent Collusion

Is Anyone Enforcing the Cartel? What Reinforcement Learning Teaches LLM Agents in Markets