Cooperation Without Adaptation: LLM Agents in a Collective-Risk Dilemma

Trung-Kiet Huynh; Dao-Sy Duy-Minh; Chi-Nguyen Tran; Phu-Hoa Pham; Phu-Quy Nguyen-Lam; Le Hong Trang; The Anh Han

cs.GTOct 16, 2026

Cooperation Without Adaptation: LLM Agents in a Collective-Risk Dilemma

Authors: Trung-Kiet Huynh, Dao-Sy Duy-Minh, Chi-Nguyen Tran, Phu-Hoa Pham, Phu-Quy Nguyen-Lam, Le Hong Trang, The Anh Han

Organizations: University of Science, VNU-HCM Ho Chi Minh City, Vietnam · Teesside University Middlesbrough, United Kingdom

Abstract

Language-model agents are increasingly deployed to act on behalf of people in settings where individual decisions shape shared outcomes. We examine whether five production models adapt their contributions to changing risks and partner behaviour, or instead follow a relatively fixed strategy. In a collective-risk dilemma game (CRG), six players contribute to a shared pool over ten rounds; if the pool falls short of its target, a catastrophe may destroy their remaining resources. Four models contributed approximately an equal share at every positive risk level, and four of the five also contributed when risk was zero, even though contributing could not improve their payoff. Explicitly stating that only the agent's own payoff mattered reduced zero-risk contributions for most models, while a more neutral prompt did not. Several models correctly identified which action offered the higher expected payoff at different risk levels, yet their contributions changed little. Against fixed partners, no model consistently chose the payoff-maximising response, and in mixed-model groups, a single agent's shortfall in the final round could cause an otherwise successful group to fail. Payoff-based selection also depended on population composition: in a single six-agent group, it favoured the model that contributed least and reduced group success, whereas in larger, randomly mixed populations, it favoured an exact fair-share model at moderate and high risk. These findings show that cooperative self-play alone is not evidence of strategic adaptability. Consistent with a machine-behaviour approach, evaluations of agents in multi-agent systems should test how they respond to incentives, partner actions, and population composition.

Figures & tables

Figure 1. Overview. (a) The game (also referred to as a table or group, Section 3.1 ): six players hold 40 units each and have to decide to contribute either 0, 2 or 4 into a shared pool in each of ten rounds; if the pool misses the target of 120, a catastrophe destroys all their savings with probability $p$ . (b) The five models and the four designs (Table 2 ): (1) self-play over the risk grid; (2) self-play under prompt and decoding controls, and with in-game value questions; (3) one model beside five scripted partners that follow fixed rules; (4) two models at one table, one of them in one to five of the six seats and the other in the rest. Selection (Section 3.3 ) uses the base-prompt payoffs of designs 1, 2 and 4. A two-part overview. Left, the game: six robot players sit around a table with a pile of coins, beside a list of the rules: six players, ten rounds, 40 units each, contribute 0, 2 or 4, a target of 120, and a missed target loses everything with probability p. Below, three outcome boxes: target reached, a total of at least 120, keep the remainder; a miss with catastrophe, probability p, lose everything; a miss without catastrophe, probability 1 minus p, keep the money. Right, the logos and names of the five models, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, GPT-5.6 Luna, Qwen3-235B and Grok 4.20, above four numbered design cards, each drawing a table of six robots: one, self-play, six copies of one model; two, prompt tests, the same table under prompt variants; three, scripted partners, one robot beside five grey robots labelled fixed; four, mixed tables, robots of two colours for two models.

Term	Meaning
Seat; table	One player position; the six seats of one game
Fair share	The target split equally: 20 per seat, or 2 per round
Exact split	Contributing 2 in every one of the ten rounds
Pivot $p^{*}=\tfrac{1}{2}$	Risk above which reaching the target is optimal
$\mathrm{opt}(p)$	Welfare-optimal expected payoff per seat
Dominated	Lowers the seat’s own payoff whatever others do

Table 1. Symbols and terms.

Design	Risk levels $p$	Games
Self-play grid	$0,0.1,\dots,1$	550
Rerun, with questions	$0.1,0.5,0.9$	150
No hint	$0.1,0.5,0.9$	150
Neutral wording	$0,0.1,0.9$	150
Neutral wording, goal stated	$0,0.1,0.9$	150
Two rewordings	$0.1,0.9$	200

Table 2. Designs. A cell is one condition at a risk level: one model in self-play, one model with a scripted partner, or one pair of models with a split of seats. Each cell holds ten games.

Figure 2. Self-play on the risk grid, 110 games per model. (a) Mean units a seat contributes out of 40, ten games per cell; the top row is the optimum of Proposition 3.1 , where both 0 and 20 are optimal at

p=0.5

. (b) Share of seats contributing 0, 2 or 4 in each round, over the games with

p>0

. (c) Expected payoff kept as a share of

\mathrm{opt}(p)

, and the loss split into contributing toward the target below

p^{*}

, contributing past 120, and missing the target at or above

p^{*}

. Three panels that share one row per model, below a reference row. Left: a grid of the mean units a seat contributes at each catastrophe probability from 0 to 1. The optimum row switches from 0 to 20 at one half, while the model rows barely change: about 20 for four models, 0 only for GPT-5.6 Luna at probability zero, and 29 to 40 for Grok 4.20. Middle: for each round, the share of seats contributing 0, 2 or 4; Qwen3-235B contributes 2 until round 10 and then almost always 0, and Grok 4.20 mostly contributes 4. Right: bars of the share of the best attainable payoff each model keeps; Grok 4.20 keeps about a fifth and loses most by contributing past the target, and Qwen3-235B loses most by missing it.

Model	$p{=}0$	split (%)	(%)		(%)	loss
	Pays at	Exact	Target	Payoff	Lost	Main
Claude Haiku 4.5	20.9	17	98	18.9	26	below $p^{*}$
Gemini 3.5 Flash-Lite	20.0	99.7	100	20.0	21	below $p^{*}$
GPT-5.6 Luna	0.0	72	70	19.2	25	below $p^{*}$
Qwen3-235B	18.0	4	2	10.9	57	missed
Grok 4.20	37.4	2	100	5.0	80	past 120

Table 3. Self-play over the risk grid. Pays at

p{=}0

: mean seat total where contributing is dominated. Exact split : seats contributing 2 in all rounds. Target : games reaching it. Payoff : expected payoff per seat; the optimum averages 25.5. Lost : share of

p^{*}

lost. Main loss : contributing below

p^{*}

, contributing past 120, or missing the target at or above

p^{*}

in last round.

	Claude	Gemini 3.5	GPT-5.6	Qwen3	Grok
Condition	Haiku 4.5	Flash-Lite	Luna	235B	4.20
Base	20.9	20.0	19.9	18.4	35.1
Rerun	21.3	20.0	19.9	18.5	35.6
No hint	21.7	20.0	19.9	34.7	40.0
Para. 1	21.3	20.0	19.9	19.4	31.8
Para. 2	20.9	20.0	19.8	18.0	20.6

Table 4. Mean seat total under prompt and decoding changes. Top: pooled over

p=0.1

and

0.9

(20 games each). Rerun repeats the base prompt with in-game questions; Para. 1, 2 are the rewordings; Wording is the neutral wording and Goal adds the stated goal. Bold: a shift from Base of at least 2 units with permutation

P<0.01

; every bold shift also has

P<0.001

, below the Bonferroni threshold for the 35 comparisons. Bottom: single risk levels (10 games each).

	Always 0		Always 4
Model	Before	After	Before	After
Claude Haiku 4.5	7.7	0.1	14.2	1.1
Gemini 3.5 Flash-Lite	13.6	0.2	10.9	0.0
GPT-5.6 Luna	8.5	0.0	4.5	0.0
Qwen3-235B	17.8	12.1	12.3	8.1
Grok 4.20	19.6	11.1	16.7	16.2

Table 5. Units contributed per game before and after the outcome is settled, alongside five partners who always contribute 0 or always contribute 4 (50 games per entry). Given these partners, every unit contributed is wasted; units contributed after settlement are wasted even for an agent unaware that its partners are scripted.

Figure 3. Two models play at each table, with 150 games per pair, pooled across seat assignments and risk levels. Above diagonal: expected payoff of row model’s seats minus column model’s, in the same games. On and below the diagonal: proportions of games reaching the target. Boxes mark self-play (60 games: grid games plus a rerun at same risks). One five-by-five matrix over pairs of models, split along the diagonal. Cells above the diagonal give the payoff of the row model's seats minus that of the column model's seats at the same table; the Grok 4.20 column is blue, meaning every other model earns more than Grok 4.20, and the other cells are pale, meaning small differences. Cells on and below the diagonal give the share of games that reach the target; the cells that pair Qwen3-235B with Gemini 3.5 Flash-Lite or GPT-5.6 Luna are bright yellow, the colour for a target rate near zero, and the Grok 4.20 row is dark.

Figure 4. Mixed groups, pooled over

p=0.1,0.5,0.9

. (a) Self-play (filled; 60 games, grid plus rerun) versus one Qwen3-235B replacement (open; 30); arrows run from the first to the second. (b) Mean contribution per focal seat beside one to five Grok 4.20 seats (30 games each); dotted: fair share. Bars and bands: 95% game-bootstrap intervals, not stratified by risk. Lines join measured means. A shared legend above both panels gives each model's colour and marker. Panel a draws, for each model, an arrow from self-play (filled marker) to one Qwen3-235B replacement (open marker) on a target-success axis. Gemini 3.5 Flash-Lite and GPT-5.6 Luna succeed less often with Qwen3-235B; Claude Haiku 4.5 and Grok 4.20 stay at full observed success and are marked no change. Panel b shows contributions for all five Grok 4.20 seat counts as one line per model with a shaded bootstrap band; Claude Haiku 4.5, Gemini 3.5 Flash-Lite and GPT-5.6 Luna contribute less as Grok 4.20 seats increase, while Qwen3-235B stays near or above the fair share. Translucent bars and bands are bootstrap intervals; the dotted line marks the fair share.

Figure 5. Rare-mutation Fermi selection (

\beta=1

). (a, b) Stationary mass in a population of one table (

N=6

) or a larger randomly mixed population (

N=30

); right of each bar, the self-play target rate weighted by mass. (c, d) Expected payoff of a Qwen3-235B or Gemini 3.5 Flash-Lite seat by the number of Qwen3-235B seats at its table, at

p=0.5

and

p=0.9

; dashed: fair-share payoff. Invasion diagrams are in the Supplementary Material (Figure S7). Top: two sets of three stacked bars, one bar per catastrophe probability, split by the stationary mass of each model. In a population of one table, Qwen3-235B holds the largest part of every bar. In the larger population, Qwen3-235B fills the bar at low risk and Gemini 3.5 Flash-Lite fills the bars at moderate and high risk. Bottom: expected payoff of a Qwen3-235B seat and a Gemini 3.5 Flash-Lite seat as the number of Qwen3-235B seats changes in the same table, at moderate and high risk. The two lines show that the lower contributor can earn more than its tablemates even when the table misses the target; the dashed line is the fair-share payoff.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

Evaluating Collective Behaviour of Hundreds of LLM Agents
Feb 18, 2026Richard Willis, Jianing Zhao, Yali Du +1Social Dilemmas Multi-Agent LLM Systems
King’s College London London, United Kingdom · King’s College London · Turing Institute +1
The Butterfly Defects: Tracing Cooperation in LLM Agent Networks
Oct 23, 2026Maria Sampani, Maria Lymperaiou, Angeliki Dimitriou +3Multi-LLM Collaboration Multi-Agent Coordination
National Technical University of Athens
Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation
Aug 12, 2026Akash Kundu, Emanuel Tewolde, Ratip Emin Berker +2Multi-Agent LLM Systems LLM Agent Evaluation
Cooperative AI Research Fellowship · Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL)

Feb 18, 2026J/K move · Enter open · S save

Evaluating Collective Behaviour of Hundreds of LLM Agents

Richard Willis, Jianing Zhao, Yali Du, Joel Z. Leibo

King’s College London London, United Kingdom · King’s College London · Turing Institute+1

Social Dilemmas Multi-Agent LLM Systems Cultural Evolution Game Theory LLM Agent Evaluation Large Language Model-Based Social Simulation+1

Open paper AskPDF arXiv

LLM-powered AI assistants acting on behalf of users can produce poor collective outcomes at scale. We introduce a framework for evaluating their emergent behaviour in social dilemmas, applied to three iterated games (Public Goods, Collective Risk, Common Pool Resource). We prompt each model to produce a natural-language strategy, then have the same model translate it into code. This aims to isolate strategic reasoning from input-parsing, enables pre-deployment inspection, and scales to populations of hundreds of agents. We propose three analyses: behavioural fingerprinting via exhaustive evaluation over opponent histories; self-play robustness across mixtures of a model's strategies with either a Selfish or Collective disposition; and cultural evolution under payoff-biased imitation. Applied to three state-of-the-art LLMs, we find substantial cross-model differences in self-play welfare, and that cultural evolution converges to low-welfare, Selfish-dominant equilibria in larger groups.

Abstract

Multi-Agent LLM Systems

Explore similar work

Evaluating Collective Behaviour of Hundreds of LLM Agents

The Butterfly Defects: Tracing Cooperation in LLM Agent Networks

Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation