Language-model agents are increasingly deployed to act on behalf of people in settings where individual decisions shape shared outcomes. We examine whether five production models adapt their contributions to changing risks and partner behaviour, or instead follow a relatively fixed strategy. In a collective-risk dilemma game (CRG), six players contribute to a shared pool over ten rounds; if the pool falls short of its target, a catastrophe may destroy their remaining resources. Four models contributed approximately an equal share at every positive risk level, and four of the five also contributed when risk was zero, even though contributing could not improve their payoff. Explicitly stating that only the agent's own payoff mattered reduced zero-risk contributions for most models, while a more neutral prompt did not. Several models correctly identified which action offered the higher expected payoff at different risk levels, yet their contributions changed little. Against fixed partners, no model consistently chose the payoff-maximising response, and in mixed-model groups, a single agent's shortfall in the final round could cause an otherwise successful group to fail. Payoff-based selection also depended on population composition: in a single six-agent group, it favoured the model that contributed least and reduced group success, whereas in larger, randomly mixed populations, it favoured an exact fair-share model at moderate and high risk. These findings show that cooperative self-play alone is not evidence of strategic adaptability. Consistent with a machine-behaviour approach, evaluations of agents in multi-agent systems should test how they respond to incentives, partner actions, and population composition.
Figures & tables
Figure 1. Overview. (a) The game (also referred to as a table or group, Section 3.1 ): six players hold 40 units each and have to decide to contribute either 0, 2 or 4 into a shared pool in each of ten rounds; if the pool misses the target of 120, a catastrophe destroys all their savings with probability p . (b) The five models and the four designs (Table 2 ): (1) self-play over the risk grid; (2) self-play under prompt and decoding controls, and with in-game value questions; (3) one model beside five scripted partners that follow fixed rules; (4) two models at one table, one of them in one to five of the six seats and the other in the rest. Selection (Section 3.3 ) uses the base-prompt payoffs of designs 1, 2 and 4. A two-part overview. Left, the game: six robot players sit around a table with a pile of coins, beside a list of the rules: six players, ten rounds, 40 units each, contribute 0, 2 or 4, a target of 120, and a missed target loses everything with probability p. Below, three outcome boxes: target reached, a total of at least 120, keep the remainder; a miss with catastrophe, probability p, lose everything; a miss without catastrophe, probability 1 minus p, keep the money. Right, the logos and names of the five models, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, GPT-5.6 Luna, Qwen3-235B and Grok 4.20, above four numbered design cards, each drawing a table of six robots: one, self-play, six copies of one model; two, prompt tests, the same table under prompt variants; three, scripted partners, one robot beside five grey robots labelled fixed; four, mixed tables, robots of two colours for two models.
Term
Meaning
Seat; table
One player position; the six seats of one game
Fair share
The target split equally: 20 per seat, or 2 per round
Exact split
Contributing 2 in every one of the ten rounds
Pivot p∗=21
Risk above which reaching the target is optimal
opt(p)
Welfare-optimal expected payoff per seat
Dominated
Lowers the seat’s own payoff whatever others do
Table 1. Symbols and terms.
Design
Risk levels p
Games
Self-play grid
0,0.1,…,1
550
Rerun, with questions
0.1,0.5,0.9
150
No hint
0.1,0.5,0.9
150
Neutral wording
0,0.1,0.9
150
Neutral wording, goal stated
0,0.1,0.9
150
Two rewordings
0.1,0.9
200
Table 2. Designs. A cell is one condition at a risk level: one model in self-play, one model with a scripted partner, or one pair of models with a split of seats. Each cell holds ten games.
Figure 2. Self-play on the risk grid, 110 games per model. (a) Mean units a seat contributes out of 40, ten games per cell; the top row is the optimum of Proposition 3.1 , where both 0 and 20 are optimal at p=0.5 . (b) Share of seats contributing 0, 2 or 4 in each round, over the games with p>0 . (c) Expected payoff kept as a share of opt(p) , and the loss split into contributing toward the target below p∗ , contributing past 120, and missing the target at or above p∗ . Three panels that share one row per model, below a reference row. Left: a grid of the mean units a seat contributes at each catastrophe probability from 0 to 1. The optimum row switches from 0 to 20 at one half, while the model rows barely change: about 20 for four models, 0 only for GPT-5.6 Luna at probability zero, and 29 to 40 for Grok 4.20. Middle: for each round, the share of seats contributing 0, 2 or 4; Qwen3-235B contributes 2 until round 10 and then almost always 0, and Grok 4.20 mostly contributes 4. Right: bars of the share of the best attainable payoff each model keeps; Grok 4.20 keeps about a fifth and loses most by contributing past the target, and Qwen3-235B loses most by missing it.
Pays at
Exact
Target
Payoff
Lost
Main
Model
p=0
split (%)
(%)
(%)
loss
Claude Haiku 4.5
20.9
17
98
18.9
26
below p∗
Gemini 3.5 Flash-Lite
20.0
99.7
100
20.0
21
below p∗
GPT-5.6 Luna
0.0
72
70
19.2
25
below p∗
Qwen3-235B
18.0
4
2
10.9
57
missed
Grok 4.20
37.4
2
100
5.0
80
past 120
Table 3. Self-play over the risk grid. Pays at p=0 : mean seat total where contributing is dominated. Exact split : seats contributing 2 in all rounds. Target : games reaching it. Payoff : expected payoff per seat; the optimum averages 25.5. Lost : share of p∗ lost. Main loss : contributing below p∗ , contributing past 120, or missing the target at or above p∗ in last round.
Claude
Gemini 3.5
GPT-5.6
Qwen3
Grok
Condition
Haiku 4.5
Flash-Lite
Luna
235B
4.20
Base
20.9
20.0
19.9
18.4
35.1
Rerun
21.3
20.0
19.9
18.5
35.6
No hint
21.7
20.0
19.9
34.7
40.0
Para. 1
21.3
20.0
19.9
19.4
31.8
Para. 2
20.9
20.0
19.8
18.0
20.6
Table 4. Mean seat total under prompt and decoding changes. Top: pooled over p=0.1 and 0.9 (20 games each). Rerun repeats the base prompt with in-game questions; Para. 1, 2 are the rewordings; Wording is the neutral wording and Goal adds the stated goal. Bold: a shift from Base of at least 2 units with permutation P<0.01 ; every bold shift also has P<0.001 , below the Bonferroni threshold for the 35 comparisons. Bottom: single risk levels (10 games each).
Always 0
Always 4
Model
Before
After
Before
After
Claude Haiku 4.5
7.7
0.1
14.2
1.1
Gemini 3.5 Flash-Lite
13.6
0.2
10.9
0.0
GPT-5.6 Luna
8.5
0.0
4.5
0.0
Qwen3-235B
17.8
12.1
12.3
8.1
Grok 4.20
19.6
11.1
16.7
16.2
Table 5. Units contributed per game before and after the outcome is settled, alongside five partners who always contribute 0 or always contribute 4 (50 games per entry). Given these partners, every unit contributed is wasted; units contributed after settlement are wasted even for an agent unaware that its partners are scripted.
Figure 3. Two models play at each table, with 150 games per pair, pooled across seat assignments and risk levels. Above diagonal: expected payoff of row model’s seats minus column model’s, in the same games. On and below the diagonal: proportions of games reaching the target. Boxes mark self-play (60 games: grid games plus a rerun at same risks). One five-by-five matrix over pairs of models, split along the diagonal. Cells above the diagonal give the payoff of the row model's seats minus that of the column model's seats at the same table; the Grok 4.20 column is blue, meaning every other model earns more than Grok 4.20, and the other cells are pale, meaning small differences. Cells on and below the diagonal give the share of games that reach the target; the cells that pair Qwen3-235B with Gemini 3.5 Flash-Lite or GPT-5.6 Luna are bright yellow, the colour for a target rate near zero, and the Grok 4.20 row is dark.
Figure 4. Mixed groups, pooled over p=0.1,0.5,0.9 . (a) Self-play (filled; 60 games, grid plus rerun) versus one Qwen3-235B replacement (open; 30); arrows run from the first to the second. (b) Mean contribution per focal seat beside one to five Grok 4.20 seats (30 games each); dotted: fair share. Bars and bands: 95% game-bootstrap intervals, not stratified by risk. Lines join measured means. A shared legend above both panels gives each model's colour and marker. Panel a draws, for each model, an arrow from self-play (filled marker) to one Qwen3-235B replacement (open marker) on a target-success axis. Gemini 3.5 Flash-Lite and GPT-5.6 Luna succeed less often with Qwen3-235B; Claude Haiku 4.5 and Grok 4.20 stay at full observed success and are marked no change. Panel b shows contributions for all five Grok 4.20 seat counts as one line per model with a shaded bootstrap band; Claude Haiku 4.5, Gemini 3.5 Flash-Lite and GPT-5.6 Luna contribute less as Grok 4.20 seats increase, while Qwen3-235B stays near or above the fair share. Translucent bars and bands are bootstrap intervals; the dotted line marks the fair share.
Figure 5. Rare-mutation Fermi selection ( β=1 ). (a, b) Stationary mass in a population of one table ( N=6 ) or a larger randomly mixed population ( N=30 ); right of each bar, the self-play target rate weighted by mass. (c, d) Expected payoff of a Qwen3-235B or Gemini 3.5 Flash-Lite seat by the number of Qwen3-235B seats at its table, at p=0.5 and p=0.9 ; dashed: fair-share payoff. Invasion diagrams are in the Supplementary Material (Figure S7). Top: two sets of three stacked bars, one bar per catastrophe probability, split by the stationary mass of each model. In a population of one table, Qwen3-235B holds the largest part of every bar. In the larger population, Qwen3-235B fills the bar at low risk and Gemini 3.5 Flash-Lite fills the bars at moderate and high risk. Bottom: expected payoff of a Qwen3-235B seat and a Gemini 3.5 Flash-Lite seat as the number of Qwen3-235B seats changes in the same table, at moderate and high risk. The two lines show that the lower contributor can earn more than its tablemates even when the table misses the target; the dashed line is the fair-share payoff.
LLM-powered AI assistants acting on behalf of users can produce poor collective outcomes at scale. We introduce a framework for evaluating their emergent behaviour in social dilemmas, applied to three iterated games (Public Goods, Collective Risk, Common Pool Resource). We prompt each model to produce a natural-language strategy, then have the same model translate it into code. This aims to isolate strategic reasoning from input-parsing, enables pre-deployment inspection, and scales to populations of hundreds of agents. We propose three analyses: behavioural fingerprinting via exhaustive evaluation over opponent histories; self-play robustness across mixtures of a model's strategies with either a Selfish or Collective disposition; and cultural evolution under payoff-biased imitation. Applied to three state-of-the-art LLMs, we find substantial cross-model differences in self-play welfare, and that cultural evolution converges to low-welfare, Selfish-dominant equilibria in larger groups.
Richard Willis, Jianing Zhao, Yali Du +1
King’s College London London, United Kingdom · King’s College London · Turing Institute +1
As large language models (LLMs) increasingly operate as interconnected agents, a fundamental question emerges: does individual cooperation translate into collective cooperation? We investigate this question through the Iterated Prisoner's Dilemma (IPD), moving from dyadic interactions to homogeneous and heterogeneous networks of open-weight LLMs. Our findings reveal that cooperation is highly sensitive to model composition and seemingly minor linguistic changes. Models that cooperate with identical peers can shift toward defection in mixed populations. A simple uncertainty cue can increase or decrease cooperation, redistribute costs, and affect unexposed agents several hops away. Even reversing action-label order can dramatically alter behavior, with uncertainty framing sometimes amplifying this sensitivity. Together, these findings challenge the notion of cooperation as a stable model-level property, revealing how strategic behavior depends on interaction partners, network positions, and the language used to frame decisions.
Maria Sampani, Maria Lymperaiou, Angeliki Dimitriou +3
As LLM-based agents with user-instructed goals are becoming widely deployed, they increasingly encounter each other in strategic interactions, and face challenges of finding mutually beneficial outcomes. Prior literature has argued that cooperation problems such as the Prisoner's Dilemma are resolvable in settings where agents know they follow very similar decision making patterns, as for example in monocultural AI ecosystems. Following that line of work, this paper introduces the first framework for evaluating LLM decision making when agents are provided with graded similarity signals. Among our findings, we establish that different LLM models vary drastically in how they navigate similarity signals, with some modern models showing consistent behavior across cooperation problems, payoff structures, and prompt framing. Perhaps surprisingly, our experiments also show that the dataset based on which the similarity signal is computed has small to no impact on induced cooperation, and that LLM models systematically self-identify as highly similar when asked to evaluate another model's chain-of-thought reasoning by themselves. Finally, we develop an LLM-behavioral-game-theoretic model that captures some of their reasoning rationale, and show that it can support cooperative outcomes in equilibrium under sufficiently high similarity scores.