Spatial Pattern Formation from Multi-Agent Learning in Public Goods Dilemmas

Yefei Zhang; Yuxuan Zhao

cs.MAOct 8, 2026

Spatial Pattern Formation from Multi-Agent Learning in Public Goods Dilemmas

Authors: Yefei Zhang, Yuxuan Zhao

Organizations: Department of Mathematics University of Illinois Urbana-Champaign Urbana, IL 61801, USA

Abstract

Spatial public goods models show that prescribed movement toward richer locations can generate spatial patterns. We ask how such patterns emerge when agents learn where to move and how learning rates shape their consequences for collective welfare. Fixed populations of cooperators and defectors independently learn movement policies using tabular Q-learning and local observations. Cooperator learning generates clusters around resource peaks, while co-adaptation changes their strength and motion. At a fixed training budget, the largest welfare losses occur when cooperators learn at high rates and defectors at low rates. In part of this regime, learned policies also generate traveling bands supported by a shared directional preference. The conditions supporting travel change with further training, so these patterns reflect training history rather than an established asymptotic outcome. Across the tested learning-rate conditions with cooperator learning, mean collective welfare falls below random movement because increased crowding outweighs gains in resource benefit. Charging agents for the crowding they impose on others during learning recovers much of the welfare loss in the tested conditions. These results connect learning rates to the emergence and welfare costs of spatial organization driven by individual rewards.

Figures & tables

Figure 1: The model and the learning loop of one agent. (a) Agent

j

on patch

i

of the ring observes its own and the two adjacent patches. (b) It bins the observation into one of

729

states, reads its private table

Q_{j}

, and chooses

a_{k}\in\{S,L,R\}

. (c) The environment evolves over the decision interval while the agent accumulates payoff. (d) At its next decision it observes

s_{k+1}

and updates

Q_{j}

symbol	meaning	value
$M$ , $L$ , $\Omega$	patches, ring length, patch volume	$64$ , $8$ , $20$
$N_{C}$ , $N_{D}$	agents per type	$448$ , $192$
$c$ , $\kappa$ , $\delta$ , $D_{\varphi}$	production, uptake, decay, diffusion	$1$ , $1$ , $10^{-3}$ , $0.03$
$r_{u}$ , $r_{v}$ , $\gamma$	benefit (C, D), crowding cost	$7$ , $6$ , $1$
$\Delta t$ , $K$ , $\tau_{\mathrm{dec}}$	time step, steps per decision, interval	$0.05$ , $10$ , $0.5$
$\beta$ , $\gamma_{\mathrm{dec}}$	discount rate, per-decision discount	$0.5$ , $0.779$

Table 1: Model and simulation parameters.

Figure 2: Spatial dynamics under frozen policies. Columns show the six learning conditions and rows show cooperators, defectors, and the public good. Cooperator learning produces aggregation, while faster cooperator learning can generate traveling bands. All panels start from the same held-out placement, and learned policies are frozen after $200$ training episodes.

Condition	$I_{U}$	$I_{V}$	Travel	$W$	$B$	$\mathcal{K}$
No movement	0.97	0.96	0/10	$2199\pm 4$	2999	352
Random movement	0.94	0.97	0/10	$2199\pm 1$	2998	352
Only D learns	0.94	0.45	0/10	$2205\pm 1$	2997	344
Only C learns	24.7	0.97	0/10	$1759\pm 49$	3066	858
Both learn $(0.05,0.05)$	7.9	1.46	0/10	$1977\pm 94$	3001	576
C co-trained, D random †	11.0	0.97	0/10	$2022\pm 25$	3039	570

Table 2: Frozen-policy evaluation of the main conditions (means over ten training seeds, matched placements): clustering of cooperators (

I_{U}

) and defectors (

I_{V}

), number of coherently traveling seeds, and welfare

W=B-N_{C}c-\mathcal{K}

with between-seed SD. Rate pairs are

(\alpha_{C},\alpha_{D})

. † Equal-rate co-trained cooperators with random defectors.

Figure 3: Resource seeking drives aggregation. (a) Under cooperator-only learning, the pooled policy favors movement toward richer patches. (b) Replaying this preference preserves clustering, while removing it collapses clustering to the random-movement level. Fixing the stay probability at one third does not remove the pattern. Blue shows cooperator-only learning at

\alpha_{C}=0.05

and green shows equal-rate co-learning at

\alpha_{C}=\alpha_{D}=0.05

Figure 4: Learning-rate dependence and directional bias after $200$ training episodes. Panels (a) and (b) show welfare loss relative to random movement, $W_{\mathrm{random}}-W_{\mathrm{learned}}$ , and cooperator clustering $I_{U}$ , averaged over ten training seeds and twenty seeds when $\alpha_{D}=0.05$ . The black outline marks cells in which at least half of the seeds travel coherently, and the dashed line marks $\alpha_{C}=\alpha_{D}$ . Panel (c) shows signed velocity against cooperator left–right action bias for the $172$ runs from the main conditions and the $5\times 5$ rate grid. These maps summarize finite-budget outcomes rather than an asymptotic phase diagram.

Figure 5: Training duration reshapes coherent travel. Each heatmap shows the fraction of ten training seeds whose frozen policies produce coherent travel. Panel (a) uses $\alpha_{D}=0.05$ , and panel (b) uses $\alpha_{D}=0.0125$ . Travel appears, disappears, and shifts across learning rates as training continues, with different behavior at the two defector learning rates. These finite-training results do not define an asymptotic phase boundary.

Figure 6: Crowding drives the welfare loss, and a toll largely reverses it. (a) Gain in resource benefit and increase in crowding cost relative to random movement. Black markers show the resulting welfare loss. (b) Welfare and (c) clustering after training with the original reward and with the crowding toll, which raises welfare and reduces clustering in all three conditions. Colors and marker shapes mark cooperator-only learning, equal-rate learning, and

(\alpha_{C},\alpha_{D})=(0.2,0.05)

, as labeled in panel a.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

	comparison	mean [95% interval]	paired difference [95% interval]	seeds
$I_{U}$	random movement	0.9 [0.9, 1.0]	–	–
$I_{U}$	only D learns	0.9 [0.9, 1.0]	identical	–
$I_{U}$	only C learns	24.7 [23.0, 26.3]	$+23.8$ [ $+22.0$ , $+25.4$ ]	10/10
$I_{U}$	both learn $(0.05,0.05)$	7.9 [6.3, 9.6]	$+6.9$ [ $+5.4$ , $+8.6$ ]	10/10
$I_{U}$	C co-trained, D random	11.0 [10.5, 11.6]	$+10.1$ [ $+9.5$ , $+10.7$ ]	10/10
$I_{U}$	both learn $(0.2,0.05)$	14.7 [11.9, 17.5]	$+13.7$ [ $+10.9$ , $+16.5$ ]	10/10

Appendix

Table 4: Seed-level uncertainty of the main comparisons. The unit of replication is the training seed (

n=10

), and the five fresh rollouts of a seed are averaged before any statistic is computed. Entries are means with

95\%

percentile bootstrap intervals over seeds from

10{,}000

resamples. Differences are taken per seed between conditions that share the seed and therefore the evaluation placement and decision phases. For the toll, the paired runs also share the training placements and phases. The last column counts the seeds in which the difference has the sign of its mean.

Figure 7: Training dynamics, averaged over ten seeds with a 10-episode running average. Exploration is still on during training and reaches

\varepsilon=0.01

at episode 160, marked by the dashed line. Left: welfare per training episode. Middle: the cooperators’ probability of moving to the richer neighbor when exactly one neighbor is richer, with the random value dotted. Right: relative resource variance, the spatial variance of the public good divided by its no-movement value. Clusters appear within the first twenty training episodes.

Figure 8: Policy recombination. Cooperator tables were trained with non-learning defectors or co-trained at equal rates. They are evaluated with untrained, randomly moving defectors or with the defector tables of the equal-rate run of the same seed. (a) Clustering. (b) Welfare. Points are training seeds, lines join the two defector policies of a seed, bars mark means, and dashed lines show random movement.

Figure 9: Cooperator occupancy on one fresh rollout of the $(0.05,0.0125)$ run with training seed $0$ . The panels show the learned tables, the distilled directional rule, and the mirror-symmetrized version of that rule, all from the same placement and decision phases. Across the two ratio- $4$ conditions, with $10$ training seeds and $5$ fresh rollouts each, the three policies travel coherently in $99$ , $100$ , and $0$ of the $100$ rollouts. Figure 2 of the main text shows all three fields for the six main conditions.

Figure 10: Complete learning-rate and directional-bias analysis of the $172$ trained runs. (a,b) Cell means of clustering and welfare on the $5\times 5$ grid. Red boxes mark cells in which at least half of the seeds travel coherently, and the dashed line marks equal rates. (c) Signed velocity $\bar{v}$ against the cooperators’ action bias $\chi_{C}$ for every trained run. (d) A fresh rollout of a ratio- $4$ run under the learned tables, the absolute rule, and the symmetrized rule. Figure 9 shows the same rollout enlarged. (e,f) Speed and clustering of the $50$ coherently traveling runs under the three policies in the broad scan. Bars are means and dots are runs.

Figure 11: Aligned profiles of the traveling bands. For each of the

50

coherently traveling runs and each snapshot

t=200,\dots,400

, the profiles are shifted so that the maximum of the cooperator profile, smoothed over five patches, is at

0

. They are reflected so that the band travels to the right and are then averaged over snapshots. Lines show the mean over runs and shading the interquartile range over runs. Dashed lines mark the maximum of each mean profile, and the horizontal axis is in patches. The defector maximum lies two patches behind the cooperator maximum.

Figure 12: Long training to

1600

episodes, with frozen checkpoints every

20

episodes and exploration

0.01

after episode

160

. The dashed line marks the main budget of

200

episodes. Thin lines are seeds and thick lines are means smoothed over three checkpoints. The ratio-

4

conditions have five seeds and the two reference conditions three. (a) Speed. (b) Magnitude of the cooperators’ action bias. The signed bias is not averaged because seeds travel in opposite directions. (c,f) Signed bias of the individual seeds of the two ratio-

4

conditions. At

(0.05,0.0125)

every seed reverses once. At

(0.2,0.05)

the bias forms, dissolves, and reverses repeatedly and is small at the end. (d) Clustering. (e) Welfare.

condition	own $\varphi_{i}$	up	stay	down	share	stay (other)	share
only C learns, C	poor	0.71	0.21	0.08	0.03	0.38	0.00
	intermediate	0.61	0.35	0.04	0.13	0.32	0.01
	rich	0.36	0.45	0.19	0.50	0.29	0.33
both learn, $(0.05,0.05)$ , C	poor	0.44	0.32	0.24	0.11	0.34	0.12
	intermediate	0.42	0.35	0.23	0.23	0.35	0.21
	rich	0.37	0.41	0.22	0.14	0.35	0.18

Appendix

Table 5: Pooled rules, means over ten seeds. In contexts with exactly one richer neighbor, the table gives for each own-patch level the probabilities of moving up, staying, and moving down, and the share of all decisions made in these contexts. In all other contexts it gives the stay probability, with moves split equally between left and right, and the share of all decisions. Shares are fractions of all decisions of that type and sum to one over the six contexts.

		learned				pooled				absolute				symmetrized				no	stay
$\alpha_{C}/\alpha_{D}$	$n$	$I_{U}$	$W$	$\|v\|$	trav.	$I_{U}$	$W$	$\|v\|$	trav.	$I_{U}$	$W$	$\|v\|$	trav.	$I_{U}$	$W$	$\|v\|$	trav.	pref.	1/3	$\chi_{C}$	$\|\chi_{C}\|$
0/0.05	10	0.9	2205	1.85	0.0	1.0	2195	1.75	0.0	1.0	2194	2.05	0.0	1.0	2194	2.03	0.0	1.0	1.1	+0.00	0.00
0.0125/0.0125	3	10.8	1915	0.07	0.0	6.5	2023	0.09	0.0	8.4	1958	0.19	0.0	9.3	1916	0.09	0.0	–	–	-0.00	0.00
0.0125/0.025	3	9.0	1983	0.12	0.0	4.3	2090	0.11	0.0	6.2	2056	0.15	0.0	5.6	2039	0.14	0.0	–	–	-0.01	0.01
0.0125/0.05	10	6.4	2043	0.18	0.0	4.7	2080	0.16	0.0	6.2	2037	0.15	0.0	5.6	2046	0.13	0.0	–	–	-0.00	0.01
0.0125/0.1	3	15.9	1788	0.03	0.0	5.6	2057	0.09	0.0	8.9	1971	0.09	0.0	9.1	1987	0.07	0.0	–	–	+0.00	0.00

Appendix

Table 6: Replay of distilled rules in the broad scan, given as cell means over all

n

trained runs of a cell. Rules are replayed from the placement and decision phases of the estimation rollout, with new action draws. Learned denotes the frozen tables. Pooled uses six contexts with left and right merged, Absolute uses 27 contexts, and Symmetrized averages the absolute rule with its mirror image. The columns give clustering

I_{U}

, welfare

W

, speed

|v|

, and the share of runs traveling coherently (trav.). For the core conditions, the columns “no pref.” and “stay 1/3” give the clustering of the pooled rule with equal up and down probabilities and with the stay probability set to

1/3

\chi_{C}

is the mean signed left–right bias and

|\chi_{C}|

the mean absolute bias. Cells are labeled by

\alpha_{C}/\alpha_{D}

condition	policy	$W$	$I_{U}$	$\|v\|$	trav.	$I_{U}$ range
only C learns	learned tables	1742	24.6	0.05	0/50 (0)	20.3–28.5
	pooled rule	1689	27.3	0.01	0/50 (0)	21.2–34.6
	pooled, equal up/down	2199	1.0	1.79	0/50 (0)	0.8–1.2
	pooled, stay 1/3	1664	28.0	0.01	0/50 (0)	20.8–39.4
	absolute rule	1613	30.2	0.07	0/50 (0)	21.7–39.7
	symmetrized rule	1635	29.5	0.03	0/50 (0)	21.6–38.7

Appendix

Table 7: Fresh-rollout replay. Rules are estimated from the evaluation rollout of each training seed and replayed on five fresh rollouts per seed. These rollouts have new placements and decision phases and are never used for estimation. The learned tables run on the same fresh rollouts. Entries are means over the ten seed means. The column trav. gives the rollouts with coherent travel out of

50

, followed in parentheses by the number of training seeds with at least one such rollout. The column range gives the minimum and maximum

I_{U}

over rollouts. Table 4 gives seed-level intervals.

	random		only C, own / toll				$(0.05,0.05)$ , own / toll				$(0.2,0.05)$ , own / toll				$\Pi_{C}$ / $\Pi_{D}$ , only C
seed	$W$	$\Pi_{C}$	$W$	$W$	$I_{U}$	$I_{U}$	$W$	$W$	$I_{U}$	$I_{U}$	$W$	$W$	$I_{U}$	$I_{U}$	own		toll
0	2198	3.39	1813	2029	21.9	11.3	1954	2206	9.2	0.8	1799	2171	15.5	2.1	3.30	1.74	3.64	2.07
1	2199	3.38	1792	2020	22.4	10.7	2004	2205	7.3	1.0	1887	2179	9.7	1.7	3.17	1.93	3.55	2.23
2	2199	3.38	1688	2012	27.4	11.0	2141	2203	3.3	1.0	1964	2181	8.5	2.0	3.14	1.46	3.55	2.21
3	2197	3.37	1715	2029	27.5	10.6	1803	2205	13.3	1.0	1564	2185	20.8	2.2	3.18	1.51	3.56	2.26
4	2199	3.39	1801	2082	23.9	8.0	2005	2203	8.3	1.1	1738	2184	15.0	1.6	3.39	1.47	3.63	2.38

Appendix

Table 8: Crowding toll, per seed. The table gives welfare

W

, clustering

I_{U}

, and the payoffs

\Pi_{C}

and

\Pi_{D}

of populations trained on the original reward (own) and on the reward minus the crowding toll (toll), with random movement for reference. The two runs of a seed share the training placements, the decision phases, and the evaluation placement.

W

is always computed from the original payoff.

$\alpha_{C}$	$\alpha_{D}$	$n$	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$\|v\|$	dir.	corr	traveling	95% interval
0.0125	0.0125	3	10.8	1915	0.38	0.41	0.07	0.21	0.88	0/3	[0.00, 0.56]
0.0125	0.025	3	9.0	1983	0.38	0.42	0.12	0.33	0.87	0/3	[0.00, 0.56]
0.0125	0.05	10	6.4	2043	0.39	0.39	0.18	0.39	0.77	0/10	[0.00, 0.28]
0.0125	0.1	3	15.9	1788	0.34	0.48	0.03	0.14	0.94	0/3	[0.00, 0.56]
0.0125	0.2	3	14.3	1855	0.35	0.46	0.03	0.11	0.92	0/3	[0.00, 0.56]
0.025	0.0125	10	9.8	1916	0.41	0.35	0.24	0.57	0.86	2/10	[0.06, 0.51]

Appendix

Table 9: The

5\times 5

learning-rate grid with frozen policies after 200 episodes. For each cell the table gives the number of training seeds

n

and the means over seeds of clustering, welfare, the cooperators’ probabilities of moving up and of staying on a rich patch, speed, main-direction share, and best-shift correlation. It also gives the number of coherently traveling seeds with its

95\%

Wilson interval for the traveling probability.

$\alpha_{C}$ / $\alpha_{D}$	seed	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$\bar{v}$	$\|v\|$	$d$	$r$	trav.
0.025 / 0.0125	0	9.4	1944	0.41	0.35	0.11	0.13	0.546	0.87
0.025 / 0.0125	1	7.4	1962	0.41	0.36	-0.02	0.10	0.265	0.86
0.025 / 0.0125	2	9.3	1940	0.41	0.37	-0.43	0.56	0.969	0.83	yes
0.025 / 0.0125	3	7.1	1989	0.43	0.34	-0.05	0.14	0.383	0.82
0.025 / 0.0125	4	9.7	1923	0.42	0.34	0.00	0.16	0.352	0.85
0.025 / 0.0125	5	6.7	2026	0.41	0.36	-0.25	0.25	0.816	0.79

Appendix

Table 10: The two ten-seed cells of ratio

\alpha_{C}/\alpha_{D}=2

that lie off the row and column of Tables 20 – 25 , per seed. The columns give clustering, welfare, the cooperators’ probabilities of moving up and of staying on a rich patch, signed velocity

\bar{v}

, speed

|v|

, main-direction share

d

, best-shift correlation

r

, and coherent travel.

condition	$\varepsilon_{\mathrm{eval}}$	$I_{U}$	$W$	trav.	$\|\chi_{C}\|$
only C	0	24.4 [23.5, 25.4]	1755 [1737, 1772]	0/50	0.01
	0.01	24.6 [23.9, 25.2]	1742 [1727, 1756]	0/50	0.01
	0.05	24.0 [23.0, 25.1]	1762 [1739, 1781]	0/50	0.01
$(0.05,0.05)$	0	7.9 [6.5, 9.3]	1995 [1955, 2033]	0/50	0.01
	0.01	7.4 [6.2, 8.9]	2004 [1964, 2042]	0/50	0.01
	0.05	6.7 [5.7, 8.0]	2023 [1988, 2054]	0/50	0.01

Appendix

Table 11: Exploration sensitivity of the frozen policies. The saved tables are evaluated with

\varepsilon_{\mathrm{eval}}\in\{0,0.01,0.05\}

on the same five fresh rollouts per training seed. Entries are means over the ten seed means with

95\%

bootstrap intervals over seeds. The column trav. counts coherently traveling rollouts out of

50

, and

|\chi_{C}|

is the mean absolute bias.

rule	followed by	$\lambda_{\varphi}$	$\lambda_{n}$	$I_{U}$	$W$	$B$	$\mathcal{K}$	trav.
random movement	–	–	–	1.0	2199	2998	352	0/50
weak resource seeking	C	1	0	39.9	1378	3047	1221	0/50
	C and D	1	0	19.2	1452	2996	1096	0/50
strong resource seeking	C	3	0	51.3	1105	3032	1478	0/50
	C and D	3	0	65.2	-721	2994	3267	0/50
resource and crowding	C	3	1	47.5	1194	3036	1394	0/50

Appendix

Table 12: Hand-designed movement rules on the same

10\times 5

fresh placements and decision phases as used for the learned policies. In rows marked ‘C’, cooperators follow the rule and defectors move at random. In rows marked ‘C and D’, both types follow it. The learned policies on the same rollouts are shown for comparison. Entries are means over rollouts, and trav. counts coherently traveling rollouts out of

50

condition	policy	$I_{U}$	$W$	$\|v\|$	trav.	$\|\chi_{C}\|$
$(0.2,0.05)$	learned tables	14.8 [11.7, 18.1]	1769	1.06 [0.93, 1.19]	49/50 (10)	0.34
	C symmetrized	13.4 [8.9, 20.6]	1794	0.41 [0.34, 0.47]	38/50 (9)	0.10
	D symmetrized	19.9 [17.2, 22.9]	1734	1.07 [1.00, 1.15]	50/50 (10)	0.42
	both symmetrized	7.3 [5.3, 9.7]	1973	0.33 [0.26, 0.40]	0/50 (0)	0.01
$(0.05,0.0125)$	learned tables	22.6 [19.5, 25.8]	1551	0.99 [0.94, 1.04]	50/50 (10)	0.33
	C symmetrized	41.2 [37.1, 45.0]	899	0.15 [0.10, 0.19]	6/50 (2)	0.06

Appendix

Table 13: Mirror-symmetrized original tables on the fresh rollouts of the two ratio-4 conditions, with ten training seeds and five rollouts each. Every agent of the indicated type acts on

\tfrac{1}{2}[\pi_{j}(a\mid s)+\pi_{j}(\mathcal{M}a\mid\mathcal{M}s)]

computed from its own frozen table. Entries are means over the ten seed means with

95\%

bootstrap intervals over seeds. The column trav. counts coherently traveling rollouts out of

50

, followed in parentheses by the training seeds with at least one such rollout.

condition	policy	disagr. C	D	$I_{U}$	$W$	trav.	$\|\chi_{C}\|$
$(0.2,0.05)$	learned tables	0.00	0.00	14.8 [11.7, 18.1]	1769	49/50	0.34
	both symmetrized	0.33	0.32	7.3 [5.3, 9.7]	1973	0/50	0.01
	control, both	0.33	0.32	3.0 [2.7, 3.4]	2136	4/50	0.07
	C symmetrized	0.33	0.00	13.4 [8.9, 20.6]	1794	38/50	0.10
	control, C	0.33	0.00	1.6 [1.5, 1.7]	2177	0/50	0.05
	uniform eps 0.1	0.06	0.06	10.6 [9.2, 12.1]	1904	49/50	0.26

Appendix

Table 14: Symmetrization compared with added randomness on the fresh rollouts of the two ratio-4 conditions, with ten training seeds and five rollouts each. “Disagreement” is the fraction of a type’s decisions at which the policy’s action differs from the action the original policy would take at the same observation. The matched control keeps the original tables and adds uniform exploration at the rate that reproduces the disagreement of the symmetrized policy. The last three rows add uniform exploration

\varepsilon

to both types. Entries are means over the ten seed means with

95\%

bootstrap intervals for

I_{U}

, and trav. counts coherently traveling rollouts out of

50

cooperators	defectors	$I_{U}$	(sd)	$W$	(sd)	$B$	$\mathcal{K}$	$\varphi_{C}$	$\varphi_{D}$
C alone	random	24.7	2.7	1759	49	3066	858	0.85	0.36
C alone	learned	45.6	16.5	883	495	3029	1698	0.76	0.55
C co-trained	random	11.0	1.0	2022	25	3039	570	0.79	0.48
C co-trained	learned	7.9	2.7	1977	94	3001	576	0.71	0.67

Appendix

Table 15: Policy recombination on the held-out evaluation placements. Entries are means over ten training seeds, with the between-seed standard deviation of

I_{U}

and

W

. “C alone” denotes cooperator tables trained with non-learning defectors, and “C co-trained” the cooperator tables of the equal-rate run of the same seed. Defectors are either untrained and move at random or carry the tables of the equal-rate run.

variant	$I_{U}$	$W$	$W_{\mathrm{rnd}}$	$\|v\|$	dir.	trav.
	only C / both / only D			ratio 4
$M=32$	26.6 / 4.4 / 1.0	847 / 1037 / 1102	1099	1.06	1.00	3/3
$M=128$	24.1 / 8.2 / 1.0	3554 / 3991 / 4410	4399	1.18	1.00	3/3
$D_{\varphi}=0.06$	21.5 / 5.5 / 0.8	1801 / 2023 / 2207	2198	0.79	0.81	2/3
$r_{v}=6.5$	23.9 / 2.0 / 0.8	1800 / 2238 / 2271	2264	0.37	0.74	0/3
main (10 seeds)	24.7 / 7.9 / 0.9	1760 / 1976 / 2205	2199	1.06	0.99	10/10

Appendix

Table 16: Environment variants with three seeds each, compared with the main model with ten. Clustering and welfare are listed for cooperator-only, both-type, and defector-only learning.

W_{\mathrm{rnd}}

is random-movement welfare. The last columns give the speed, directional consistency, and number of coherently traveling seeds at

(0.2,0.05)

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ none	random	learned	$\rho$ none	random	learned	$W$ none	random	learned
0	0.42	0.33	0.83	0.94	21.93	0.38	0.33	0.85	2202	2199	1811
1	0.43	0.33	1.06	0.79	22.40	0.43	0.42	0.82	2205	2199	1792
2	0.41	0.33	0.90	0.80	27.41	0.27	0.42	0.85	2198	2198	1691
3	0.35	0.33	0.92	0.95	27.49	0.35	0.28	0.81	2204	2197	1716
4	0.39	0.33	1.17	1.05	23.85	0.36	0.27	0.85	2193	2199	1800
5	0.42	0.33	0.85	1.04	24.31	0.28	0.30	0.82	2199	2201	1777

Appendix

Table 17: Only cooperators learn (

\alpha_{C}=0.05

\alpha_{D}=0

), main model, per seed.

I_{U}

\rho_{U\varphi}

, and

W

for the no-movement, random-movement, and learned policies from the same initial placement.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ none	random	learned	$\rho$ none	random	learned	$W$ none	random	learned
0	0.41	0.41	0.83	0.94	9.17	0.38	0.33	0.88	2202	2199	1953
1	0.42	0.41	1.06	0.79	7.31	0.43	0.42	0.87	2205	2199	2004
2	0.41	0.40	0.90	0.80	3.31	0.27	0.42	0.83	2198	2198	2141
3	0.39	0.40	0.92	0.95	13.25	0.35	0.28	0.93	2204	2197	1803
4	0.41	0.41	1.17	1.05	8.29	0.36	0.27	0.91	2193	2199	2004
5	0.40	0.42	0.85	1.04	10.30	0.28	0.30	0.93	2199	2201	1850

Appendix

Table 18: Both learn (

\alpha_{C}=\alpha_{D}=0.05

), main model, per seed.

I_{U}

\rho_{U\varphi}

, and

W

for the no-movement, random-movement, and learned policies from the same initial placement.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ none	random	learned	$\rho$ none	random	learned	$W$ none	random	learned
0	0.34	0.37	0.83	0.94	0.94	0.38	0.33	0.66	2202	2199	2205
1	0.33	0.38	1.06	0.79	0.79	0.43	0.42	0.64	2205	2199	2206
2	0.34	0.37	0.90	0.80	0.80	0.27	0.42	0.59	2198	2198	2204
3	0.34	0.38	0.92	0.95	0.95	0.35	0.28	0.62	2204	2197	2205
4	0.33	0.37	1.17	1.05	1.05	0.36	0.27	0.62	2193	2199	2205
5	0.33	0.36	0.85	1.04	1.04	0.28	0.30	0.65	2199	2201	2207

Appendix

Table 19: Only defectors learn (

\alpha_{C}=0

\alpha_{D}=0.05

), main model, per seed.

I_{U}

\rho_{U\varphi}

, and

W

for the no-movement, random-movement, and learned policies from the same initial placement.

$\alpha_{C}$	seed	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$P_{\text{leave}\|\text{poor}}^{C}$	$P_{\text{up}}^{D}$
0.05	0	21.9	1811	0.42	0.38	0.85	0.33
0.05	1	22.4	1792	0.43	0.38	0.74	0.33
0.05	2	27.4	1691	0.41	0.39	0.83	0.33
0.05	3	27.5	1716	0.35	0.47	0.76	0.33
0.05	4	23.9	1800	0.39	0.40	0.70	0.33
0.05	5	24.3	1777	0.42	0.37	0.84	0.33

Appendix

Table 20: Learning-rate sweep, per seed, part 1 of 3. Clustering, welfare, and learned behavior of the frozen policy.

$\alpha_{C}$	$\alpha_{D}$	seed	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$P_{\text{leave}\|\text{poor}}^{C}$	$P_{\text{up}}^{D}$
0.05	0.05	7	8.5	1959	0.41	0.37	0.68	0.42
0.05	0.05	8	8.4	1936	0.41	0.37	0.67	0.43
0.05	0.05	9	4.8	2061	0.41	0.37	0.68	0.41
0.05	0.1	0	10.0	1984	0.40	0.42	0.67	0.41
0.05	0.1	1	17.9	1673	0.39	0.39	0.66	0.37
0.05	0.1	2	8.2	2018	0.41	0.38	0.67	0.41

Appendix

Table 21: Learning-rate sweep, per seed, part 2 of 3. Clustering, welfare, and learned behavior of the frozen policy.

$\alpha_{C}$	$\alpha_{D}$	seed	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$P_{\text{leave}\|\text{poor}}^{C}$	$P_{\text{up}}^{D}$
0.0125	0.05	4	12.2	1887	0.37	0.39	0.65	0.41
0.0125	0.05	5	10.6	1936	0.38	0.41	0.64	0.38
0.0125	0.05	6	3.9	2107	0.40	0.38	0.66	0.41
0.0125	0.05	7	4.0	2122	0.41	0.38	0.67	0.40
0.0125	0.05	8	5.7	2034	0.39	0.38	0.65	0.42
0.0125	0.05	9	4.2	2084	0.39	0.39	0.66	0.41

Appendix

Table 22: Learning-rate sweep, per seed, part 3 of 3. Clustering, welfare, and learned behavior of the frozen policy.

$\alpha_{C}$ / $\alpha_{D}$	seed	$\bar{v}$	$\|v\|$	main dir.	corr
0.05 / 0	0	-0.01	0.03	0.097	0.93
0.05 / 0	1	-0.01	0.03	0.112	0.91
0.05 / 0	2	0.04	0.05	0.245	0.94
0.05 / 0	3	0.01	0.21	0.107	0.95
0.05 / 0	4	-0.00	0.01	0.031	0.95
0.05 / 0	5	0.02	0.03	0.112	0.93

Appendix

Table 23: Motion of the cooperator profile, per seed, part 1 of 3. The table gives the mean signed velocity

\bar{v}

and the mean speed

|v|

in patches per time unit at lag

\Delta=5

, the share

d

of windows in the run’s main direction to three decimals, the correlation

r

at the best shift, and whether the run travels coherently, that is, whether

r>0.7

and

d>0.9

$\alpha_{C}$ / $\alpha_{D}$	seed	$\bar{v}$	$\|v\|$	main dir.	corr
0.05 / 0.2	7	0.04	0.04	0.189	0.96
0.05 / 0.2	8	-0.03	0.03	0.163	0.94
0.05 / 0.2	9	-0.01	0.04	0.128	0.91
0.05 / 0.1	0	-0.04	0.07	0.270	0.86
0.05 / 0.1	1	0.06	0.06	0.276	0.95
0.05 / 0.1	2	-0.02	0.13	0.286	0.81

Appendix

Table 24: Motion of the cooperator profile, per seed, part 2 of 3. The table gives the mean signed velocity

\bar{v}

and the mean speed

|v|

in patches per time unit at lag

\Delta=5

, the share

d

of windows in the run’s main direction to three decimals, the correlation

r

at the best shift, and whether the run travels coherently, that is, whether

r>0.7

and

d>0.9

$\alpha_{C}$ / $\alpha_{D}$	seed	$\bar{v}$	$\|v\|$	main dir.	corr	trav.
0.05 / 0.025	4	-0.52	0.52	0.985	0.83	yes
0.05 / 0.025	5	0.58	0.58	0.995	0.82	yes
0.05 / 0.025	6	-0.41	0.41	0.903	0.78	yes
0.05 / 0.025	7	0.43	0.43	0.990	0.90	yes
0.05 / 0.025	8	0.39	0.39	0.954	0.86	yes
0.05 / 0.025	9	0.42	0.42	0.985	0.81	yes

Appendix

Table 25: Motion of the cooperator profile, per seed, part 3 of 3. The table gives the mean signed velocity

\bar{v}

and the mean speed

|v|

in patches per time unit at lag

\Delta=5

, the share

d

of windows in the run’s main direction to three decimals, the correlation

r

at the best shift, and whether the run travels coherently, that is, whether

r>0.7

and

d>0.9

condition	episode	$\|v\|$	dir.	$I_{U}$	$W$	$P_{\text{up}}^{C}$	$P_{\text{stay}\|\text{rich}}^{C}$	$\|\chi_{C}\|$
only C learns, $(0.05,0)$ (3 seeds)	100	0.04	0.07	25.2	1739	0.35	0.47	0.00
	300	0.03	0.08	26.6	1714	0.40	0.40	0.00
	500	0.02	0.06	24.7	1759	0.41	0.39	0.00
	700	0.04	0.13	23.2	1764	0.40	0.41	0.00
	900	0.02	0.09	27.7	1722	0.38	0.42	0.01
	1100	0.01	0.05	25.4	1757	0.43	0.38	0.01

Appendix

Table 26: Longer training, with means over seeds at each checkpoint of the frozen tables. Checkpoints are every 200 episodes of the runs of Figure 12 . The columns give speed

|v|

, main-direction share, clustering

I_{U}

, welfare

W

P_{\text{up}}^{C}

P_{\text{stay}|\text{rich}}^{C}

, and the mean absolute cooperator bias.

condition	seed	coherent	median $\|v\|$	$\|\chi_{C}\|_{1600}$	$I_{U}^{200}$	$I_{U}^{1400\text{--}1600}$	$W^{200}$	$W^{1400\text{--}1600}$
only C learns, $(0.05,0)$	0	0/71	0.03	0.00	24.0	24.4	1782	1757
	1	0/71	0.03	0.00	25.7	24.8	1678	1728
	2	0/71	0.03	0.01	23.3	25.5	1706	1707
both learn, $(0.05,0.05)$	0	0/71	0.11	0.00	7.5	20.4	2042	1564
	1	1/71	0.12	0.01	4.7	21.1	2120	1588
	2	0/71	0.13	0.04	9.7	19.4	2004	1610

Appendix

Table 27: Longer training, per seed, for the runs of Figure 12 with

81

frozen checkpoints every

20

episodes. Coherent counts the checkpoints at or after episode

200

that travel coherently, out of

71

. The table also gives the median

|v|

over the same checkpoints and the number of reversals, which are sign changes of

\bar{v}

along the sequence of coherent checkpoints at or after episode

200

. The last columns give

|\chi_{C}|

at episode

1600

, and

I_{U}

and

W

at episode

200

and averaged over episodes

1400

–

1600

seed	$W$	$B$	$K$	$\Pi_{C}$	$\Pi_{D}$	$\varphi_{C}$	$\varphi_{D}$	$n_{C}$	$n_{D}$
0	2019	3038	571	3.47	2.42	0.79	0.49	21.2	10.1
1	2040	3037	549	3.47	2.53	0.78	0.51	20.0	10.4
2	2033	3035	554	3.48	2.47	0.78	0.50	20.3	10.3
3	1978	3048	622	3.45	2.24	0.80	0.46	23.4	10.2
4	2063	3040	529	3.58	2.41	0.79	0.48	19.3	9.9
5	2018	3039	573	3.48	2.39	0.79	0.48	21.3	10.0

Appendix

Table 28: Welfare decomposition per seed, cooperators co-trained, defectors random:

W=B-N_{C}c-\mathcal{K}

seed	$W$	$B$	$K$	$\Pi_{C}$	$\Pi_{D}$	$\varphi_{C}$	$\varphi_{D}$	$n_{C}$	$n_{D}$
0	1954	3002	600	2.95	3.29	0.71	0.67	20.6	14.5
1	2004	3000	548	3.04	3.34	0.71	0.68	18.4	14.2
2	2141	2998	410	3.27	3.53	0.70	0.69	13.3	11.7
3	1803	3005	754	2.71	3.08	0.72	0.66	26.3	17.2
4	2005	2999	547	3.05	3.33	0.71	0.67	18.3	14.2
5	1851	3003	703	2.79	3.13	0.71	0.66	24.3	16.6

Appendix

Table 29: Welfare decomposition per seed, cooperators co-trained, defectors learned:

W=B-N_{C}c-\mathcal{K}

seed	$W$	$B$	$K$	$\Pi_{C}$	$\Pi_{D}$	$\varphi_{C}$	$\varphi_{D}$	$n_{C}$	$n_{D}$
0	1813	3062	800	3.30	1.74	0.84	0.37	31.6	9.7
1	1792	3054	814	3.17	1.93	0.82	0.41	31.8	10.5
2	1688	3070	933	3.14	1.46	0.86	0.33	37.2	10.4
3	1715	3072	909	3.18	1.51	0.86	0.34	36.2	10.3
4	1801	3069	821	3.39	1.47	0.86	0.33	32.4	9.8
5	1777	3072	847	3.32	1.51	0.86	0.33	33.7	9.6

Appendix

Table 30: Welfare decomposition per seed, only cooperators learn:

W=B-N_{C}c-\mathcal{K}

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ random	learned	$\rho$ random	learned	$W$ random	learned
0	0.34	0.33	0.94	28.89	0.33	0.81	2199	1682
1	0.36	0.34	0.79	23.54	0.42	0.83	2199	1761
2	0.32	0.33	0.80	33.50	0.42	0.81	2198	1622
3	0.38	0.33	0.95	25.03	0.28	0.83	2197	1742
4	0.33	0.33	1.05	31.50	0.27	0.79	2199	1647
5	0.35	0.33	1.04	35.26	0.30	0.80	2201	1560

Appendix

Table 31: Robustness, relative-only observation (225 states), only cooperators learn, per seed.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ random	learned	$\rho$ random	learned	$W$ random	learned
0	0.42	0.42	0.94	4.08	0.33	0.85	2199	2107
1	0.42	0.41	0.79	3.91	0.42	0.81	2199	2125
2	0.42	0.41	0.80	4.91	0.42	0.91	2198	2084
3	0.42	0.43	0.95	4.33	0.28	0.85	2197	2088
4	0.41	0.41	1.05	7.19	0.27	0.90	2199	2030
5	0.41	0.42	1.04	5.72	0.30	0.90	2201	2024

Appendix

Table 32: Robustness, relative-only observation (225 states), both learn, per seed.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ random	learned	$\rho$ random	learned	$W$ random	learned
0	0.34	0.37	0.94	0.94	0.33	0.66	2199	2205
1	0.33	0.38	0.79	0.79	0.42	0.65	2199	2206
2	0.33	0.38	0.80	0.80	0.42	0.62	2198	2204
3	0.33	0.38	0.95	0.95	0.28	0.60	2197	2205
4	0.33	0.37	1.05	1.05	0.27	0.65	2199	2205
5	0.33	0.39	1.04	1.04	0.30	0.62	2201	2206

Appendix

Table 33: Robustness, relative-only observation (225 states), only defectors learn, per seed.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ random	learned	$\rho$ random	learned	$W$ random	learned
0	0.40	0.40	0.94	8.65	0.33	0.89	2199	1968
1	0.42	0.42	0.79	4.22	0.42	0.84	2199	2118
2	0.39	0.41	0.80	8.98	0.42	0.91	2198	2005
3	0.42	0.42	0.95	4.29	0.28	0.83	2197	2116
4	0.42	0.42	1.05	4.88	0.27	0.92	2199	2090

Appendix

Table 34: Robustness, neighbor differences plus own occupancy (675 states), both learn, per seed.

seed	$P_{\text{up}}^{C}$	$P_{\text{up}}^{D}$	$I_{U}$ random	learned	$\rho$ random	learned	$W$ random	learned
0	0.37	0.33	0.94	31.31	0.33	0.81	2199	1621
1	0.37	0.33	0.79	28.54	0.42	0.79	2199	1664
2	0.37	0.33	0.80	23.80	0.42	0.85	2198	1781
3	0.38	0.33	0.95	27.79	0.28	0.80	2197	1719
4	0.37	0.33	1.05	23.79	0.27	0.84	2199	1775

Appendix

Table 35: Robustness, neighbor differences plus own occupancy (675 states), only cooperators learn, per seed.

Figure 13: The $15\times 15$ learning-rate map after $200$ training episodes, with ten training seeds per cell, twenty for $\alpha_{D}=0.05$ , and one frozen-policy evaluation per seed. (a) Fraction of seeds that travel coherently. (b) Mean pattern speed $|v|$ . The dashed line is $\alpha_{C}=\alpha_{D}$ . Figure 4 of the main text shows the welfare loss and clustering of the same cells.

Figure 14: Fine sweeps of

\alpha_{C}

at five fixed values of

\alpha_{D}

after

200

training episodes, averaged over ten training seeds and over twenty for

\alpha_{D}=0.05

. The panels show the fraction of coherently traveling seeds, pattern speed, clustering, and welfare. Dashed lines mark random movement.

Figure 15: Training length. Clustering (a,b) and welfare (c,d) of the frozen policies after

50

800

training episodes, averaged over ten training seeds. The dotted line marks

\alpha_{C}=\alpha_{D}

and the dashed line random movement.

	traveling seeds, by $\alpha_{D}$					$I_{U}$ , by $\alpha_{D}$					$W$ , by $\alpha_{D}$
$\alpha_{C}$	0.0125	0.025	0.05	0.1	0.2	0.0125	0.025	0.05	0.1	0.2	0.0125	0.025	0.05	0.1	0.2
0.00625	0/10	0/10	0/20	0/10	0/10	7.1	4.2	3.5	5.7	15.4	2036	2111	2125	2072	1809
0.007015	0/10	0/10	0/20	0/10	0/10	7.8	4.9	3.9	7.3	13.9	2013	2090	2118	2009	1838
0.007875	0/10	0/10	0/20	0/10	0/10	8.5	5.3	3.8	8.6	16.6	2001	2077	2114	1986	1801
0.008839	0/10	0/10	0/20	0/10	0/10	10.9	6.8	5.7	6.1	17.0	1937	2033	2054	2055	1801
0.009921	0/10	0/10	0/20	0/10	0/10	10.6	6.2	5.4	9.0	17.0	1934	2046	2067	1974	1782

Appendix

Table 36: Finer learning-rate sweep with frozen policies after 200 episodes and one evaluation per training seed. The table gives the number of coherently traveling seeds, the mean clustering

I_{U}

, and the mean welfare

W

for

43

values of

\alpha_{C}

at five fixed values of

\alpha_{D}

. Each cell has ten training seeds, and twenty for

\alpha_{D}=0.05

	$\alpha_{D}=0.05$											$\alpha_{D}=0.0125$
	traveling seeds					$I_{U}$						traveling seeds					$I_{U}$
$\alpha_{C}$	50	100	200	400	800	50	100	200	400	800	$\alpha_{C}$	50	100	200	400	800	50	100	200	400	800
0.03536	0	0	0	0	0	12.9	9.0	9.4	10.2	13.6	0.0125	0	0	0	0	0	14.9	11.7	10.7	7.1	6.2
0.03968	0	0	0	0	0	12.3	11.2	8.0	8.8	15.4	0.01403	0	0	0	0	0	13.7	13.1	10.0	9.2	5.8
0.04454	0	0	0	0	0	12.4	8.0	7.7	8.1	15.3	0.01575	0	0	0	0	3	14.8	15.5	7.7	5.0	6.4
0.05	0	0	0	0	0	12.0	8.9	7.9	7.8	15.7	0.01768	0	0	0	1	3	15.3	13.1	10.0	7.1	7.4

Appendix

Table 37: Training length. The table gives the number of training seeds, out of ten, whose frozen policies travel coherently at the checkpoints after 50, 100, 200, 400, and 800 episodes of one 800-episode run per seed. It also gives the mean clustering

I_{U}

at the same checkpoints.

	original reward			with toll
$(\alpha_{C},\alpha_{D})$	$W$	$I_{U}$	traveling	$W$	$I_{U}$	traveling	$\Delta W$	95% interval	recovered
$(0.05,0.05)$	1977	7.9	0/10	2203	1.06	0/10	226	[170, 285]	102%
$(0.05,0.0125)$	1599	22.6	10/10	2193	1.45	0/10	594	[491, 701]	99%
$(0.2,0.00625)$	781	49.2	10/10	1617	20.83	8/10	837	[598, 1053]	59%
$(0.1,0.00625)$	893	45.1	10/10	2042	6.50	3/10	1150	[960, 1330]	88%
$(0.1414,0.008839)$	1106	35.9	10/10	2020	7.34	4/10	914	[731, 1114]	84%

Appendix

Table 38: Crowding toll at further rate pairs. All runs use 200 episodes and ten training seeds and are evaluated with the original payoff on the held-out placement of each seed, with

W

averaged over snapshots as in Table 8 . The first row repeats the equal-rate condition of the main text.

\Delta W

is the paired difference with its

95\%

bootstrap interval over seeds. Recovered is

(W_{\mathrm{toll}}-W)/(W_{\mathrm{random}}-W)

with

W_{\mathrm{random}}=2199

Explore similar work

The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection
May 18, 2026Benedict Russell, Chin-wing Leung, Paolo TurriniSocial Dilemmas Policy Gradient
Mathematics Institute Warwick University · Department of Computer Science Warwick University
Equilibrium stability as a driver of cooperation among Q-learners
Jul 15, 2026Janusz M. Meylahn, Maximilian SchäferGame-Playing Agents Repeated Games
University of Twente, Department of Applied Mathematics, Drienerloolaan 5, 7522 NB Enschede, The Netherlands. · Institut Mines-Télécom Business School, Department of Data Analytics, Economics, and Finance. 9 Rue Charles Fourier, 91000 Evry, France.
Quantum Frog: Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game
Apr 22, 2026Saad MankariousMulti-Agent Coordination Game-Playing Agents

May 18, 2026, cs.MAJ/K move · Enter open · S save

The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection

Benedict Russell, Chin-wing Leung, Paolo Turrini

Mathematics Institute Warwick University · Department of Computer Science Warwick University

Social Dilemmas Policy Gradient Game Theory Evolutionary Game Theory Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

In social dilemmas self-interested learning agents face the choice between the societal benefit of cooperation and the immediate reward of defection. Significant evidence exists on the benefits of assortment mechanisms such as partner selection for the emergence of cooperation, but this is largely available through agent-based simulations. In this paper, we provide an analytical solution to the problem, studying the policy-gradient dynamics in a multi-agent environment with partner selection. We show how partner selection changes the opponent distribution and hence the reward landscape, and prove this promotes cooperation under simple rules known from the literature. In particular, we find that population variance is a necessary condition for cooperation to emerge. Using a two-dimensional Wiener process, we extend the dynamics to capture the stochastic effects of partner selection and the resulting opponent distribution. We derive a sufficient condition for the population to be cooperation-promoting and prove the existence of a stationary distribution. Simulations confirm that the stochastic model accurately captures the policy-gradient dynamics and clarifies how the learning rate affects the emergence of cooperation.

Abstract

Explore similar work

The Dynamics of Policy Gradient in Social Dilemmas with Partner Selection

Equilibrium stability as a driver of cooperation among Q-learners

Quantum Frog: Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game