Discovering Learning Rules that Shape Others

Hong Jun Lee; Ziyan Wang; Jun Wang; Stefanos Leonardos; Yali Du

cs.LGNov 5, 2026

Discovering Learning Rules that Shape Others

Authors: Hong Jun Lee, Ziyan Wang, Jun Wang, Stefanos Leonardos, Yali Du

Organizations: King’s College London London, United Kingdom · Harvard University Cambridge, MA, United States · University College London London, United Kingdom

Abstract

In general-sum games, self-interested learners can reach outcomes that are poor for everyone, which has motivated methods that shape the opponent's learning. Methods like Learning with Opponent-Learning Awareness (LOLA), which differentiates through the opponent's learning process, and Model-Free Opponent Shaping (M-FOS), which uses model-free optimisation methods for long-horizon opponent shaping, have enabled mutual cooperation and exploitation of learners. However, these methods require access to the opponent's parameters, policy, rewards or learning rule during deployment. In single-agent reinforcement learning, update rules discovered by meta-learning have achieved state-of-the-art results (e.g. DiscoRL). These rules are separate from the policy they train and read only the agent's own experience. We hereby extend this framework to general-sum settings by introducing Discovered Update Rule for Opponent Shaping (DUOS). To the best of our knowledge, this is the first method to discover an update rule for an agent in general-sum games. DUOS meta-trains the rule to maximise the agent's expected return over its lifetime, and updates an inner network using the agent's own experience. We show that our discovered rule exhibits behaviour consistent with opponent shaping without reading opponent policies or parameters, and ranks 1st in normalised mean score and 1st in self-play among tournament entrants.

Figures & tables

Figure 1. Overview of \method . After every match, the meta-network $\eta$ reads the agent’s experience $\mathcal{D}^{i}_{k}$ and outputs log-odds changes $y_{k}$ , to which the inner network $\theta$ is fitted. The opponent’s policy, parameters, learning rule and payoffs are never observed. Meta-training perturbs $y_{k}$ with Gaussian noise and updates $\eta$ with a likelihood-ratio gradient whose baseline $b_{k}$ comes from lifetimes against the same opponent (Section 4 ). A diagram of DUOS in three parts. Centre: in each match of a lifetime, the agent's own experience (its states, own and opponent actions, own rewards and its payoff matrix) feeds the meta-network, a GRU and an MLP, which outputs log-odds changes. The inner network is fitted to them and gives the policy for the next match. Right: the opponent, whose policy, parameters, learning rule and payoffs DUOS does not observe. Bottom: meta-training adds Gaussian noise to the log-odds changes and updates the meta-network with a likelihood-ratio gradient whose baseline comes from four lifetimes against the same opponent, drawn from a training mixture of self-play, naive learners, always-cooperate and always-defect, plus exploiters.

Figure 2. IPD learning trajectories over 50 matches. Left: rewards of \method against a naive learner (NL). Centre: joint-action frequencies against NL ( \method ’s action first, C: cooperation, D: defection). Right: mutual cooperation in self-play, within and across training seeds. Curves average five \method seeds. Shaded bands show 95%

t

-intervals. Three line charts over the 50 matches of an IPD lifetime. Left: against a naive learner, the reward of DUOS rises to about 4.4 per round within about 20 matches, while the naive learner's reward falls to about 0.6. Centre: the share of rounds in which DUOS defects while the naive learner cooperates rises to 77 percent. Mutual cooperation settles at 16 percent and mutual defection at 7 percent. Right: in self-play, the share of mutually cooperative rounds rises from about 0.25 to 1 within about 15 matches, for copies trained with the same or different seeds.

	Cooperate	Defect
Cooperate	$(3,3)$	$(0,5)$
Defect	$(5,0)$	$(1,1)$

Table 1. Payoff matrix for each

2\times 2

game.

Figure 3. Tournament scores after 50 matches (reward per round) or 16 episodes for Coin Game (reward per step). Self-play is excluded. Top row: \method trained on each game. Bottom row: \method trained on IPD playing zero-shot. Five dot plots of tournament scores, one per game, with the methods sorted by score and DUOS highlighted. Error bars show one standard error over training seeds. Top row, DUOS trained in each game: it ranks first in IPD (2.18 per round), second in IMP (0.07, after SHAPER) and third in Coin Game (0.048 per step, after SHAPER and M-FOS). Bottom row, DUOS playing its IPD rule zero-shot: third in Chicken (1.52) and fifth in Stag Hunt (3.42).

Method	IPD	IMP	Chicken	Stag Hunt	Coin	Matrix mean	Five-game mean	Rank
\method (ours)	1.000	0.841	0.829	0.727	0.680	0.849	0.815 [0.78, 0.85]	1
SHAPER	0.631	1.000	0.032	0.902	1.000	0.641	0.713 [0.64, 0.78]	2
M-FOS	0.583	0.646	0.000	1.000	0.844	0.557	0.614 [0.57, 0.66]	3
NL	0.150	0.000	0.868	0.637	0.302	0.414	0.391 [0.35, 0.43]	4
LOLA-OM	0.659	0.350	0.690	0.868	–	0.642	–	–
POLA-OM*	0.386	0.708	0.005	0.925	–	0.506	–	–

Table 2. Normalised tournament scores. Each game’s scores are normalised in accordance with the scoring criteria (1 = best, 0 = worst). Matrix mean: four

2\times 2

matrix games. Five-game mean: all five games for the methods that play all five (95% seed-bootstrap intervals). Rank: by five-game mean. NL in Coin Game is a PPO learner ( Schulman et al., 2017 ) . Bold indicates the best result in each column.

Method	IPD	Chicken	Stag Hunt	Coin	Matrix mean	Four-game mean	Rank
\method (ours)	1.000	1.000	0.883	1.000	0.961	0.971 [0.969, 0.973]	1
Reciprocator	0.621	0.804	0.000	0.954	0.475	0.595 [0.590, 0.601]	2
NL	0.031	0.848	0.101	0.900	0.327	0.470 [0.465, 0.476]	3
M-FOS	0.005	0.073	0.970	0.774	0.349	0.456 [0.396, 0.499]	4
SHAPER	0.327	0.000	1.000	0.000	0.442	0.332 [0.274, 0.364]	5
LOLA-OM	0.428	0.516	0.554	–	0.499	–	–

Table 3. Normalised self-play scores: each method’s mean own reward against a same-seed copy of itself per round, averaged over a lifetime of 50 matches (per step over 16 episodes in Coin Game). Each game’s scores are normalised in accordance with the scoring criteria. IMP is omitted because it is zero-sum. ‡ Outside the tournament (two learners of each run play each other).

Figure 4. Learning curves of the variants trained for self-play. (a) IPD: mean over 5 seeds (s.e.

\leq 0.009

). (b) Coin Game (3 seeds), reward per step. (c) \method (self-play) in the three matrix games with normalised reward: reward per round divided by the game’s mutual-cooperation payoff (3, 3 and 4), s.e.

\leq 0.007

. Three line charts. (a) IPD: the self-play reward of DUOS rises from 2.25 to 3 within about ten matches. M-FOS (meta-self-play) stays just below 3 from the first match, and its noise-off version is at 3 throughout. (b) Coin Game: DUOS rises from 0 to about 0.23 over 16 episodes while M-FOS stays flat near 0.07. Thin lines show the individual seeds. (c) DUOS (self-play) reaches the mutual-cooperation payoff within about ten matches in IPD, Chicken and Stag Hunt.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Row vs. column	M-FOS	M-FOS-H	SHAPER	COALA-PG*	LOLA-OM	POLA-OM*	NL	Reciprocator	\method	Mean score	Rank
M-FOS	–	1.05	1.42	1.25	1.66	1.43	3.88	2.61	2.00	1.91	4
M-FOS-H	1.04	–	1.38	1.26	1.36	1.11	3.13	2.45	2.09	1.73	6
SHAPER	1.35	1.29	–	1.34	1.92	1.61	3.53	2.39	2.13	1.94	3
COALA-PG*	1.18	1.13	1.36	–	1.46	1.36	2.02	2.18	1.62	1.54	9
LOLA-OM	1.53	1.31	1.77	1.37	–	1.34	2.90	2.72	2.76	1.96	2
POLA-OM*	1.36	1.09	1.59	1.36	1.39	–	3.75	2.28	1.48	1.79	5

Appendix

Table S1. IPD tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method. Self-play is excluded from the mean score. \method ’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. \method ’s row is in bold.

Row vs. column	M-FOS	M-FOS-H	SHAPER	LOLA-OM	POLA-OM*	NL	\method	Mean score	Rank
M-FOS	–	$-$ 0.04	$-$ 0.35	0.16	$-$ 0.12	0.47	$-$ 0.06	0.01	5
M-FOS-H	0.04	–	$-$ 0.15	0.04	$-$ 0.05	0.40	$-$ 0.01	0.04	3
SHAPER	0.35	0.15	–	0.11	0.01	0.13	$-$ 0.01	0.12	1
LOLA-OM	$-$ 0.16	$-$ 0.04	$-$ 0.11	–	$-$ 0.02	$-$ 0.02	$-$ 0.16	$-$ 0.08	6
POLA-OM*	0.12	0.05	$-$ 0.01	0.02	–	0.00	0.00	0.03	4
NL	$-$ 0.47	$-$ 0.40	$-$ 0.13	0.02	0.00	–	$-$ 0.19	$-$ 0.20	7

Appendix

Table S2. IMP tournament after 50 matches. Each entry is the row method’s mean reward per round against the column method, averaged over the two seats (so the matrix is antisymmetric). Self-play is excluded from the mean score. \method ’s diagonal entry pools same-seed and cross-seed pairings of its training seeds. \method ’s row is in bold.

Row vs. column	M-FOS	SHAPER	LOLA-OM	POLA-OM*	NL	Reciprocator	\method	Mean score	Rank
M-FOS	–	0.06	0.38	0.15	3.91	3.23	0.30	1.34	7
SHAPER	0.26	–	0.23	0.11	3.92	3.29	0.27	1.35	5
LOLA-OM	0.11	0.06	–	0.40	3.18	2.65	2.54	1.49	4
POLA-OM*	0.09	0.03	0.51	–	3.87	2.83	0.73	1.34	6
NL	0.98	0.98	2.34	1.04	–	2.08	1.74	1.53	2
Reciprocator	0.82	0.82	2.36	0.81	2.69	–	1.83	1.56	1

Appendix

Table S3. Chicken tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. \method plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.

Row vs. column	M-FOS	SHAPER	LOLA-OM	POLA-OM*	NL	Reciprocator	\method	Mean score	Rank
M-FOS	–	3.97	3.77	3.89	3.93	2.91	3.57	3.67	1
SHAPER	3.97	–	3.63	3.91	3.82	2.82	3.34	3.58	3
LOLA-OM	3.83	3.68	–	3.51	3.74	3.07	3.47	3.55	4
POLA-OM*	3.92	3.93	3.53	–	3.76	2.91	3.57	3.60	2
NL	3.94	3.85	3.73	3.75	–	2.26	2.49	3.34	6
Reciprocator	3.33	3.12	3.15	2.95	1.92	–	2.06	2.76	7

Appendix

Table S4. Stag Hunt tournament after 50 matches, with SHAPER: the row method’s mean reward per round against each column method. \method plays its IPD rule zero-shot. Its self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score.

Row vs. column	M-FOS	SHAPER	NL (PPO)	Reciprocator	\method	Mean score	Rank
M-FOS	–	$-$ 0.298	0.411	0.567	0.097	0.194	2
SHAPER	0.032	–	0.534	0.643	0.127	0.334	1
NL (PPO)	$-$ 0.447	$-$ 0.754	–	0.259	$-$ 0.216	$-$ 0.289	4
Reciprocator	$-$ 0.618	$-$ 0.865	$-$ 0.272	–	$-$ 0.482	$-$ 0.559	5
\method (ours)	$-$ 0.119	$-$ 0.373	0.230	0.454	0.014	0.048	3

Appendix

Table S5. Coin Game tournament after 16 episodes: the row method’s mean reward per step against each column method. \method ’s self-play (diagonal) entry pools same-seed and cross-seed pairings of its training seeds and is excluded from the score. NL is M-FOS’s PPO naive learner. M-FOS (stronger than in its paper) and Reciprocator are flagged (Appendix C ).

Method	IPD	IMP	Chicken	Stag Hunt	Coin	Matrix mean	Five-game mean
\method (ours)	1.000	0.841	1.000	0.787	0.680	0.907	0.862 [0.84, 0.88]
M-FOS	0.583	0.646	0.000	1.000	0.844	0.557	0.614 [0.57, 0.66]
NL	0.150	0.000	0.196	0.596	0.302	0.236	0.249 [0.21, 0.29]
LOLA-OM	0.659	0.350	0.842	0.905	–	0.689	–
POLA-OM*	0.386	0.708	0.040	0.920	–	0.514	–
SHAPER	0.631	1.000	–	–	1.000	–	–

Appendix

Table S6. Normalised tournament scores in which SHAPER plays only IPD, IMP and Coin Game. Columns are as in Table 2 , without the rank column. SHAPER was only added to Chicken and Stag Hunt after these scores were computed.

	Chicken		Stag Hunt
Method	without	with	without	with
M-FOS	1.60 (6)	1.34 (7)	3.61 (1)	3.67 (1)
SHAPER	–	1.35 (5)	–	3.58 (3)
LOLA-OM	1.78 (2)	1.49 (4)	3.52 (3)	3.55 (4)
POLA-OM*	1.60 (5)	1.34 (6)	3.54 (2)	3.60 (2)
NL	1.64 (4)	1.53 (2)	3.24 (5)	3.34 (6)

Appendix

Table S7. Chicken and Stag Hunt scores (rank) after 50 matches, without SHAPER (original field) and with SHAPER.

Game	$K$	With PSRO	Pre-PSRO	Difference
IPD	50	2.182 (1)	2.162 (1)	+0.020 [ $-$ 0.151, +0.191]
	10	1.945 (1)	1.922 (3)	+0.023 [ $-$ 0.024, +0.069]
IMP	50	0.073 (2)	0.029 (4)	+0.043 [+0.022, +0.065]
	10	0.054 (2)	0.031 (2)	+0.023 [+0.009, +0.037]
Chicken	50	1.520 (3)	1.531 (2)	$-$ 0.011 [ $-$ 0.055, +0.034]
	10	1.362 (2)	1.367 (2)	$-$ 0.005 [ $-$ 0.032, +0.023]

Appendix

Table S8. Mean score (rank in its own field) of \method with and without PSRO after

K

matches, and their difference (with PSRO minus pre-PSRO) paired by seed with a 95%

t

-interval. Bold: the interval excludes 0. Chicken and Stag Hunt: fields with SHAPER.

Baseline	Check	Target	Ours	Outcome	Tournament
M-FOS, IPD	Table 8 (authors’ payoffs), 2 seeds	M-FOS $-0.51\pm 0.1$ , NL $-2.14\pm 0.2$	M-FOS $-0.476$ , $-0.474$ , NL $-2.174$ , $-2.176$	pass	entrant
M-FOS, IMP	Table 5, authors’ code (B) vs ours at lr 0.002 (C)	0.20, B $\geq 0.18$ , $\|C-B\|\leq 0.03$	B 0.194, C 0.194	pass	entrant (lr 0.002)
M-FOS, Chicken	Table 12	M-FOS above NL	M-FOS 3.94, NL 0.99 (5/5 seeds)	pass	entrant
M-FOS, Stag Hunt	no paper result	n/r	M-FOS 3.93, NL 3.95	–	entrant
M-FOS, meta-self-play	self-play, $K=50$	n/r	IPD 2.988, IMP 0.012	–	self-play only
SHAPER, IPD	recovery, 3 seeds	3.15	2.967	fail	entrant (flagged)

Appendix

Table S9. Baseline fidelity in the matrix games (I): reproduction checks against each baseline’s paper. Target: the target or bar set before the check, and n/r means no target.

Baseline	Check	Target	Ours	Outcome	Tournament
POLA-OM*	POLA’s steps in our payoff units	$5/c$ , $0.3/c$ , $0.1c$	$c=5,2,4,4$	–	entrant
LOLA-OM	LOLA’s Table 3 (exact LOLA, authors’ payoffs)	n/r (paper $-1.06$ , $-1.98$ )	LOLA $-1.04$ , NL $-1.98$	match	entrant
POLA-OM*	Outer POLA vs authors’ code, Fig. 3 and Table 3	n/r	within $10^{-9}$ , both reproduced	match	entrant
Reciprocator, IPD	shaping gate, 8 seeds	$\geq 5/8$	paper formula 0/8, released code 8/8	pass (code)	entrant (code)
COALA-PG*, pure shaping	Fig. 5A	return $\geq 0.40$	1.376, 1.401	pass	entrant (IPD)
COALA-PG*, mixed pool	Fig. 5C	recovery $\geq 0.5$	0.035	fail	–

Appendix

Table S10. Baseline fidelity in the matrix games (II), as in Table S9 . Counts are seeds meeting the condition, and

c

is the payoff-range factor.

Baseline	Check	Target	Ours	Outcome	Tournament
M-FOS, authors’ code	updates 951–1000, 3 seeds	M-FOS $\geq 33.2$ , NL $\leq-18.5$ in $\geq 2/3$ seeds	70.89 / $-75.00$ , 2/3 seeds	pass (stronger)	–
M-FOS, pax	same, not gated	n/r	M-FOS 113.55, 105.21, 75.94	–	entrant (flagged)
M-FOS, meta-self-play	authors’ code, self-play, 3 seeds	$\geq 15.4$ in $\geq 2/3$ seeds	8.74, 14.44, 22.42	fail	–
M-FOS, meta-self-play, pax	self-play, 3 seeds, not gated	n/r	29.95, 25.98, $-3.48$	–	self-play only
SHAPER	paper’s ES budget	n/r	3/3 seeds completed	–	entrant
Reciprocator	NL own-coin rate, 4 seeds	$\geq 0.52$ in $\geq 3/4$ seeds, mean $\geq 0.56$ , controls $\leq 0.52$	0.617 (4/4), controls 0.473, 0.531	uninformative	entrant (flagged)

Appendix

Table S11. Baseline fidelity in Coin Game, as in Table S9 . M-FOS returns are in the authors’ units, summed over 240 paid steps (M-FOS / NL).

Hyperparameter	Value
Rounds per match $T$	20
Matches per training lifetime $N$	100
Objective horizons $h$	10, 25, 50, 100
Discount $\gamma$	0.96
Meta-network GRU units	32
Meta-network MLP hidden units	64, 64

Appendix

Table S12. Hyperparameters of \method in the matrix games.

Hyperparameter	Value
Steps per episode	16
Episodes per lifetime	16
Parallel games	512
Inner network hidden units	32, 32
Adam second moment	from the $\alpha\equiv 1$ gradient
Advantage $\hat{A}_{t}$	return minus ridge baseline

Appendix

Table S13. Hyperparameters of \method in Coin Game.

Hyperparameter	Value
Actor and critic hidden units	256
M-FOS-H GRU units	16
Learning rate	$2\times 10^{-4}$
Learning rate, IMP	$2\times 10^{-3}$ †
PPO epochs	4
PPO clip	0.2

Appendix

Table S14. Hyperparameters of M-FOS and M-FOS-H in the matrix games. † The paper’s Table 14 gives

2\times 10^{-4}

, at which our IMP runs reach 0.134 (paper 0.20). The authors’ naive-learner script uses

2\times 10^{-3}

, which reaches 0.194 (Table S9 ). The meta-self-play curriculum keeps

2\times 10^{-4}

Hyperparameter	Value
Convolution channels	16, 16
Linear and GRU units	16, 16
Learning rate	$2\times 10^{-4}$
PPO epochs	16
PPO clip	0.2
Entropy coefficient	0.01

Appendix

Table S15. Hyperparameters of M-FOS in Coin Game.

Hyperparameter	Value
GRU units	16 †
ES population	1,000
Generations	1,000 †
ES learning rate	0.1 †
Perturbation scale	0.04
Opponents per candidate	10

Appendix

Table S16. Hyperparameters of SHAPER in the matrix games. † The paper’s Table 11 (IPD) lists a GRU of 25 units, 5,000 generations and an ES learning rate of 0.01. We use the authors’ configuration files (whose GRU size matches the parameter count in the paper’s Table 20), as in Coin Game (Table S17 ).

Hyperparameter	Value
Linear and GRU units	16, 16
ES population	4,000
Generations	3,000
ES learning rate	0.1 †
Perturbation scale	0.04
L2 penalty	0.1

Appendix

Table S17. Hyperparameters of SHAPER in Coin Game. † The paper’s table lists 0.01. We use the authors’ configuration file.

Hyperparameter	Value
LOLA-OM step size	1
LOLA-OM look-ahead step	1
LOLA-OM discount $\gamma$	0.96, 0.9, 0.96, 0.96
POLA-OM* inner step ( $5/c$ )	1, 2.5, 1.25, 1.25
POLA-OM* outer step ( $0.3/c$ )	0.06, 0.15, 0.075, 0.075
POLA-OM* KL weight ( $0.1c$ )	0.5, 0.2, 0.4, 0.4

Appendix

Table S18. Hyperparameters of LOLA-OM and POLA-OM*. Lists give IPD, IMP, Chicken and Stag Hunt. Here

c

is the payoff-range factor. Step sizes are on the unnormalised value

V/(1-\gamma)

, as in Table S21 .

Hyperparameter	Value
Hawk width, heads	32, 2
PPO iterations	3,000
Meta-batch	128
Inner batch	16
Matches $\times$ rounds per lifetime (meta-episode)	$20\times 10$
Optimiser steps per iteration	1 †

Appendix

Table S19. Hyperparameters of COALA-PG* in IPD. † The paper’s Table 4 gives 4 epochs

\times

2 minibatches. A speed check found one step per iteration to match the pace of its Fig. 5A.

Matrix games
Hyperparameter	Value
Reciprocal reward weight	5.0
Hidden units	2
Learning rate	0.005
Discount $\gamma$	0.96
PPO epochs, clip, entropy	10, 0.1, 0.02

Appendix

Table S20. Hyperparameters of Reciprocator.

Hyperparameter	Value
Matrix games (NL)
Step size on $V/(1-\gamma)$ (25 on $V$ of Eq. ( 1 ))	1
Discount $\gamma$	0.96
Coin Game (PPO NL)
Convolution channels	16, 16
Linear units	16

Appendix

Table S21. Hyperparameters of the naive learners.

Setting	Value
Matrix games
Matches per lifetime $K$	50
Matches per lifetime $K$ , secondary	10
Rounds per match	20
Lifetimes per pair of training seeds (IMP: per seat order)	128
Trained seeds ( \method , M-FOS, M-FOS-H, COALA-PG*)	5

Appendix

Table S22. Tournament protocol. Intervals for normalised scores are 95% percentile bootstrap intervals over 2,000 resamples (10,000 for Table 3 ) of each learned method’s training seeds within each game, recomputing scores, ranks and the min–max normalisation in every resample. Lifetimes are not resampled.

Matrix games
Method	Own trajectory	Opponent’s actions	Opponent’s policy	Opponent’s payoffs	Assumed learning rule	Extra
\method	last match a	observed	no	no	no	own payoffs, own table
M-FOS	no	no	table b	no	no	own table
M-FOS-H	last match c	observed	no	no	no	own table
SHAPER	every round d	observed	no	no	no	none
COALA-PG*	every round d	observed	no	no	no	none

Appendix

Table S23. Information each method uses at deployment.

Opponent	Control	PSRO	$d$ [95% CI]	Seeds PSRO better
M-FOS	$-$ 0.020	+0.124	+0.144 [+0.057, +0.231]	5/5
M-FOS-H	$-$ 0.199	+0.026	+0.225 [+0.145, +0.305]	5/5
SHAPER	$-$ 0.092	+0.019	+0.111 [+0.065, +0.158]	5/5
Pairings won (all 6 IMP opponents)	3/6	6/6

Appendix

Table S24. What PSRO adds: the deployed rule against its pre-PSRO base (control) on the same 5 seeds.

d

: after minus before, with a paired 95%

t

-interval. (a) IMP head-to-head margin after 50 matches. (b) Exploitability: a fresh exploiter’s value against the rule minus the value that a reference strategy (IPD: tit-for-tat, IMP: the Nash player) obtains against the rule. Lower is better. IMP has separate rules for the two seats, so it has 10 runs (2 seats

\times

5 seeds). Its interval pairs the 5 seeds, each averaged over its two seats.

Game	Measure	Trained	Zero-shot	$d$ [95% CI]	Seeds
Chicken	Tournament score	2.105	1.810	+0.295 [+0.218, +0.371]	5/5
	Rank (of 6)	3	1	–	–
	Own return vs M-FOS	0.979	0.108	+0.871 [+0.857, +0.886]	5/5
	Margin vs M-FOS	$-$ 2.893	$-$ 0.195	$-$ 2.697 [ $-$ 2.753, $-$ 2.642]	0/5
	Self-play return	2.903	2.920	$-$ 0.017 [ $-$ 0.033, $-$ 0.001]	0/5
	Exploitability	0.885	0.020	+0.865 [+0.849, +0.880]	0/5

Appendix

Table S25. Zero-shot transfer: the IPD rule played zero-shot compared with rules trained in Chicken and Stag Hunt, each entered in place of it in the same field, after 50 matches (5 seeds, in the field without SHAPER of Table S7 , with the trained rules using the IPD recipe, not retuned).

d

: trained minus zero-shot, with a paired 95%

t

-interval. Seeds: seeds on which the trained rule is better. Self-play return: same-seed pairs from the same tournament runs. Exploitability is as in Table S24 , with tit-for-tat as the reference. A higher score can come with a lower rank because the other entrants’ scores also change: the Chicken-trained rule yields to Straight, which raises M-FOS’s score from 1.60 to 2.31 and POLA-OM*’s from 1.60 to 2.20.

Opponent	IPD	IMP	Chicken	Stag Hunt	Coin
M-FOS	$-$ 0.268 (0/5)	+0.124 (4/5)	$-$ 0.195 (0/5)	+0.141 (5/5)	$-$ 0.216 (0/3)
M-FOS-H	$-$ 0.260 (0/5)	+0.026 (3/5)	–	–	–
SHAPER	$-$ 0.201 (0/5)	+0.019 (4/5)	$-$ 0.206 (0/5)	+0.120 (5/5)	$-$ 0.500 (0/3)
COALA-PG*	$-$ 0.017 (2/5)	–	–	–	–
LOLA-OM	$-$ 0.182 (0/5)	+0.313 (5/5)	$-$ 0.102 (0/5)	+0.092 (5/5)	–
POLA-OM*	$-$ 0.197 (0/5)	+0.004 (4/5)	$-$ 0.219 (0/5)	+0.065 (5/5)	–

Appendix

Table S26. Mean head-to-head margin of \method (its reward minus the opponent’s, per round, or per step in Coin Game) and, in parentheses, the seeds with a positive margin. Matrix games after 50 matches, Coin Game after 16 episodes. –: not in that game.

Measure	Actual	Frozen	$d$ [95% CI]
NL cooperation, last match	0.929 $\pm$ 0.010	0.141 $\pm$ 0.019	$-$ 0.788 [ $-$ 0.823, $-$ 0.754]
\method reward per round, last match	4.387 $\pm$ 0.017	1.371 $\pm$ 0.051	$-$ 3.016 [ $-$ 3.172, $-$ 2.861]
\method reward per round, lifetime	3.961 $\pm$ 0.024	1.421 $\pm$ 0.046	$-$ 2.540 [ $-$ 2.721, $-$ 2.358]
NL reward per round, lifetime	0.900 $\pm$ 0.030	1.145 $\pm$ 0.023	+0.245 [+0.209, +0.281]

Appendix

Table S27. Frozen-table control in IPD against NL, after 50 matches (5 seeds

\times

128 lifetimes). Actual: \method ’s recorded lifetimes. Frozen: the same NL, from the same first-match policy, against \method ’s final (match-50) table held fixed from the first match. Exact expectations over 20-round matches. Mean

\pm

s.e. over seeds.

d

: frozen minus actual, with a paired 95%

t

-interval (all 5 seeds have the same sign).

Game-Theoretic RL

See all 34 papers

From Single-Agent Self-Discovery to Multi-Agent Adaptation: Scaling Laws of Partner Feedback and Convention Rigidity in Meta Reinforcement Learning Algorithms
Nov 1, 2026Haomin Luo, Ersha LiMulti-Agent Reinforcement Learning Meta-Reinforcement Learning
University of Cambridge · Models2 AI
Differentiable Belief-based Opponent Shaping
May 27, 2026Aarav G Sane, Karthik Sivachandran, Rohan PalejaMulti-Agent Coordination Imperfect-Information Games
Department of Computer Science Purdue University
When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability
Oct 13, 2026Marcel Petran, Stefan EdelkampOpponent Modeling Multi-Agent Reinforcement Learning
Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Nov 1, 2026J/K move · Enter open · S save

From Single-Agent Self-Discovery to Multi-Agent Adaptation: Scaling Laws of Partner Feedback and Convention Rigidity in Meta Reinforcement Learning Algorithms

Haomin Luo, Ersha Li

University of Cambridge · Models2 AI

Multi-Agent Reinforcement Learning Meta-Reinforcement Learning Multi-Agent Systems Cooperative MARL Multi-Agent Coordination

Open paper AskPDF arXiv

A key milestone toward Recursive Self-Improvement (RSI) is replacing hand-crafted optimizers with meta-learned update rules (such as Disco103 and OPEN) that discover how agents learn. Yet when a self-discovered RL model trains with adapting teammates, does co-evolution sustain capability gains or trap agents in brittle conventions? Across 1,800+ runs in continuous collective robotics (VMAS) and discrete coordination (Overcooked), we establish five laws of multi-agent adaptation. First, four-cell counterfactual partner swaps expose an evaluation illusion: apparent team progress often masks focal regression (+1.066 team gain concealing -0.119 focal loss) and inflates transfer by +0.649 to +1.210. Second, a 288-run physical-coupling sweep (

α\in [0, 1.5]

) reveals a linear scaling law (

β= +0.255

p = 0.000488

, Spearman

ρ= 1.0

): live reciprocal feedback keeps Disco103 invariant (AUC 0.297--0.322), whereas severing feedback triggers a monotonic collapse (-0.172 to -0.490), with OPEN establishing a sensitivity spectrum (Disco103 > OPEN > MAPPO). Third, extending navigation training eightfold (1.05M steps) enables Disco103 to overtake MAPPO (+0.087 team return, +0.133 bank gain), while decoupled training boosts cross-partner completion by +10.16% (56.25% vs. 46.09%). Fourth, in Overcooked, MAPPO's self-play conventions induce a 28.13% cross-partner collision rate, whereas Disco103 cuts collisions to 12.77% and achieves 3.33x higher bank return (9.96 vs. 2.99; net gain +6.43), further improved by decoupled meta-adaptation (+1.39 with scripted partners). Finally, perturbation pulses (

p=0.00098

), auxiliary entropy diagnostics (8.485 nats), and 48 multi-lifetime meta-evolved rules (

L_2 \in [2.11, 4.91]

) trace these dynamics to partner-gradient blindness and auxiliary drift, informing multi-agent self-discoverd algorithms.

Abstract

Game-Theoretic RL

Explore similar work

From Single-Agent Self-Discovery to Multi-Agent Adaptation: Scaling Laws of Partner Feedback and Convention Rigidity in Meta Reinforcement Learning Algorithms

Differentiable Belief-based Opponent Shaping

When Does Opponent Modeling Pay Off? Spatial Inference Stabilises Reinforcement Learning under Partial Observability