From Single-Agent Self-Discovery to Multi-Agent Adaptation: Scaling Laws of Partner Feedback and Convention Rigidity in Meta Reinforcement Learning Algorithms

Haomin Luo; Ersha Li

cs.MANov 1, 2026

From Single-Agent Self-Discovery to Multi-Agent Adaptation: Scaling Laws of Partner Feedback and Convention Rigidity in Meta Reinforcement Learning Algorithms

Authors: Haomin Luo, Ersha Li

Organizations: University of Cambridge · Models2 AI

Abstract

A key milestone toward Recursive Self-Improvement (RSI) is replacing hand-crafted optimizers with meta-learned update rules (such as Disco103 and OPEN) that discover how agents learn. Yet when a self-discovered RL model trains with adapting teammates, does co-evolution sustain capability gains or trap agents in brittle conventions? Across 1,800+ runs in continuous collective robotics (VMAS) and discrete coordination (Overcooked), we establish five laws of multi-agent adaptation. First, four-cell counterfactual partner swaps expose an evaluation illusion: apparent team progress often masks focal regression (+1.066 team gain concealing -0.119 focal loss) and inflates transfer by +0.649 to +1.210. Second, a 288-run physical-coupling sweep ( $α\in [0, 1.5]$ ) reveals a linear scaling law ( $β= +0.255$ , $p = 0.000488$ , Spearman $ρ= 1.0$ ): live reciprocal feedback keeps Disco103 invariant (AUC 0.297--0.322), whereas severing feedback triggers a monotonic collapse (-0.172 to -0.490), with OPEN establishing a sensitivity spectrum (Disco103 > OPEN > MAPPO). Third, extending navigation training eightfold (1.05M steps) enables Disco103 to overtake MAPPO (+0.087 team return, +0.133 bank gain), while decoupled training boosts cross-partner completion by +10.16% (56.25% vs. 46.09%). Fourth, in Overcooked, MAPPO's self-play conventions induce a 28.13% cross-partner collision rate, whereas Disco103 cuts collisions to 12.77% and achieves 3.33x higher bank return (9.96 vs. 2.99; net gain +6.43), further improved by decoupled meta-adaptation (+1.39 with scripted partners). Finally, perturbation pulses ( $p=0.00098$ ), auxiliary entropy diagnostics (8.485 nats), and 48 multi-lifetime meta-evolved rules ( $L_2 \in [2.11, 4.91]$ ) trace these dynamics to partner-gradient blindness and auxiliary drift, informing multi-agent self-discoverd algorithms.

Figures & tables

Task	Comparator	Full-training mean difference	Last-50 mean difference
simple adversary v3	IPPO	-1.2948 [-1.8708, -0.6955]	-0.6816 [-1.4061, 0.0115]
simple adversary v3	MAPPO	-3.2139 [-3.8083, -2.5618]	-2.0433 [-3.0532, -1.4034]
simple spread v3	IPPO	2.1032 [1.1201, 3.0882]	2.2441 [0.1441, 4.0856]
simple spread v3	MAPPO	-2.9257 [-3.5545, -2.2492]	-3.3035 [-4.6051, -1.8829]
simple tag v3	IPPO	-3.6350 [-3.9168, -3.3962]	-5.8658 [-7.1030, -4.8254]
simple tag v3	MAPPO	-6.4143 [-6.7764, -6.0673]	-9.4879 [-11.6218, -7.5665]

Table 1: Foundational particle-environment baseline comparisons (

n=8

paired seeds per task). Full-training mean episode return differences and final-50-episode mean return differences are reported with pointwise 95% confidence intervals.

Figure 1: Counterfactual partner-routing and source-matching interventions across cooperative benchmarks. Point estimates and 95% paired-block bootstrap confidence intervals (

n=12

paired blocks per task) for scheduled partner reset-switch (left), recent-trajectory replay filtering (center), and single-lifetime source-feedback matching (right) in native task units.

Task	Contrast	n	Estimate [95% CI]	Holm p
signal coordination	switch interaction	12	-0.0050 [-0.0351, 0.0258]	1.0000
signal coordination	recent replay repair	12	0.0235 [0.0142, 0.0334]	0.0010
navigation	switch interaction	12	0.0145 [-0.0427, 0.0842]	1.0000
navigation	recent replay repair	12	-0.0021 [-0.0292, 0.0255]	0.8843
navigation	source feedback matching	12	-0.0138 [-0.0568, 0.0228]	0.8086
overcooked cramped room	source feedback matching	12	0.5260 [-0.6536, 1.6406]	0.8086

Table 2: Primary paired-block confirmation contrasts across counterfactual partner-routing, recent-replay filtering, and single-lifetime source-feedback matching (

n=12

paired blocks per task). Intervals are 95% paired-block bootstrap confidence intervals; Holm

p

-values are adjusted within each two-task family.

Task	Source	Target data	n	Bank gain [95% CI]	Team [95% CI]
navigation	eta0	coupled	12	0.0996 [0.0764, 0.1255]	0.4016 [0.2554, 0.5078]
navigation	eta0	decoupled	12	0.0635 [0.0314, 0.0975]	0.4589 [0.3792, 0.5333]
navigation	coupled	coupled	12	0.0385 [-0.0162, 0.1267]	0.4183 [0.3018, 0.5579]
navigation	coupled	decoupled	12	0.0468 [-0.0049, 0.1275]	0.4494 [0.3645, 0.5547]
navigation	decoupled	coupled	12	0.0109 [0.0022, 0.0195]	0.3773 [0.3481, 0.4034]
navigation	decoupled	decoupled	12	0.0055 [-0.0122, 0.0202]	0.3624 [0.2961, 0.4313]

Table 3: Single-lifetime source-feedback adaptation transfer across navigation and Overcooked (

n=12

independently paired source blocks, two target repeats averaged per block). The unadapted base rule (

\eta_{0}

) provides a shared reference anchor.

Figure 2: Horizon-scaling reversal and decoupled anti-overfitting in cooperative navigation (

1.05\text{M}

steps). Solid trajectories denote coupled partner training; dashed trajectories denote decoupled reference-partner training. Shaded bands indicate pointwise 95% paired-block bootstrap intervals (

n=12

blocks). Extending the training horizon eightfold enables Disco103 to overtake MAPPO in both original-team return (

+0.087

) and held-out bank gain (

+0.133

), while decoupled training elevates Disco103’s cross-partner completion rate by

+10.16

percentage points (

56.25\%

vs.

46.09\%

). Matched short-horizon OPEN trajectories (

131\text{k}

steps) are shown over their evaluated domain.

Figure 3: Divergence between co-adapted self-play score and cross-partner generalization in Overcooked ( cramped_room ). Outcomes across

n=12

paired training blocks at

262,160

steps. While MAPPO achieves the highest original-team self-play return (

18.49

), it suffers the steepest cross-partner bank regression (

-9.81

points) due to rigid trajectory conventions (

28.13\%

collision rate), whereas coupled Disco103 preserves

3.33\times

higher absolute cross-partner return (

9.96

vs.

2.99

; net gain advantage

+6.43

points).

Task	n	Response contrast [95% CI]	97.5% CI	Holm p
wind flocking	12	-0.2336 [-0.3695, -0.1239]	[-0.3914, -0.1115]	0.0010
navigation	12	-0.3345 [-0.4525, -0.2227]	[-0.4708, -0.2075]	0.0010

Table 4: Controlled behavioral-perturbation confirmation contrasts (

\xi=0.01

65{,}656

main-world steps,

n=12

paired blocks per task). Values report Disco103 minus MAPPO in live-minus-restored partner-mediated response (

\Delta_{C}

Task	n	Score [95% CI]	97.5% CI	Holm p	Pass
wind flocking	12	0.0002 [-0.0019, 0.0022]	[-0.0022, 0.0024]	0.8887	No
navigation	12	-0.0016 [-0.0032, 0.0000]	[-0.0035, 0.0003]	0.1768	No

Table 5: Prospective transfer-prediction confirmation outcomes (

n=12

paired blocks per task). Positive scores indicate out-of-sample squared-error reduction when augmenting baseline early competence features with scalar perturbation response in the frozen ridge predictor.

Figure 4: Controlled behavioral perturbation response and prospective transfer prediction. Left: Paired contrasts in learner-mediated partner response (

n=12

confirmation blocks per task,

3,456

branches), demonstrating that Disco103 exhibits a significantly smaller short-run perturbation amplification than MAPPO in both wind flocking (

-0.234,p=0.00098

) and navigation (

-0.334,p=0.00098

). Right: Out-of-sample squared-error reductions on

1.05\text{M}

-step bank gains when augmenting baseline competence features with early scalar response in a frozen ridge predictor (

n=12

confirmation blocks).

Method	Training data	FT minus F0 return [95% CI]
disco103	coupled	-3.3802 [-6.3751, -0.3698]
disco103	decoupled	-8.7240 [-10.3958, -6.7448]
mappo	coupled	-9.8125 [-10.5417, -9.0521]
mappo	decoupled	-9.2083 [-10.4375, -7.8385]
open	coupled	-6.6094 [-10.5783, -2.4219]
open	decoupled	-9.2083 [-12.4010, -5.2135]

Table 6: Cross-partner generalization on the 10-partner held-out evaluation bank in Overcooked ( cramped_room ,

92{,}160

steps,

n=12

paired training blocks). Values report trained-minus-initial focal return (

G_{\mathcal{B}}

) with pointwise 95% block-bootstrap confidence intervals.

Source	Target data	Bank gain [95%]	FT completion [95%]	Completion change [95%]
eta0	coupled	0.3750 [-0.1276, 0.9115]	0.1208 [0.1173, 0.1246]	-0.0046 [-0.0081, -0.0010]
eta0	decoupled	-0.4714 [-0.8594, -0.0911]	0.1132 [0.1100, 0.1161]	-0.0122 [-0.0155, -0.0089]
coupled	coupled	-0.3646 [-1.2969, 0.4089]	0.1148 [0.1046, 0.1229]	-0.0105 [-0.0212, -0.0021]
coupled	decoupled	-1.2344 [-2.5079, -0.1224]	0.1122 [0.1017, 0.1191]	-0.0132 [-0.0237, -0.0065]
decoupled	coupled	0.2057 [-0.0260, 0.4453]	0.1214 [0.1191, 0.1234]	-0.0040 [-0.0068, -0.0016]
decoupled	decoupled	-0.1380 [-0.4427, 0.1302]	0.1208 [0.1181, 0.1232]	-0.0046 [-0.0076, -0.0020]

Table 7: Cross-regime meta-adapted rule transfer in Overcooked across all six source–target feedback combinations at

46{,}080

steps (

n=12

paired source blocks, two target repeats averaged per block, pointwise 95% block-bootstrap intervals).

Figure 5: Cross-regime meta-adapted rule transfer in Overcooked (

432

checkpoint evaluations across

12

paired source blocks). Held-out 10-partner bank return gains (left) and completion rate changes (right) at

131,080

steps relative to initial focal policies. Rules meta-adapted under decoupled source experience and trained with coupled target partners ( decoupled / coupled ) reverse the negative bank return of coupled / coupled (

+0.2057

vs.

-0.3646

), achieving a

+1.3932

gain (

95\%

[0.38,2.46]

) with competent scripted partners.

Alpha	Interaction [95%]	Disco coupled	Disco reference	MAPPO coupled	MAPPO reference
0	0.4845 [0.3873, 0.5754]	0.2969	-0.1722	0.5453	0.5607
0.25	0.4938 [0.4168, 0.5707]	0.3016	-0.1977	0.5395	0.5340
0.5	0.6069 [0.5339, 0.6818]	0.3148	-0.3366	0.5269	0.4825
0.75	0.6648 [0.5793, 0.7538]	0.3218	-0.3474	0.4871	0.4827
1	0.7460 [0.6391, 0.8623]	0.3133	-0.4371	0.4578	0.4534
1.5	0.8366 [0.7514, 0.9552]	0.3023	-0.4903	0.4772	0.5212

Table 8: Parametric physical-coupling sweep across 288 wind-flocking runs (

\alpha\in[0,1.5]

n=12

paired blocks per cell). Columns report the Method

\times

Feedback interaction (with pointwise 95% block-bootstrap intervals) and constituent original-team learning-curve AUCs.

Figure 6: First-principles scaling law of physical coupling across 288 wind-flocking runs (

\alpha\in[0,1.5]

). Left: Original-team learning-curve AUC across physical coupling strength

\alpha

, showing that coupled Disco103 remains invariant across the entire coupling spectrum (

\text{AUC}\approx 0.297\text{--}0.302

) while decoupled Disco103 collapses monotonically from

-0.172

-0.490

. Center: Monotonic growth of the Method

\times

Feedback interaction from

+0.485

(

\alpha=0

) to

+0.837

(

\alpha=1.5

) (Spearman

\rho=1.0

, Page’s trend

p=1.48\times 10^{-8}

). Right: Primary within-block interaction slopes across all

n=12

paired blocks (

\beta=+0.2547

, 95% CI

[0.2099,0.3057]

, exact sign-flip

p=0.000488

Figure 7: Cross-partner bank evaluation across the parametric physical-coupling grid (

576

checkpoint evaluations). Absolute trained-focal return (left) and gain over initial focal policies (right) against the 10-partner evaluation bank across

\alpha\in[0,1.5]

. Coupled training yields strictly positive cross-partner gains for Disco103 across all coupling levels (

+0.127

+0.297

), surpassing coupled MAPPO at

\alpha=0.25

(

+0.246

vs.

-0.008

Task	Horizon	Method / regime	n	Initial $J_{00}$	Final $J_{11}$	Net gain
wind flocking	131,120	open / coupled	12	-0.5570	0.7860	+1.3431
wind flocking	131,120	open / decoupled	12	-0.5570	0.6182	+1.1752
wind flocking	1,048,640	open / coupled	4	-0.5326	1.2308	+1.7634
wind flocking	1,048,640	open / decoupled	4	-0.5326	0.9907	+1.5233
navigation	1,048,640	disco103 / coupled	4	-0.3491	1.7881	+2.1372
navigation	1,048,640	mappo / coupled	4	-0.4091	1.6819	+2.0910

Table 9: Matched-substrate learned-optimizer ( OPEN ) and extended-horizon curriculum comparisons across wind flocking and navigation. Each cell reports initial team return

J_{00}

, final team return

J_{11}

, and net team gain

J_{11}-J_{00}

across paired seed blocks.

Figure 8: Parameter reorganization across 48 multi-lifetime meta-adapted update rules (

512

outer updates across

4

policy lifetimes,

6.84\times 10^{7}

world steps). Euclidean (

L_{2}

) parameter displacement from the base Disco103 rule (

\eta_{0}

) and computational profile across

n=12

paired source blocks under four multi-agent meta-training architectures ( matched_coupled , matched_reference , dual_same , and dual_cross ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Set	Arm	Task	Method / partner	n	AUC [95% CI]	Team gain [95% CI]
RC	critic	navigation	mappo / fixed initial	4	0.3490 [0.2947, 0.4032]	0.7693 [0.6350, 0.9036]
RC	critic	navigation	mappo / online	4	0.8073 [0.7284, 0.8661]	1.6148 [1.3169, 1.7874]
RC	critic	wind flocking	mappo / fixed initial	4	0.1647 [0.1347, 0.1901]	0.7795 [0.6493, 0.9096]
RC	critic	wind flocking	mappo / online	4	0.2813 [0.1900, 0.3463]	1.2280 [1.0830, 1.3414]
RC	mechanism	navigation	disco103 / fixed initial	4	-0.2495 [-0.3265, -0.1657]	-0.0257 [-0.0637, 0.0123]
RC	mechanism	navigation	disco103 / fixed trained	4	0.2316 [0.1025, 0.3607]	-0.0005 [-0.0910, 0.0901]

Appendix

Table 10: Summary of the 70-cell control and capacity-extension matrix across partner regimes, critic information structures, policy widths, and source transfer conditions (

N=904

total evaluated runs).

Batch	Task	Method / route / source / phase	Records	Final team	AUC
expanded validation	wind flocking	/learning/eta0/independent target	4	0.4047	0.3411
expanded validation	wind flocking	/learning/eta0/seed split	4	0.4189	0.3304
expanded validation	wind flocking	/learning/etaF/independent target	8	0.4509	0.3739
expanded validation	wind flocking	/learning/etaF/seed split	8	0.3984	0.3566
expanded validation	wind flocking	/learning/etaL/independent target	8	0.4591	0.3778
expanded validation	wind flocking	/learning/etaL/seed split	8	0.3718	0.3715

Appendix

Table 11: Exploratory multi-task screening endpoints grouped by experimental family, task, learning method, partner regime, and rule initialization.

Family	Task	Contrast / condition	n	Estimate [95% CI]
method gaps	wind flocking	Disco103 minus MAPPO independent bank endpoint fixed initial	12	-0.3031 [-0.5122, -0.0979]
method gaps	wind flocking	Disco103 minus MAPPO independent bank focal gain fixed initial	12	-0.3320 [-0.5447, -0.1266]
method gaps	wind flocking	Disco103 minus MAPPO original team endpoint fixed initial	12	-0.4830 [-0.5884, -0.3867]
method gaps	wind flocking	Disco103 minus MAPPO independent bank endpoint online	12	0.2997 [-0.0399, 0.7657]
method gaps	wind flocking	Disco103 minus MAPPO independent bank focal gain online	12	0.2708 [-0.0732, 0.7341]
method gaps	wind flocking	Disco103 minus MAPPO original team endpoint online	12	-0.2663 [-0.3649, -0.1817]

Appendix

Table 12: Supplementary contrasts from the multi-task screening, architectural control, capacity, and preliminary long-horizon cohorts.

Task	Method	Selected LR	Entropy	AUC difference [95% CI]
wind flocking	disco103	0.0003	0.03	0.0014 [-0.0212, 0.0271]
wind flocking	mappo	0.001	0.01	0.0623 [0.0018, 0.1208]
navigation	disco103	0.001	0.0	0.0164 [-0.0427, 0.0778]
navigation	mappo	0.001	0.0	0.1214 [-0.0118, 0.2679]

Appendix

Table 13: Out-of-sample hyperparameter confirmation contrasts (

n=8

independent confirmation blocks per task) comparing sweep-selected against default learning-rate and entropy configurations.

Source seed	Contrast	Target blocks	Team AUC difference	Team endpoint difference
2301	etaF minus eta0 frozen	4	0.0049	0.0036
2301	etaL minus eta0 frozen	4	0.0022	0.0063
2301	etaL minus etaF frozen	4	-0.0027	0.0027
2301	etaF minus eta0 learning	4	0.0074	-0.0113
2301	etaL minus eta0 learning	4	0.0093	0.0080
2301	etaL minus etaF learning	4	0.0019	0.0193

Appendix

Table 14: Exploratory five-rule transfer evaluation in wind_flocking and navigation (

n=8

paired blocks per task) comparing alternative single-agent discovered rules against Disco103.

Task	Metric	n	Estimate [95% CI]	Holm p
joint passage	auc	12	0.2292 [0.0134, 0.4312]	0.1377
joint passage	portable gain	12	-0.5389 [-0.8253, -0.2187]	–
joint passage	team	12	0.1671 [-0.0441, 0.3757]	–
joint passage	team gain	12	0.1671 [-0.0441, 0.3757]	–
joint passage	success	12	0.0000 [0.0000, 0.0000]	–
navigation	auc	12	0.0064 [-0.0654, 0.0759]	0.8662

Appendix

Table 15: Primary factorial confirmation contrasts (

n=12

paired blocks per task,

N=144

runs) isolating coupled versus decoupled partner experience routing across wind_flocking , navigation , and joint_passage .

Task	Batch	Method / arm	n	Team	Bank gain	Team succ.	Bank succ.
signal coordination	mechanism confirm	disco103/control reset	12	27.6979	-0.0208	0.7578	0.4067
signal coordination	mechanism confirm	disco103/switch reset	12	20.0911	-0.0217	0.2500	0.3264
signal coordination	mechanism confirm	disco103/replay	12	20.0990	0.0122	0.2500	0.3485
signal coordination	mechanism confirm	mappo/control reset	12	31.4036	0.0208	1.0000	0.3333
signal coordination	mechanism confirm	mappo/switch reset	12	29.9766	0.0286	0.9167	0.2500
signal coordination	mechanism confirm	mappo/replay	12	31.4323	0.0104	1.0000	0.2083

Appendix

Table 16: Complete cell means for counterfactual reset-switch, recent-replay filtering,

1.05\text{M}

-step extended-horizon navigation, matched short-horizon OPEN, and external Overcooked cohorts.

Task	Contrast / metric	n	Estimate [95% CI]	Holm p
signal coordination	switch interaction / post auc portable gain	12	-0.0050 [-0.0351, 0.0258]	1.0000
signal coordination	switch interaction / portable gain	12	-0.0087 [-0.0456, 0.0260]	–
signal coordination	switch interaction / post auc team	12	0.1381 [-2.1293, 2.5351]	–
signal coordination	switch interaction / post auc success	12	0.0149 [-0.1252, 0.1646]	–
signal coordination	switch minus control disco103 / post auc portable gain	12	-0.0064 [-0.0226, 0.0089]	–
signal coordination	replay minus control disco103 / post auc portable gain	12	0.0116 [-0.0037, 0.0265]	–

Appendix

Table 17: Complete register of 149 paired-block contrasts across the feedback-routing, replay-filtering, source-adaptation, and extended-horizon confirmation cohorts.

Step size	Comparisons	Above tolerance	Maximum relative error	Maximum absolute error
1e-03	432	1	1.22e-06	1.16e-07
1e-04	432	0	9.84e-08	1.16e-09
1e-05	432	0	7.34e-07	2.81e-11

Appendix

Table 18: Exact two-player differentiable game diagnostic across step sizes (

432

comparisons per step size across two payoff matrices, 24 initializations, three gradient modes, and inner horizons

K\in\{1,4,16\}

Task	Block	Response contrast
wind flocking	1	-0.1743
wind flocking	2	-0.2536
wind flocking	3	-0.0354
wind flocking	4	-0.0684
wind flocking	5	-0.3319
wind flocking	6	-0.8049

Appendix

Table 19: Block-level paired perturbation-response contrasts (

\Delta_{C}

65{,}656

steps,

\xi=0.01

) across all

n=12

confirmation blocks per task.

Task	Method	Repeat correlation	Mean bank gain	M0 loss	M1 loss
wind flocking	disco103	0.1206	0.1713	0.2578	0.2603
wind flocking	mappo	0.6624	-0.9709	0.7142	0.7115
navigation	disco103	-0.0670	1.1499	0.3317	0.3309
navigation	mappo	0.0554	1.1966	1.0981	1.1021

Appendix

Table 20: Measurement repeatability of early perturbation response and per-method standardized ridge prediction losses (

L_{0}

vs.

L_{1}

) across confirmation blocks.

Task	Block	Prediction score
wind flocking	1	-0.0002
wind flocking	2	-0.0060
wind flocking	3	-0.0023
wind flocking	4	0.0079
wind flocking	5	-0.0016
wind flocking	6	0.0009

Appendix

Table 21: Block-level out-of-sample squared-error reduction scores (

S

) at

1{,}048{,}640

steps across all

n=12

confirmation blocks per task.

Method / feedback	View	Metric	Change [95% CI]
disco103/coupled	BC	team return mean	-0.1693 [-0.4688, 0.1823]
disco103/coupled	BC	native completion fraction	-0.0163 [-0.0267, -0.0046]
disco103/coupled	BC	active interact	-0.0097 [-0.0192, 0.0002]
disco103/coupled	BC	movement	-0.0867 [-0.1337, -0.0445]
disco103/coupled	BC	mutual collision block	0.0411 [0.0075, 0.0812]
disco103/coupled	RL	team return mean	0.1042 [0.0000, 0.2344]

Appendix

Table 22: Partner-stratified micro-behavioral decomposition in Overcooked ( cramped_room ,

92{,}160

steps) across Behavioral-Cloning ( BC ), Reinforcement-Learned ( RL ), Scripted , and 10-partner aggregate ( all10 ) strata (

n=12

paired blocks, pointwise 95% block-bootstrap intervals relative to initial focal policies).

Source	Target	Stratum	Return gain [95%]	Completion gain [95%]
eta0	coupled	BC	-0.3125 [-0.4883, -0.1237]	-0.0179 [-0.0254, -0.0104]
eta0	coupled	Scripted	2.3828 [-0.1693, 5.1172]	0.0072 [-0.0000, 0.0137]
eta0	coupled	RL	0.0586 [0.0065, 0.1107]	0.0029 [0.0003, 0.0055]
eta0	decoupled	BC	-0.5729 [-0.7292, -0.4167]	-0.0296 [-0.0374, -0.0218]
eta0	decoupled	Scripted	-1.2891 [-3.2161, 0.5990]	-0.0059 [-0.0104, -0.0013]
eta0	decoupled	RL	0.0391 [-0.0000, 0.0781]	0.0020 [0.0000, 0.0039]

Appendix

Table 23: Partner-stratified cross-regime source-transfer return and completion gains in Overcooked at

46{,}080

steps (

n=12

paired source blocks, pointwise 95% block-bootstrap intervals).

Method	Data	Alpha	AUC [95%]	Initial return [95%]	Adjusted AUC [95%]
disco103	coupled	0	0.2969 [0.2750, 0.3179]	-0.2953 [-0.3323, -0.2561]	0.5922 [0.5454, 0.6331]
disco103	reference	0	-0.1722 [-0.2740, -0.0650]	-0.2953 [-0.3323, -0.2561]	0.1231 [0.0061, 0.2475]
mappo	coupled	0	0.5453 [0.5048, 0.5825]	-0.3127 [-0.3639, -0.2607]	0.8580 [0.7804, 0.9295]
mappo	reference	0	0.5607 [0.5333, 0.5897]	-0.3127 [-0.3639, -0.2607]	0.8734 [0.8115, 0.9296]
disco103	coupled	0.25	0.3016 [0.2768, 0.3222]	-0.3256 [-0.3590, -0.2897]	0.6272 [0.5879, 0.6643]
disco103	reference	0.25	-0.1977 [-0.2908, -0.1021]	-0.3256 [-0.3590, -0.2897]	0.1279 [0.0348, 0.2265]

Appendix

Table 24: Complete 24-cell original-team learning-curve AUC, initial return, and initial-return-adjusted AUC across the 288-run parametric physical-coupling sweep (

\alpha\in[0,1.5]

n=12

paired blocks per cell).

Method	Data	Alpha	F0 bank return [95%]	FT bank return [95%]	Gain [95%]
disco103	coupled	0	-0.4029 [-0.4296, -0.3785]	-0.1751 [-0.2581, -0.0807]	0.2277 [0.1519, 0.3187]
disco103	reference	0	-0.4029 [-0.4296, -0.3785]	-0.4909 [-0.9450, -0.1335]	-0.0880 [-0.5354, 0.2666]
mappo	coupled	0	-0.4133 [-0.4554, -0.3655]	-0.0864 [-0.2417, 0.0586]	0.3268 [0.1698, 0.4817]
mappo	reference	0	-0.4133 [-0.4554, -0.3655]	-0.3097 [-0.6210, -0.0316]	0.1035 [-0.2029, 0.3965]
disco103	coupled	0.25	-0.4341 [-0.4598, -0.4104]	-0.1880 [-0.2758, -0.0983]	0.2461 [0.1675, 0.3288]
disco103	reference	0.25	-0.4341 [-0.4598, -0.4104]	-0.5272 [-0.8999, -0.2214]	-0.0931 [-0.4610, 0.2159]

Appendix

Table 25: Complete 24-cell cross-partner bank evaluation across the parametric physical-coupling sweep (

5{,}760

checkpoint evaluations against the 10-partner bank,

n=12

paired blocks per cell).

Block	Slope
0	0.1959
1	0.3420
2	0.2723
3	0.4610
4	0.1074
5	0.1990

Appendix

Table 26: Within-block linear interaction slopes (

b

, native AUC units per unit of physical coupling

\alpha

) across all

n=12

paired training blocks.

Arm	Block	Outer steps	Rule distance	Physical worlds	Meta-train time (h)
matched coupled	0	512	3.8481	1425408	0.6258
matched reference	0	512	4.4615	1425408	0.6322
dual same	0	512	4.6713	1425408	3.0594
dual cross	0	512	4.7360	1425408	3.1242
matched coupled	1	512	4.2957	1425408	0.6172
matched reference	1	512	3.0041	1425408	0.6281

Appendix

Table 27: Complete meta-training configuration, validation metrics, and downstream rank synthesis for all 48 meta-evolved multi-agent candidate update rules.

Task	Horizon	Method / regime	n	Initial $J_{00}$	Final $J_{11}$	Net gain
wind flocking	131,120	open / coupled	12	-0.5570	0.7860	+1.3431
wind flocking	131,120	open / decoupled	12	-0.5570	0.6182	+1.1752
wind flocking	1,048,640	disco103 / coupled	5	-0.5095	1.2071	+1.7167
wind flocking	1,048,640	disco103 / decoupled	5	-0.5095	1.2323	+1.7418
wind flocking	1,048,640	disco103 / curriculum	5	-0.5095	1.1970	+1.7065
wind flocking	1,048,640	mappo / coupled	5	-0.5158	1.2912	+1.8070

Appendix

Table 28: Complete empirical outcomes across all 92 evaluated runs in the matched-substrate learned-optimizer ( OPEN ) and extended-horizon curriculum comparison cohort (18 condition summaries across paired seed blocks).

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 389 papers

Oct 15, 2026

MetaShare: Failure-Aware Collective Self-Improvement for Open-Ended Agents

Humans often learn fastest not by repeating success, but by remembering which failures not to repeat. Open-ended self-improving agents can share experience across lineages, as in GEA, or edit their own improvement mechanisms, as in DGM-Hyperagents, yet failed improvement attempts are rarely retained as reusable cross-branch knowledge, and search can collapse onto redundant strategies. We introduce MetaShare, a failure-aware framework that records successful and unsuccessful agent modifications together with signed performance deltas. Before each generation, MetaShare retrieves diverse high-performing strategies alongside high-confidence failures from other lineages, letting agents reuse effective mechanisms while avoiding known dead ends. A task-blind auditor periodically injects absent or underrepresented mechanism categories, while embedding-based de-ranking limits redundancy during retrieval. Multiple meta-agents propose modifications in parallel at heterogeneous temperatures behind a deterministic selector that gates applicability, scope, compilability, and size. Experiments across paper review, coding, math grading, robotics reward design, and financial audit-risk assessment show that MetaShare's median matches or exceeds DGM-Hyperagents' final performance and reaches its own best-performing agent in 1.7--2.9

\times

fewer evolution iterations.

Vishal Bhutani, Prasang Gupta, Sumanth Chundru +1

PricewaterhouseCoopers (PwC)

Self-Improving Agents Self-Evolving Agents Multi-Agent Collaboration

Ask PDF

Oct 8, 2026

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Ad-hoc collaboration often requires agents to identify and adhere to some shared convention within a cooperative task. Existing work on reinforcement learning (RL) for ad-hoc collaboration focuses on training agents that adapt to the conventions established by their partners. These methods fail to consider the possibility that while some partners might follow only a single fixed convention, others may themselves be capable of adapting to multiple conventions. Here we present ConventionPlay, an RL-based approach that teaches agents to discover their partner's optimal convention by training against a learned population of partners that exhibit different degrees of adaptability across conventions. Some of these partners follow a single, fixed convention, while others are able to adapt to a subset of the possible conventions for the task in question. The existence of partners that support a limited subset of conventions forces agents trained against this population to actively probe their partner's capabilities, and steer their partner towards the most effective joint strategy that they are capable of following. Our experimental results demonstrate that agents trained via ConventionPlay achieve superior performance to existing ad-hoc collaboration methods against test populations of partners that are compatible with multiple conventions.

Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

Department of Computer Science Sheffield, UK · Pittsburgh, PA 15213, USA

Multi-Agent Collaboration Ad Hoc Teamwork Cooperative MARL

Ask PDF

Aug 8, 2025

Unsupervised Partner Design Enables Robust Ad-hoc Teamwork

We introduce Unsupervised Partner Design (UPD), a population-free multi-agent reinforcement learning method for robust ad-hoc teamwork. UPD generates training partners on-the-fly and selects them adaptively based on a learnability criterion, removing the need for pre-trained partner populations or manual parameter tuning. We show that this simple mechanism enables effective partner diversity and can be extended to joint partner-environment selection when a procedural level generator is available. Across Level-Based Foraging, Overcooked-AI, and the Overcooked Generalisation Challenge, UPD consistently achieves strong performance compared to both population-based and population-free baselines. In a human-AI user study, agents trained with UPD achieve higher returns and are rated as more adaptive, more human-like, and less frustrating than all evaluated baseline methods.

Constantin Ruhdorfer, Matteo Bortoletto, Victor Oei +2

Collaborative Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.

Multi-Agent Collaboration Ad Hoc Teamwork Robust RL Cooperative MARL

Ask PDF

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

MetaShare: Failure-Aware Collective Self-Improvement for Open-Ended Agents

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Unsupervised Partner Design Enables Robust Ad-hoc Teamwork