Self-Referenced Social Preferences: Cooperation without Observing Others Rewards

Mohamed Ayman Mohamed; Harshil Kotamreddy; Marcos Menon Jose

cs.AIOct 6, 2026

Self-Referenced Social Preferences: Cooperation without Observing Others Rewards

Authors: Mohamed Ayman Mohamed, Harshil Kotamreddy, Marcos Menon Jose

Organizations: Amazon Vancouver, Canada · Nvidia Santa Clara, United States · Itaú Unibanco São Paulo, Brazil

Abstract

Social preferences can promote cooperation in multi-agent reinforcement learning, but existing approaches often require agents to observe the rewards of their peers. In many real-world interactions, however, an agent can, as humans do, observe others' behavior and outcomes without access to their private reward signals. We introduce self-referenced social preferences, in which each agent learns a model of its own reward, applies it to other agents' observed transitions to assess their outcomes from its own perspective, and feeds these self-referenced assessments into standard social preferences. We study two ways to incorporate these assessments: modifying the learning reward, or using them to weight policy updates. We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest, which require volunteering, public-good contribution, and resource restraint, respectively. Across all three environments, agents learn cooperative behavior without observing others' rewards, including in settings where independent learners fail to cooperate, and frequently achieve more equitable divisions of jointly produced returns than agents with access to true rewards. The effective integration point depends on the social preference: inequity aversion works best in the reward together with a value look-ahead, whereas a purely benevolent preference benefits from policy-update weighting. Under partial observability, the policy-update approach continues to support cooperation. These results show that explicit access to other agents' reward signals is not necessary for learning cooperative behavior: social preferences can instead be grounded in self-referenced assessments of others' outcomes derived from their observed behavior.

Figures & tables

Panel of Figure 1 — Figure 1. Self-referenced social preferences. (a) During training agent $i$ receives the transitions of every other agent $j$ , but never $j$ ’s reward. (b) Agent $i$ ’s reward model, fitted on its own transitions, assesses $j$ ’s transitions, giving self-referenced estimates $\hat{r}_{ij}^{t}$ , whose running sums $x_{ij}^{t}$ feed an unchanged social term $F_{i}$ . SRR adds the term to the reward; SRA evaluates the critic on $j$ ’s observations and weights the scores into the policy-update coefficient $\Psi_{i}^{t}$ . The ledger discounts $i$ ’s gains when $i$ has been ahead. Diagram illustrating self-referenced social preferences, agent observations, and reward modeling.

Figure 2. Collective return over training. Each panel shows plain PPO, the TR learner, and both of our methods: SRA and SRR+V . Lines are means over eight seeds and shaded regions are pointwise 95%

t

intervals. The dotted line is the Escape Room optimum,

17

. Commons Harvest TR-IA is outside the plotted range, with final mean

C=-1{,}935

, and is marked at the panel edge.

Figure 3. Who receives the return and who does the work in Clean Up with five agents. Top: every agent’s return over training; agents are ranked within each seed by their final return and averaged by rank over the eight seeds, darkest the richest. Bottom: each agent’s final share of the apples, solid, and of the cleaning, dashed, with 95% confidence intervals; 20% is an equal share.

Figure 4. Partial view: collective return over training in Clean Up and Commons Harvest with a

15\times 15

window, for feed-forward and GRU agents. Lines are means over eight held-out seeds and shaded regions pointwise 95% Student-

t

intervals.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Learner	PPO reward	actor coefficient	$f_{i}$ target
Plain PPO	$r_{i}$	own GAE	–
TR	$r_{i}+F_{i}(x^{\rm true})$	own GAE	–
SRR	$r_{i}+F_{i}(x^{\rm pred})$	own GAE	$r_{i}$
SRA	$r_{i}$	$\widetilde{\Psi}_{i}$	$r_{i}$
+ Standing	above $-\kappa\,\mathrm{ahead}_{i}[r_{i}]_{+}$	unchanged	unchanged

Appendix

Table 1. The learning signal of each learner.

V_{i}^{\rm obj}

is trained on the return of the reward in the second column.

Figure 5. One state of each game as two agents observe it, after a few random steps, with the full view (top) and the partial view (bottom). Escape Room ( $N=5$ ): the one-hot place of every agent, the observer’s own first and the others in a fixed order; agent 1 is at the lever and agent 4 at the door. The observation is not spatial, so it has no partial version. Clean Up: the $47\times 47$ window centred on the observer (red circle), large enough to contain the whole $25\times 18$ map from any position, shown with one colour per plane (orange self, blue others, green apples, brown waste, light blue river, grey walls; beams yellow). Commons Harvest: the $73\times 73$ RGB window centred on the observer and turned to the direction it faces, so the two agents see the map at different angles; apples are green, the observer and the others have one fixed colour each. The partial view is the central $15\times 15$ part of the same window (dashed box, top), used in the partial-view experiments of Appendix G . Everything outside the map is empty.

Game	Observation	Actions
Escape Room	one-hot place (start, lever, door) of every agent, its own first; $3N$ values; no partial view	3: stay, lever, door
Clean Up	window centred on the agent, 7 binary planes: self, others, apple, waste, river, wall, beam; $47\times 47$ full, $15\times 15$ partial	6: four moves, stay, clean
Commons Harvest	RGB window centred on the agent and turned to its facing direction, self and others each in one fixed colour; $73\times 73$ full, $15\times 15$ partial	8: four moves, stay, two turns, fire

Appendix

Table 2. Observation and actions of one agent. Every observation is arranged relative to its agent. The full view covers the whole state and is used in the main experiments; the partial view keeps only the

15\times 15

neighbourhood of the agent. Figure 5 shows examples of both.

	Escape Room	Clean Up	Commons Harvest
Shared network	MLP, two layers of 64 tanh units	$3\times 3$ convolutions with 16 and 32 channels (second with stride 2), 128 ReLU units	CleanRL Atari network: convolutions $8\times 8$ /4 (32), $4\times 4$ /2 (64), $3\times 3$ /1 (64), 512 ReLU units
Reward model $f_{i}$	MLP, two layers of 64 tanh units, on $(o,a,o^{\prime})$	two $3\times 3$ convolutions (16 channels) on the stacked $(o,o^{\prime})$ , max- and mean-pooled, then 64 units with $a$	one layer of 64 tanh units on the shared network’s features of $o$ and $o^{\prime}$ (not trained through) and $a$
Steps; parallel envs	500k; 16	5M; 8	5M; 8
Rollout per env	16, 32, or 128 (tuned)	256	128
Learning rate	$2.5\times 10^{-4}$ or $10^{-3}$ (tuned)	$10^{-3}$	$10^{-3}$
Entropy coefficient	0.01 or 0.05 (tuned)	0.01 ( $N=5$ ); 0.05 ( $N=3$ ) ∗	0.01

Appendix

Table 3. Networks and PPO settings. The actor and the critic are linear heads on one shared network. Escape Room tunes the PPO setting per learner (Table 4 ); Clean Up and Commons Harvest use one setting for all learners, chosen in the PPO stage of the selection protocol.

Learner (weights)	LR, entropy	Rollout
Plain PPO	$10^{-3}$ , 0.01	32
TR-EI ( $\alpha=1$ )	$10^{-3}$ , 0.01	128
TR-IA ( $\alpha/\beta=0/1$ )	$10^{-3}$ , 0.01	32
TR-SVO ( $\alpha=1$ , $\phi=\pi/4$ )	$10^{-3}$ , 0.01	128
SRA-EI ( $\alpha=3$ )	$2.5\times 10^{-4}$ , 0.05	128
SRA-IA ( $\alpha/\beta=0/0.5$ )	$10^{-3}$ , 0.05	128

Appendix

Table 4. Escape Room,

N=5

: PPO setting of each learner at its collective-return pick (learning rate, entropy coefficient, rollout length per environment).

Figure 6. Clean Up tuning landscape. Each point is one setting evaluated on the three tuning seeds, placed by its mean collective return and mean Nash welfare and coloured by learner family; grey points are variants not analysed in the paper. The circle and the diamond mark the settings that the mean-minus-standard-deviation score selects on collective return and on Nash welfare, among all candidates at that group size.

Figure 7. Our three self-referenced learners, which never observe another agent’s reward, against plain PPO on what each game tests: cooperation in Escape Room, productivity and fairness in Clean Up, and efficiency, sustainability, and welfare in Commons Harvest. Each axis spans the means of all learners in that game, true-reward ones included; in Commons Harvest the true-reward agents reach further on efficiency and sustainability. Eight held-out seeds. Radar plot comparing plain PPO with three self-referenced learners on six properties across the three games.

	selected on collective return			selected on Nash welfare
Learner	$C$	$\mathcal{NW}$	$L$	$C$	$\mathcal{NW}$	$L$
Escape Room ( $N=5$ )
Plain PPO	$-0.3\pm 0.4$	$4.95\pm 0.08$	$-0.25\pm 0.39$	same setting
TR-EI	$\mathbf{17.0\pm 0.0}$	$\mathbf{6.96\pm 0.00}$	$-1.00\pm 0.00$	same setting
SRA-EI	$15.7\pm 3.0$	$6.78\pm 0.42$	$-1.00\pm 0.00$	same setting
SRR-EI	$0.0\pm 0.0$	$5.00\pm 0.00$	$0.00\pm 0.00$	same setting

Appendix

Table 5. Every learner at the hyperparameter setting selected on tuning seeds by collective return and by Nash welfare (“same setting” when both objectives selected it).

C

\mathcal{NW}

, and

L

are computed from each seed’s final-window per-agent means. Entries are held-out-seed means

\pm

95% Student-

t

intervals; they describe the frozen selections and are not used to reselect them.

Method (selection)	$C$	$\mathcal{NW}$	$L$	Sust.
Plain PPO	$211\pm 2$	$30.1\pm 0.2$	$27.9\pm 0.7$	$46\pm 1$
TR-EI (both)	$657\pm 86$	$5.6\pm 1.6$	$0.6\pm 0.7$	$195\pm 55$
TR-SVO (both)	$\mathbf{744\pm 128}$	$26.5\pm 12.7$	$3.2\pm 7.6$	$\mathbf{349\pm 56}$
TR-IA (C)	$-1{,}935$	$0.2$	$-771$	$88$
TR-IA (NW)	$-1{,}678$	$1.0$	$-1{,}037$	$28$
SRA-EI (both)	$416\pm 86$	$35.9\pm 3.9$	$8.5\pm 4.2$	$195\pm 44$

Appendix

Table 6. Commons Harvest, seven agents, eight seeds held out from tuning (4–11). Values are means

\pm

95% Student-

t

interval half-widths where shown; bold marks the best per column. In parentheses, the metric that selected the setting (C: collective return; NW: Nash welfare; both: the same setting). Sust.: mean step at which apples are eaten. ∗ Means of SRA-IA and SRR-IA (both) and of SRR-EI , SRR-SVO , and SRR-IA+V (NW); half-widths

\leq 4.2

(

C

\leq 0.6

(

\mathcal{NW}

), and

\leq 1.7

(

L

L

: lowest agent return. Self-scaling control: SVO with

\phi=0

, which only rescales the agent’s own reward.

Env.	Method (selection)	$C$	$\mathcal{NW}$	$L$	Behavior
Escape Room	Plain PPO	$-0.25\pm 0.39$	$4.95\pm 0.08$	$-0.25\pm 0.39$	no reliable escape
	TR-EI (C)	$\mathbf{17.00\pm 0.00}$	$6.96\pm 0.00$	$-1.00\pm 0.00$	concentrated exits, SD 0.49
	SRA-EI	$15.75\pm 2.95$	$6.78\pm 0.42$	$-1.00\pm 0.00$	concentrated exits, SD 0.48
	SRR-IA + Standing (NW)	$13.08\pm 0.28$	$\mathbf{7.50\pm 0.03}$	$\mathbf{+1.27\pm 0.12}$	criterion 8/8, SD 0.13
Clean Up	Plain PPO	$248\pm 198$	$37.0\pm 29.8$	$5.4\pm 4.8$	–
	TR-SVO (C)	$1{,}405\pm 316$	$64.6\pm 37.8$	$5.3\pm 12.5$	cleaning share $0.86\pm 0.16$

Appendix

Table 7. Escape Room and Clean Up with five agents, eight held-out seeds.

C

is collective return,

\mathcal{NW}

Nash welfare (

c=5

in Escape Room,

0

in Clean Up),

L

the lowest agent return. In parentheses, the metric that selected a setting when the two differ. The last column describes behaviour: the standard deviation of the exit rates in Escape Room, about 0.5 when the same agents always exit and 0 when all exit equally often, and in Clean Up the share of the cleaning done by the two lowest earners, 0.40 for an equal split.

Figure 8. Nash welfare over training for the runs of Figure.3 in the main paper. Columns are games and rows are social operators; every learner is at its collective-return selection. Lines are means over eight held-out seeds and shaded regions are pointwise 95% Student-

t

intervals, with a five-bin moving average.

Figure 9. Lowest agent return over training for the runs, with the same layout and intervals as Figure 8 . The grey line marks zero. Commons Harvest TR-IA falls below the plotted range, to a final mean of

-771

, and is marked at the panel edge.

$N$	$M$	PPO	TR	Best self-ref.	SRA-EI	$\geq 90\%$
2	1	0.00	1.00	1.00 ( SRR-EI )	1.00	7/16
3	2	0.00	1.00	1.00 ( SRR-IA )	0.69	3/16
5	3	$-$ 0.01	1.00	1.00 ( SRR-IA )	0.93	3/16
7	4	0.00	1.00	1.00 ( SRR-IA )	0.82	2/16
10	5	0.00	1.00	1.00 ( SRR-IA )	0.69	2/16
12	6	0.00	1.00	1.00 ( SRR-IA )	0.00	2/16

Appendix

Table 8. Escape Room: collective return as a fraction of the optimum, held-out seeds. “Best self-ref.” is the best of 16 self-referenced variants (four operators

\times

four placements) selected on tuning seeds; the last column counts how many of the 16 reach 90% of the optimum.

Figure 10. The value look-ahead. Collective return over training for the reward placement without the look-ahead, SRR , and with it, SRR+V ; plain PPO and the true-reward learner are drawn faintly for reference. Every learner is at its collective-return selection; lines are means over eight held-out seeds and shaded regions, drawn for every learner except plain PPO, are pointwise 95% Student-

t

intervals; where all seeds agree, as for the true-reward learners at the Escape Room optimum, the interval has no width. Learners below the plotted range are marked at the panel edge with their final means.

Figure 11. Where the self-referenced estimate enters, by group size. Escape Room collective return divided by the optimum for each operator, with the estimate in the policy update, SRA , in the reward, SRR , and in the reward with the value look-ahead, SRR+V , against the true-reward learner. Every cell is tuned separately at its

N

on tuning seeds; points are means over eight held-out seeds and shaded regions, drawn for SRA and SRR+V , are 95% Student-

t

intervals.

Figure 12. Final collective return, Nash welfare, and lowest agent return against the number of agents

N

for plain PPO, TR-EI , SRA-EI , TR-IA , SRR-IA+V , TR-SVO , and SRA-SVO , each at its collective-return selection for that

N

; means over eight held-out seeds, bars are 95%

t

-intervals. The Escape Room optimum grows with

N

(dotted line). Downward triangles mark means below the plotted range: in Commons Harvest, TR-IA at

N=7

ends at

-1{,}935

, and at

N=9

TR-IA , SRR-IA+V , SRA-EI , TR-EI , and TR-SVO end at

-12{,}818

-5{,}735

-3{,}770

-2{,}268

, and

-1{,}720

. Commons Harvest Nash welfare and lowest agent return here are the per-step logged values, which differ slightly from the values computed from per-agent means in the other tables.

Figure 13. Who receives the return and who does the work for all nine Clean Up learners with five agents. One block per operator; columns are the true-reward learner, SRA , and SRR+V , each at its collective-return selection. Top row of each block: every agent’s return over training, agents ranked within each seed by their final return and averaged by rank over the eight held-out seeds, darkest the richest. Bottom row: each agent’s final share of the apples, solid, and of the cleaning, dashed, with 95% Student-

t

intervals; the dotted line is the equal share of 20%.

Method	Setting	$C$	$\mathcal{NW}$	$L$	Returns
Plain PPO	–	$373\pm 206$	$104.9\pm 61.8$	$48.3\pm 45.2$	172 / 153 / 48
SRA-EI (NW)	$\alpha{=}2$	$768\pm 69$	$245.6\pm 24.5$	$172.8\pm 27.9$	335 / 260 / 173
SRA-SVO	$\alpha{=}7$	$807\pm 36$	$258.9\pm 13.5$	$196.1\pm 23.3$	369 / 242 / 196
SRA-IA	$0/1$	$689\pm 32$	$228.7\pm 10.7$	$208.6\pm 13.8$	257 / 223 / 209
SRR-IA+V	$0/0.03$	$664\pm 213$	$211.2\pm 74.1$	$160.3\pm 75.1$	296 / 207 / 160
TR-IA	$0/0.03$	$820\pm 37$	$\mathbf{265.4\pm 16.1}$	$\mathbf{215.7\pm 31.9}$	361 / 243 / 216

Appendix

Table 9. Clean Up, three agents, eight held-out seeds (95% CIs). IA settings are

\alpha/\beta

; SVO uses

\phi=\pi/3

. The last column lists the mean return of the highest through the lowest earner.

Method	Setting	$C$	$\mathcal{NW}$	$L$	Returns, high $\to$ low
Plain PPO	–	$313\pm 193$	$33.7\pm 24.0$	$5.2\pm 8.2$	70 / 60 / 51 / 49 / 47 / 31 / 5
SRA-EI (NW)	$\alpha{=}3$	$1{,}953\pm 174$	$\mathbf{198.5\pm 18.2}$	$34.2\pm 13.7$	426 / 409 / 383 / 352 / 285 / 64 / 34
SRA-SVO (NW)	$\alpha{=}5$	$1{,}356\pm 111$	$161.8\pm 13.9$	$39.0\pm 9.8$	266 / 252 / 244 / 239 / 231 / 85 / 39
SRR-IA (NW)	$0.003/0.1$	$1{,}069\pm 116$	$151.1\pm 16.2$	$\mathbf{116.3\pm 17.8}$	178 / 166 / 162 / 155 / 150 / 142 / 116
SRR-IA+V (NW)	$0.003/0.1$	$1{,}006\pm 133$	$140.9\pm 19.8$	$100.3\pm 26.9$	181 / 166 / 154 / 143 / 133 / 129 / 100
TR-IA (NW)	$0/0.03$	$345\pm 296$	$40.9\pm 37.5$	$11.4\pm 14.8$	76 / 65 / 62 / 55 / 52 / 23 / 11

Appendix

Table 10. Clean Up, seven agents, eight held-out seeds (7–14; 95% CIs). Each learner at the setting selected on the three tuning seeds, by the metric in parentheses. The last column lists the mean return of the highest through the lowest earner.

Figure 14. Who receives the return in Escape Room with five agents, as in Figure 13 . Nash welfare uses the offset

c=5

of Eq. ( 15 ).

Figure 15. Who receives the return in Commons Harvest with seven agents, as in Figure 13 . TR-IA collapses and lies below the plotted range.

Figure 16. Agent outcomes under the two tuning objectives. Columns are the three games; rows are the configurations selected on tuning seeds by collective return (top) and Nash welfare (bottom), without the self-scaling controls. Each large panel shows the return of every agent over training: agents are ordered within each seed by their final-window return and then averaged by rank across the eight held-out seeds (darkest: highest final-return rank). The narrow panel shows all eight final seed values and the rank mean

\pm

95% Student-

t

interval. Insets give the final

C

and

\mathcal{NW}

. Ranked curves describe how the return is divided; they do not follow one agent across seeds.

Figure 17. Standing in the repeated game, Escape Room. (a, b) SRR-IA with five agents, without and with standing, the only difference between the two: the fraction of episodes in which each agent exits, ranked within each seed by its final exit rate and averaged by rank over the eight held-out seeds; the dotted line is the even share, two exits in five. (c) Collective return divided by the optimum for the same two runs. (d–g) Lowest agent return over training without and with standing for the four exact pairs. Means over seeds 11–18 with pointwise 95% Student-

t

intervals. Figure 18 shows every seed.

Figure 18. Every held-out seed of the standing pairs in Escape Room over training: the lowest agent return, top, and the collective return over the optimum, bottom, without standing in grey and with it in colour. Thin lines are single seeds, seeds 11–18, and bold lines their mean. Without standing, every seed settles at a loss of one point for its worst-off agent, and most groups reach the optimum; with standing, the worst-off agent stays above zero on almost every seed, and the group settles at 70 to 80% of the optimum. † The two TR-IA pairs also differ in their PPO settings.

ER( $N$ , $M$ ), with-standing configuration	$C/C^{\star}$	$L_{s}$	Exit SD	Shared seeds
(3,2) SRA-EI , $\kappa{=}4$	$0.711\pm 0.035$	$1.514\pm 0.220$	$0.030\pm 0.015$	$0/8\to 8/8$
(5,3) SRA-EI , $\kappa{=}4$	$0.797\pm 0.002$	$0.114\pm 0.119$	$0.129\pm 0.003$	$0/8\to 6/8$
(5,3) SRR-IA , $\kappa{=}1$	$0.769\pm 0.016$	$1.268\pm 0.122$	$0.131\pm 0.016$	$0/8\to 8/8$
(5,3) TR-IA , $\kappa{=}1$ †	$0.735\pm 0.032$	$0.942\pm 0.403$	$0.107\pm 0.020$	$0/8\to 8/8$
(7,4) SRR-IA , $\kappa{=}2$	$0.718\pm 0.039$	$1.201\pm 0.282$	$0.104\pm 0.040$	$0/8\to 8/8$
(7,4) TR-IA , $\kappa{=}2$ †	$0.691\pm 0.134$	$0.476\pm 0.783$	$0.122\pm 0.050$	$0/8\to 6/8$

Appendix

Table 11. The four exact pairs of Figure 17 and the two descriptive TR-IA pairs, held-out seeds 11–18. Values for the with-standing arm are means

\pm

95% Student-

t

interval half-widths. A seed counts as shared when

L_{s}>0

and the exit rates have a standard deviation below 0.20; the last column gives the shared seeds without

\rightarrow

with standing. Every without-standing arm is 0/8. † The TR-IA pairs also differ in their PPO settings.

Figure 19. How many agents pull the lever at the first step of an episode in Escape Room with standing, over 400 evaluation episodes per run, for the four configurations with standing. Bars are means over training seeds and dots single seeds. The line is the binomial distribution that independent choices with the observed pull rate would give; the episode is solved at once only when exactly

M

agents pull.

Control	Gate	$N=3$	$N=5$	$N=7$
Critic trained on raw returns	off	$8.00\to 7.44$	$15.91\to 13.62$	$21.60\to 19.97$
	on	$5.98\to 4.67$	$13.52\to 13.03$	–
Standing as a policy input	off	$8.00\to 8.00$	$15.91\to 15.99$	$21.60\to 21.43$
	on	$5.98\to 6.40$	$13.52\to 13.53$	–
One shared true ledger	on	$5.98\to 5.70$	$13.52\to 13.49$	–
with standing input	on	$6.40\to 6.17$	$13.53\to 13.54$	–

Appendix

Table 12. Collective return of the selected SRA-EI learner in Escape Room, without and with each control, and without or with the standing gate at

\kappa=4

; three seeds per cell. The optimum is 8, 17, and 26 at

N=3

, 5, and 7.

Figure 20. Commons Harvest with recurrent agents and the partial view, trained for 14 million frames; the dotted line marks 7 million, the length of the runs in Figure.4 in the main paper. Rows are operators and columns the collective return, Nash welfare, and lowest agent return; each line is one candidate setting, solid and dashed for the two settings of a learner, means over the eight held-out seeds with pointwise 95% Student-

t

intervals. One TR-SVO setting had not finished when the figure was made.

Figure 21. Who receives the return in Clean Up when agents 2 and 4 earn two points per apple. Rows are operators and columns the true-reward learner and ours, each at the setting selected under shared rewards. Each line is one agent’s return over training, ranked within each seed from the highest to the lowest earner and averaged by rank over the eight held-out seeds. Our learners reach a higher Nash welfare and leave the poorest agent more than the true-reward learner of the same operator; under SRR-IA+V the agents eat almost equally many apples, so the two agents with the higher value end with about twice the return of the others.

Figure 22. What each learner gives the agents that value apples more, in Clean Up with five agents when agents 2 and 4 value an apple at 2. One column per operator, comparing the true-reward learner with ours, each at the setting selected under shared rewards. Top: return per agent over training for the two high-value agents, solid, and the three others, dashed, means over the eight held-out seeds with pointwise 95% Student-

t

intervals. Middle and bottom: final apples eaten and share of the cleaning per agent, dark for the high-value agents and light for the others, with 20% an equal share of the cleaning; bars are means with 95% intervals and dots single seeds.

Method	Rewards	Obs./act.	Policies	Critic
Inequity aversion ( Hughes et al., 2018 )	$\bullet$
Social value orientation ( McKee et al., 2020 )	$\bullet$
Prosocial reward ( Peysakhovich and Lerer, 2018 )	$\bullet$
VDN, QMIX ( Sunehag et al., 2018 ; Rashid et al., 2018 )	$\bullet$ (team)	$\bullet$		$\bullet$
COMA ( Foerster et al., 2018b )	$\bullet$ (team)	$\bullet$		$\bullet$
MADDPG ( Lowe et al., 2017 )		$\bullet$	$\circ$	$\bullet$

Appendix

Table 13. What each method uses from the other agents during training (

\bullet

: used;

\circ

: in some variants): rewards, observations and actions, policies or gradients, and a centralized critic or joint value. At execution every method listed acts on the agent’s own observation.

Explore similar work

Jul 6, 2026cs.AI

Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas

Inducing cooperation among distributed agents is still a difficult problem in the field of multi-agent reinforcement learning (MARL), particularly in social dilemma situations. There, individual interests are misaligned with the common good and individual rationality leads to suboptimal group outcomes. In contrast, humans are able to achieve cooperation with one another in such situations. A common explanation for such cooperative behavior is that individuals have social preferences. In order to achieve cooperation in MARL, we design a new utility function integrating altruistic preferences (incentive for other's reward) and fairness preferences (incentive for equality) from social psychology and behavioral economics, namely, Altruistic and Fairness Preference (AFP), a reward-sharing mechanism which converts one's own and other's rewards to incentives for cooperative behavior. We performed comparative experiments with standard RL and inequity aversion agents in two challenging sequential social dilemma games, and showed that AFP agents successfully achieved mutual cooperation with more collective rewards and higher equity than the baselines. To further understand the progression of AFP during training, we subsequently explore the effects of altruistic preferences and fairness preferences on agents' behavior. The results suggest that altruistic preferences encourage agents to contribute to the public goods, and fairness preferences induce mutual behavior between agents.

Yu Wei, Yukiko Ogura, Yoshiyuki Ohmura +3

The University of Tokyo

Cooperation Multi-Agent Reinforcement Learning

Ask PDF

Aug 7, 2026cs.MA

Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction

Multi-agent simulations are widely used to study complex social and ecological systems, where rich and often unexpected emergent behaviors arise from local interactions. A large body of prior work has focused on analyzing such emergent dynamics across domains. In this paper, we move beyond analyzing emergent behavior and introduce a learning-based mechanism for actively shaping it via social reward modeling. We introduce Multi-Agent Reward Prediction (MARP), a simple framework that extends preference-based reward modeling to multi-agent reinforcement learning. While the framework is designed to be applicable across multi-agent settings, the present empirical validation is limited to a single environment, and we therefore present MARP as a proof of concept within the studied domain. Rather than relying on handcrafted rewards, MARP learns a shared reward model from episode-level evaluations of collective outcomes, enabling decentralized agents to align their behavior with global social objectives. We study MARP in the Harvest Game, a canonical sequential social dilemma modeling common-pool resource management and related real-world challenges. Our results show that MARP can be tuned to produce behavior that is more closely aligned with target social metrics than standard reward-based baselines, while the learned reward model captures subtle environmental structure without explicit programming. Crucially, MARP supports multiple and composite social objectives within a single training regime. By modifying only the high-level evaluation metric, the same framework seamlessly aligns agent behavior with diverse goals, including sustainability, equality, and peace, as well as combinations of individual and group-level objectives. These findings demonstrate that emergent multi-agent behavior can be treated not only as a phenomenon to study, but as a target of principled, data-driven regulation.

Assaf Caftory, Almog Zemach, Moshe Butman +1

EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Faculty of Computer Science, The College of Management Academic Studies, Rishon LeZion, Israel · School of Communication, Reichman University, Herzliya, Israel

Multi-Agent Reinforcement Learning Emergence Agent Behavior Modeling

Ask PDF

Jun 16, 2026cs.LG

Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents

Humans often acquire new skills by observing others, since observed behaviors implicitly reveal how to act in an environment. However, observations drawn from a heterogeneous population introduce conflicting behavioral signals, making it difficult to determine which behaviors are worth imitating. We address this challenge with General Reward Inference and Disentanglement (GRID), a social learning method that extracts universally useful behaviors from a heterogeneous population of demonstrators pursuing different goals. GRID decomposes per-agent reward functions into a general reward, capturing behaviors shared across all agents, and specific rewards, capturing individual preferences and objectives. Training exclusively on the general reward provides a new paradigm of generalist pretraining. It yields a generalist agent that internalizes universal environmental competencies, such as safety and basic task proficiency, without the mode-averaging bias that afflicts standard learning from demonstration techniques. This generalist serves as a superior prior for fine-tuning to downstream tasks, including preferences unseen during training. Experiments across a synthetic basis function decomposition, multi-agent Craftax, and a continuous autonomous driving simulator (Highway-Env) confirm that GRID successfully disentangles reward structure in a semantically meaningful way, outperforms standard learning from demonstration baselines, and enables more efficient and stable specialization.

Caleb Chang, Davin Win Kyi, Natasha Jaques +1

University of Washington · NVIDIA

Multi-Agent Reinforcement Learning Agent Behavior Modeling Autonomous Driving Simulation

Ask PDF