Investigating the Impact of Subgraph Social Structure Preference on the Strategic Behavior of Networked Mixed-Motive Learning Agents

Xinqi Gao; Mario Ventresca

cs.MAApr 4, 2026

Investigating the Impact of Subgraph Social Structure Preference on the Strategic Behavior of Networked Mixed-Motive Learning Agents

Authors: Xinqi Gao, Mario Ventresca

Organizations: Xinqi Gao is with the Department of Industrial Engineering, Purdue University, West Lafayette, IN 47906, USA · Mario Ventresca is with the Department of Industrial Engineering, Purdue University, West Lafayette, IN 47906, USA

Abstract

Limited work has examined the strategic behaviors of relational networked learning agents under social dilemmas, and has overlooked the intricate social dynamics of complex systems. We address the challenge with Socio-Relational Intrinsic Motivation (SRIM), which endows agents with diverse preferences over sub-graphical social structures in order to study the impact of agents' personal preferences over their sub-graphical relations on their strategic decision-making under sequential social dilemmas. Our results in the Harvest and Cleanup environments demonstrate that preferences over different subgraph structures (degree-, clique-, and critical connection-based) lead to distinct variations in agents' reward gathering and strategic behavior: individual aggressiveness in Harvest and individual contribution effort in Cleanup. Moreover, agents with different subgraphical structural positions consistently exhibit similar strategic behavioral shifts. Our proposed BCI metric captures structural variation within the population, and the relative ordering of BCI across social preferences is consistent in Harvest and Cleanup games for the same topology, suggesting the subgraphical structural impact is robust across environments. These results provide a new lens for examining agents' behavior in social dilemmas and insight for designing effective multi-agent ecosystems composed of heterogeneous social agents.

Figures & tables

Method	Reward	Mixed	Topology	Sub-	$\#$ Agent
	Shaping	Motive		Graph	$N$
Haeri [ 8 ]	✓	$\times$	✓	$\times$	3
Hughes et al. [ 4 ]	✓	✓	$\times$	$\times$	5
Jaques et al. [ 9 ]	✓	✓	$\times$	$\times$	5
Willis $\&$ Luck [ 10 ]	$\times$	✓	$\times$	$\times$	2
Roesch et al. [ 11 ]	✓	✓	$\times$	$\times$	5

TABLE I: Comparison with state-of-the-art marl methods.

Fig. 1: Individual base reward under the baseline model exhibits identical decaying trajectories. 5 agents with network-free and preference-free protocol learn and compete independently under Harvest and Cleanup.

Topology	$Med_{H}$	$IQR_{H}$ a	$Med_{C}$	$IQR_{C}$	LB-UB b
Bipartite	0.525	0.520-0.529	0.501	0.500-0.504	0.500-0.667
Complete	0.234	0.234-0.234	0.234	0.234-0.234	0.234-0.234
Cycle	0.500	0.500-0.500	0.500	0.500-0.500	0.500-0.500
House	0.378	0.368-0.380	0.331	0.315-0.349	0.111-0.500
Star	0.023	0.022-0.032	0	0- $10^{-10}$	0.000-0.750
Symmetric	0.228	0.226-0.230	0.220	0.219-0.224	0.219-0.316

TABLE II: Bridging capacity across variations over 5-agent network configurations under traditional neighbor preference.

Fig. 2: The distribution of BCI for various topologies under different agent social preferences under Harvest and Cleanup. The y-axis represents the BCI metric. ‘NN’: Nearest-Neighbor preferences, ‘CN’: Clique-Neighbor preferences, ‘HBN’: Critical-Connection Neighbor preferences. Topologies are present in each subplot. Agents’ colors are consistent across all subfigures.

Fig. 3: Individual mean base reward under different agent preferences across different topologies. Top row: Nearest Neighbor Preference ( $\alpha=1$ ); Middle row: Clique-Neighbor Preference ( $\beta=1$ ); Bottom row: Critical-Connection Neighbor Preference ( $\omega=1$ ). Non-active parameters set to 0. Topologies are shown in the top-right (top-left) corner of each subplot. Star, House has 5 agents; A1 has 9 agents; The agents’ colors are consistent across all subfigures. A2 (intermediate size N=7) shows similar trajectories across preferences; its visualization (same for Fig.5) is therefore deferred to App.Sec.III for readability.

	1st Stage a : After Exploration Trials					2nd Stage: Transient Stage					3rd Stage: At Convergence
Agent-id	0	1	2	3	4	0	1	2	3	4	0	1	2	3	4
Wheel	1.23 b	2.73	2.57	3.56	2.30	46.40	29.85	22.74	21.14	28.75	16.56	5.26	7.76	2.79	13.45
House	3.58	1.99	2.13	3.57	5.73	40.13	77.83	37.93	39.11	75.06	5.18	20.07	19.70	6.45	17.52
Bipartite	1.37	0.87	8.48	3.01	3.55	220.81	120.42	49.12	28.74	47.68	6.21	8.35	4.99	3.65	4.84
Symmetric	0.57	0.61	4.31	3.89	2.62	114.04	211.06	36.39	31.70	54.69	19.47	9.37	9.89	3.93	8.55
Star	1.22	6.71	34.20	40.62	10.34	503.17	128.39	146.90	110.74	82.79	18.51	17.46	24.52	22.07	13.33

TABLE III: Mean Summed individual aggressiveness under nearest neighbor social preference for each stage across five seeds.

Fig. 4: This figure shows overall agents’ individual aggressiveness over the entire experimental agent steps under (a) clique neighbor preference (CN) in 5-agent House and 9-agent A1 topology, and (b) critical-connection preference (HBN) in 5-agent Star and 9-agent A1 topology. The upper subregion subfigure shows agents’ individual aggressiveness in random exploration and transient stages with x-axis limit at

2e7

and original y-scale. This lower subregion subfigure shows agents’ individual aggressiveness at convergence, with the x-axis range:

[0.6e8,1e8]

, and y-axis range:

[0,0.15]

Fig. 5: Individual Good Contribution under different agent preferences across different topologies in Cleanup. Top row: Nearest Neighbor Preference; Middle row: Clique-Neighbor Preference; Bottom row: Critical-Connection Neighbor Preference. Colors are consistent with Fig.3 legend.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Fig. 6: Network Configurations. The first row presents the 5-agent underlying network topologies from left to right:Complete, Wheel, House, Bipartite, Symmetric, and Star. The second row presents the additional three customized network structures: A1, A2, SBM from left to right.

Panel of Figure 10 — Fig. 7: Individual mean base reward under different agent preferences under A2 topology with N=7 (Agent 0 to Agent 7). Top Row: Harvest, Bottem Row: Cleanup. $1^{st}$ column: under the Nearest Neighbor Preference ( $\alpha=1$ ); $2^{nd}$ column: Clique-Neighbor Preference ( $\beta=1$ ); $3^{rd}$ column: Critical-Connection Neighbor Preference ( $\omega=1$ ). Non-active parameters set to 0.

Fig. 8: This figure shows overall agents’ individual aggressiveness over the entire experimental agent steps under (a) clique neighbor preference (CN) in 9-agent A1 and 7-agent A2 topology, and (b) critical-connection preference (HBN) in 5-agent Wheel and 7-agent A2 topology. The upper subregion subfigure shows agents’ individual aggressiveness in random exploration and transient stages with x-axis limit at

2e7

and original y-scale. This lower subregion subfigure shows agents’ individual aggressiveness at convergence, with the x-axis range:

[0.6e8,1e8]

, and y-axis range:

[0,0.15]

Fig. 9: Individual Good Contribution under different agent preferences under A2 in Cleanup.

1^{st}

column: Nearest Neighbor Preference;

2^{nd}

column: Clique-Neighbor Preference;

3^{rd}

column: Critical-Connection Neighbor Preference. Colors are consistent with Fig.3 legend in manuscript.

Fig. 10: Mean utilitarian metric under House topology with different agent preferences in Harvest. ’DWT’ represents the dynamic weights tuning over mixed agent preferences by a population-based training approach, ‘NN’: nearest-neighbor preferences, ‘CN’: the clique-neighbor preferences, ‘HBN’: the critical-connection neighbor preferences. Error Bars illustrate a

95\%

confidence intervals over 5 random seeds.

Multi-Agent Systems

Multi-Agent Systems studies how autonomous agents coordinate, compete, communicate, and learn in shared or interdependent tasks.

See all 675 papers

Self-Referenced Social Preferences: Cooperation without Observing Others Rewards
Oct 6, 2026Mohamed Ayman Mohamed, Harshil Kotamreddy, Marcos Menon JoseCooperative MARL Multi-Agent Reinforcement Learning
Amazon, Vancouver, Canada · Nvidia, Santa Clara, United States · Itaú Unibanco, São Paulo, Brazil
Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas
Jul 6, 2026Yu Wei, Yukiko Ogura, Yoshiyuki Ohmura +3Social Dilemmas Cooperative MARL
The University of Tokyo
Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction
Aug 7, 2026Assaf Caftory, Almog Zemach, Moshe Butman +1Multi-Agent Reinforcement Learning Emergent Behavior in Multi-Agent Systems
EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Faculty of Computer Science, The College of Management Academic Studies, Rishon LeZion, Israel · School of Communication, Reichman University, Herzliya, Israel

Oct 6, 2026J/K move · Enter open · S save

Self-Referenced Social Preferences: Cooperation without Observing Others Rewards

Mohamed Ayman Mohamed, Harshil Kotamreddy, Marcos Menon Jose

Amazon, Vancouver, Canada · Nvidia, Santa Clara, United States · Itaú Unibanco, São Paulo, Brazil

Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Social preferences can promote cooperation in multi-agent reinforcement learning, but existing approaches often require agents to observe the rewards of their peers. In many real-world interactions, however, an agent can, as humans do, observe others' behavior and outcomes without access to their private reward signals. We introduce self-referenced social preferences, in which each agent learns a model of its own reward, applies it to other agents' observed transitions to assess their outcomes from its own perspective, and feeds these self-referenced assessments into standard social preferences. We study two ways to incorporate these assessments: modifying the learning reward, or using them to weight policy updates. We evaluate the approach on three sequential social dilemmas, Escape Room, Clean Up, and Commons Harvest, which require volunteering, public-good contribution, and resource restraint, respectively. Across all three environments, agents learn cooperative behavior without observing others' rewards, including in settings where independent learners fail to cooperate, and frequently achieve more equitable divisions of jointly produced returns than agents with access to true rewards. The effective integration point depends on the social preference: inequity aversion works best in the reward together with a value look-ahead, whereas a purely benevolent preference benefits from policy-update weighting. Under partial observability, the policy-update approach continues to support cooperation. These results show that explicit access to other agents' reward signals is not necessary for learning cooperative behavior: social preferences can instead be grounded in self-referenced assessments of others' outcomes derived from their observed behavior.

Abstract

Multi-Agent Systems

Explore similar work

Self-Referenced Social Preferences: Cooperation without Observing Others Rewards

Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas

Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction