Evolutionary Stability Does Not Guarantee Learning Accessibility: A Multi-Agent Reinforcement Learning Perspective on Cooperation Emergence

Yijie Wang

cs.AISep 23, 2026

Evolutionary Stability Does Not Guarantee Learning Accessibility: A Multi-Agent Reinforcement Learning Perspective on Cooperation Emergence

Authors: Yijie Wang

Organizations: Liupanshui Normal University Liupanshui, Guizhou, China

Abstract

Cooperation emergence is a central problem in multi-agent systems because decentralized agents must coordinate while adapting to the changing behavior of others. Evolutionary game theory identifies strategically stable outcomes, but stability under a population adjustment dynamic need not imply that finite-sample learning agents can reach the same outcome through local reward feedback. We study this distinction in a transparent three-agent governance-motivated game involving a government, a platform firm, and users. We derive replicator dynamics for the fixed stage-game incentives, evaluate the cooperative evolutionary basin on a symmetric initial-condition grid, and compare it with learning-basin estimates for three decentralized value-based learners. The learning analysis uses independent Q-learning with $\varepsilon$ -greedy action selection, scaled Boltzmann exploration, and SA--EA BQL under the same payoff environment and outcome criterion. The evolutionary basin has volume $V_E=1.00$ on the sampled grid. The empirical learning basin is $0.88$ for $\varepsilon$ -IQL and $0.00$ for both scaled Boltzmann and SA--EA BQL. Diagnostic traces show that broader action diversity and nonzero value separation can coexist with failure to sustain the cooperative joint action in this fixed configuration. These results indicate that evolutionary stability and learning accessibility are distinct properties of a coupled game--learning system. The shared-bike setting is a motivating application; the broader contribution is a framework for comparing population-level stability with the finite-sample accessibility of cooperation under specified multi-agent learning dynamics.

Figures & tables

Figure 1: Conceptual framework of evolutionary stability and learning accessibility. The same multi-agent payoff environment is examined through two distinct dynamics. The evolutionary branch updates population strategy frequencies through fitness comparison and replicator adjustment. The MARL branch maps joint actions to realized payoffs, individual Q-value updates, policy updates, and the next interaction.

$g$	$f$	$u$	$(\Pi_{G},\Pi_{F},\Pi_{U})$
0	0	0	$(-L+\lambda_{w}F,\ \pi_{e}+E-\lambda_{w}F,\ 0)$
0	0	1	$(-L+\lambda_{w}F+\beta,\ \pi_{e}+E-\lambda_{w}F,\ a-c_{u})$
0	1	0	$(S-\delta B,\ \pi_{e}-c+\delta B,\ 0)$
0	1	1	$(S-\delta B+\beta,\ \pi_{e}-c+\delta B+\alpha,\ a-c_{u}+q)$
1	0	0	$(-C_{g}+\theta S-L+\lambda_{s}F,\ \pi_{e}+E-\lambda_{s}F,\ 0)$
1	0	1	$(-C_{g}+\theta S-L+\lambda_{s}F+\beta,\ \pi_{e}+E-\lambda_{s}F,\ a-c_{u}+r)$

Table 1: Verified stage-game payoffs

(\Pi_{G},\Pi_{F},\Pi_{U})

Item	Configuration
Agents	3 (government, platform firm, users)
Action space	Binary action for each agent
Training horizon	20,000 episodes per run
Random seeds	50
Initial conditions	Symmetric initial-preference grid
Algorithms	$\varepsilon$ -IQL; scaled Boltzmann; SA–EA BQL

Table 2: Fixed experimental protocol for the learning-basin comparison.

Figure 2: Comparison between evolutionary stability and learning accessibility. Although the replicator dynamics reaches the cooperative basin for all evaluated initial conditions, finite-sample reinforcement learning exhibits learning-mechanism-dependent accessibility.

Figure 3: Replicator stability on the frozen symmetric initial-condition grid. (a) Three verified trajectories project toward the cooperative corner

(1,1,1)

. (b) Each evaluated value of

p

converges cooperatively; this discrete classification does not infer a continuous basin.

Figure 4: Learning-basin curves for the three frozen reinforcement-learning comparisons. Points are observed success probabilities at the nine evaluated symmetric initial preferences; no interpolation is used.

Figure 5: Theory–learning gap in cooperative basin volume. The interval for

\varepsilon

-IQL is the frozen bootstrap 95% confidence interval; the two zero estimates are exact outcomes in their respective 450-run grids.

Figure 6: Exploration diversity alone does not determine cooperative emergence. Points summarize the three players within each seed; error bars show empirical 2.5th–97.5th percentiles across 50 seeds. Action coverage is measured over the recorded run, whereas entropy and Q-value separation use the terminal logged episode.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Item	Fixed configuration
Number of agents	3 (government, platform firm, users)
Action space	Binary action for each agent
Episodes per run	20,000
Random seeds	50
Initial-preference grid	$0.1,0.2,\ldots,0.9$ (symmetric)
Algorithms	$\varepsilon$ -IQL; scaled Boltzmann; SA–EA BQL

Appendix

Table 3: Fixed configuration for the learning-basin comparison.

Emergent Behavior in Multi-Agent Systems

Emergent behavior in multi-agent systems studies how local agent interactions produce collective patterns such as cooperation, norms, and instability.

See all 170 papers

The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
Apr 17, 2026Deep Kumar Ganguly, Chandradithya S Jonnalagadda, Pratham Chintamani +1Multi-Agent Reinforcement Learning Multi-Agent Coordination
Technical University of Munich (TU Munich) · Brown University · Indian Institute of Technology Tirupati (IIT Tirupati)
An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback
Sep 5, 2026Lichen Wang, Shijia Hua, Linjie LiuMean-Field MARL Multi-Agent Reinforcement Learning
College of Science, Northwest A & F University, Yangling, 712100, China
Learning Dynamic Games for Cooperative Iterative Tasks
Oct 20, 2026Mark Pustilnik, Francesco BorrelliMulti-Agent Control Game Theory
University of California at Berkeley, USA

Apr 17, 2026J/K move · Enter open · S save

The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning

Deep Kumar Ganguly, Chandradithya S Jonnalagadda, Pratham Chintamani, Adithya Ananth

Technical University of Munich (TU Munich) · Brown University · Indian Institute of Technology Tirupati (IIT Tirupati)

Multi-Agent Reinforcement Learning Multi-Agent Coordination Non-Stationary RL Game Theory Risk-Sensitive RL Cooperative Game Theory+3

Open paper AskPDF arXiv

Cooperative equilibria are fragile. When agents learn alongside each other rather than in a fixed environment, the process of learning destabilizes the cooperation they are trying to sustain: every gradient step an agent takes shifts the distribution of actions its partner will play, turning a cooperative partner into a source of stochastic noise precisely where the cooperation decision is most sensitive. We study how this co-learning noise propagates through the structure of coordination games, and find that the cooperative equilibrium, even when strongly Pareto-dominant, is exponentially unstable under standard risk-neutral learning, collapsing irreversibly once partner noise crosses the game's critical cooperation threshold. The natural response to apply distributional robustness to hedge against partner uncertainty makes things strictly worse: risk-averse return objectives penalize the high-variance cooperative action relative to defection, widening the instability region rather than shrinking it, a paradox that reveals a fundamental mismatch between the domains where robustness is applied and instability originates. We resolve this by showing that robustness should target the policy gradient update variance induced by partner uncertainty, not the return distribution. This distinction yields an algorithm whose gradient updates are modulated by an online measure of partner unpredictability, provably expanding the cooperation basin in symmetric coordination games. To unify stability, sample complexity, and welfare consequences of this approach, we introduce the Price of Paranoia as the structural dual of the Price of Anarchy. Together with a novel Cooperation Window, it precisely characterizes how much welfare learning algorithms can recover under partner noise, pinning down the optimal degree of robustness as a closed-form balance between equilibrium stability and sample efficiency.

Abstract

Emergent Behavior in Multi-Agent Systems

Explore similar work

The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning

An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

Learning Dynamic Games for Cooperative Iterative Tasks