An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

Lichen Wang; Shijia Hua; Linjie Liu

cs.MASep 5, 2026

An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

Authors: Lichen Wang, Shijia Hua, Linjie Liu

Organizations: College of Science, Northwest A & F University, Yangling, 712100, China

Abstract

Multi-agent reinforcement learning in networked populations is governed by the interaction between individual adaptation, local encounters, and changing environmental conditions. To study this interaction, we formulate a coupled learning--environment model in which agents update stateless $Q$ -values on a fixed graph, while their population-average behavior drives an environmental variable that dynamically modifies the payoff matrix. Under a first-order mean-field closure, we derive a deterministic transport equation for the population distribution of $Q$ -values and couple it with a projected discrete update for the environmental state. The resulting model is evaluated against finite-network Monte Carlo simulations on random regular, Erdős--Rényi, Barabási--Albert, and random geometric graphs. Across the tested parameter ranges, the mean-field system reproduces the main macroscopic cooperation and environmental trajectories, and the trajectory-level root-mean-square error generally decreases with population size and average degree. The analysis further shows that environmental feedback reshapes the learned action-value ordering, while reinforcing feedback can produce pronounced dependence on the initial learning bias and resource level. The environmental timescale also plays an important role: a rapid response can drive the resource state to a boundary before learning adapts, whereas a slower response preserves the interaction between behavioral learning and environmental recovery. These results provide a population-level description of coupled reinforcement learning and environmental dynamics and characterize the performance of the mean-field approximation within the tested network and parameter ranges.

Figures & tables

q_{i}(A_{i}(t),t+1)\leftarrow q_{i}(A_{i}(t),t)+\alpha\bigl(r_{i}(t)-q_{i}(A_{i}(t),t)\bigr)

Algorithm 1 Finite-Network Monte Carlo Simulation of Stateless

Q

-Learning with Environmental Feedback

Fig. 1: Micro-to-macro representation of the coupled learning–environment dynamics for the binary-action case

\mathcal{A}=\{C,D\}

. The finite-network Monte Carlo simulation and the mean-field transport equation follow the same synchronous learning-round convention and are coupled through the population-average strategy and the projected environmental update.

Fig. 2: Comparison of agent-based simulations and mean-field transport predictions under Env. 3 across four network topologies. Panels A–D correspond to RG, ER, BA, and RGG networks, respectively. Within each panel, the three columns use $(\eta_{1},\eta_{2},n_{0})=(2,8,0.5)$ , $(5,5,0.5)$ , and $(8,2,0.8)$ , respectively. The first row shows the initial $Q$ -value densities, the second row shows the population-average cooperation and defection probabilities, and the third row shows the environmental state $n(t)$ . Markers denote averages over 100 independent agent-based runs, and solid lines denote the corresponding mean-field predictions.

Fig. 3: Trajectory-level root-mean-square errors between finite-network simulations and the mean-field transport prediction under Env. 3. Panels A–D show the RMSE of the population-average cooperation trajectory

\bar{x}(C,t)

, whereas Panels E–H show the RMSE of the environmental trajectory

n(t)

. The columns correspond to RG, ER, BA, and RGG networks, respectively. The horizontal axis denotes the population size

N

, and the curves correspond to average degrees

k=4

10

, and

16

. Each data point represents the mean RMSE over 100 simulations, and the error bars indicate the corresponding standard deviation.

Fig. 4: Population-level learning dynamics on a BA network under the two static payoff configurations. Panels A and B compare the population-average cooperation and defection probabilities obtained from Monte Carlo (MC) simulations and the mean-field (MF) model under Env. 1 and Env. 2, respectively. Panel C shows the MC population-average values of

q(C)

and

q(D)

, with solid and dashed curves corresponding to Env. 1 and Env. 2, respectively.

Fig. 5: Late-time Monte Carlo estimates under Env. 3 in the $(\beta,\theta)$ parameter plane. Panels A–C show the population-average cooperation probability $\bar{x}(C)$ , the environmental state $n$ , and the population-average action-value difference $\bar{q}(C)-\bar{q}(D)$ , respectively.

Fig. 6: Projected mean-field trajectories under reinforcing payoff feedback (Env. 4) in the population-average cooperation–environment plane

(\bar{x}(C),n)

. Panels A–C correspond to

\theta=0.1

1.0

, and

2.0

, respectively, and arrows indicate the direction of increasing time.

Fig. 7: Mean-field transport predictions under different environmental response timescales. Panel A corresponds to Env. 3, and Panel B corresponds to Env. 4. In each panel, the upper and lower subplots show the population-average cooperation level

\bar{x}(C,t)

and the environmental state

n(t)

, respectively.

Mean-Field MARL

See all 10 papers

Mar 4, 2026

Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling

Many large-scale platforms and networked control systems have a centralized decision maker interacting with a massive population of agents under strict observability constraints. Motivated by such applications, we study a cooperative Markov game with a global agent and

n

homogeneous local agents in a communication-constrained regime, where the global agent only observes a subset of

k

local agent states per time step. We propose an alternating learning framework

(\texttt{ALTERNATING-MARL})

, where the global agent performs subsampled mean-field

Q

-learning against a fixed local policy, and local agents update by optimizing in an induced MDP. We prove that these approximate best-response dynamics converge to an

\widetilde{O}(1/\sqrt{k})

-approximate Nash Equilibrium, while separating the sample complexities between the joint state and action spaces. Finally, we validate our results in numerical simulations for multi-robot control.

Emile Anand, Ishani Karmarkar

Georgia Institute of Technology, Atlanta, GA · Stanford University, Palo Alto, CA

Mean-Field MARL Reinforcement Learning Cooperative MARL Multi-Agent Coordination Partially Observable RL

Ask PDF

Feb 20, 2026

Mean-Field Reinforcement Learning without Synchrony

Mean-field reinforcement learning (MF-RL) scales multi-agent RL to large populations by reducing each agent's dependence on others to a single summary statistic -- the mean action. However, this reduction requires every agent to act at every time step; when some agents are idle, the mean action is simply undefined. Addressing asynchrony therefore requires a different summary statistic -- one that remains defined regardless of which agents act. The population distribution

μ\in Δ(\mathcal{O})

-- the fraction of agents at each observation -- satisfies this requirement: its dimension is independent of

N

, and under exchangeability it fully determines each agent's reward and transition. Existing MF-RL theory, however, is built on the mean action and does not extend to

μ

. We therefore construct the Temporal Mean Field (TMF) framework around the population distribution

μ

from scratch, covering the full spectrum from fully synchronous to purely sequential decision-making within a single theory. We prove existence and uniqueness of TMF equilibria, establish an

O(1/\sqrt{N})

finite-population approximation bound that holds regardless of how many agents act per step, and prove convergence of a policy gradient algorithm (TMF-PG) to the unique equilibrium. Experiments on a resource selection game and a dynamic queueing game confirm that TMF-PG achieves near-identical performance whether one agent or all

N

act per step, with approximation error decaying at the predicted

O(1/\sqrt{N})

rate.

Shan Yang

Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.

Mean-Field MARL Reinforcement Learning Multi-Agent Reinforcement Learning Asynchronous RL Sequential MARL

Ask PDF

Jul 1, 2026

Mean Field Reinforcement Learning

This monograph provides an introduction to mean field reinforcement learning through the lens of Markov decision processes arising from large-population stochastic control with mean field interactions and common noise. Starting from the connection between multi-agent reinforcement learning and mean field control, it develops the probabilistic, mathematical, and control-theoretic framework needed to formulate representative-agent learning problems, analyze their relationship with finite-population systems, and study both general and linear-quadratic models. The presentation includes dynamic programming principles, propagation-of-chaos limits, and theoretical analyses of tabular Q-learning and policy-gradient methods. It also discusses numerical implementations, including tabular schemes and deep reinforcement learning methods such as deep deterministic policy gradient. The goal is to give readers a coherent bridge between mean field control theory and reinforcement learning methodology, emphasizing the mathematical structure of the problems and the design of tractable learning approaches for large stochastic populations.

René Carmona, Mathieu Laurière

Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China

Reinforcement Learning Q-Learning Policy Gradient Methods Mean-Field MARL Policy Learning

Ask PDF

Abstract

Mean-Field MARL

Explore similar work

Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling

Mean-Field Reinforcement Learning without Synchrony

Mean Field Reinforcement Learning