Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

Gunes Akbas; Naci Saldi; Berkay Anahtarci; Can Deha Kariksiz

math.OCOct 11, 2026

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

Authors: Gunes Akbas, Naci Saldi, Berkay Anahtarci, Can Deha Kariksiz

Organizations: Department of Mathematics, Bilkent University, Ankara 06800, Türkiye · Department of Mathematical Engineering, Özyeğin University, İstanbul 34794, Türkiye

Abstract

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kernel, so relative value iteration does not require the minorizing measure. When the Lipschitz constant of the kernel is below the minorization mass and the temperature exceeds an explicit threshold, the equilibrium fixed-point operator is a contraction: an equilibrium exists, its population distribution is unique, and a fixed-point iteration converges to it geometrically. We then learn the equilibrium model-free from a simulator, combining soft Q-learning at a fixed population distribution with an empirical transition kernel. We prove a finite-sample bound for soft Q-learning in the span seminorm, with temperature-independent constants, and derive convergence rates for the learned population distribution and policy. A congestion-aware routing game illustrates the results.

Figures & tables

Model primitives
Route-quality table	$g$	$\left[\begin{smallmatrix}0.85&0.78\\ 0.82&0.68\end{smallmatrix}\right]$
Congestion slope	$\lambda$	$0.15$
Connection value	$w$	$1.00$
Route prices	$(\mathrm{price}_{\mathsf{S}},\mathrm{price}_{\mathsf{C}})$	$(0.50,\,0.20)$
Congestion penalty	$\kappa$	$0.40$
Entropy temperature	$\delta$	$2.00$

Table 1: Parameters of the congestion-aware routing instance. In the route-quality table

g

, rows are indexed by the state

x\in\{G,B\}

and columns by the action

a\in\{\mathsf{S},\mathsf{C}\}

. The derived constants satisfy

K_{1}<\varepsilon

and

\delta>\delta^{\star}

, so the mean-field operator is a contraction.

component	mean-field term $m^{*}$
$m_{G}$ (good connection)	$0.761626$
$m_{B}$ (congestion)	$0.238374$

Table 2: Equilibrium computed by the deterministic mean-field method. The associated optimal entropy-regularized average reward is

\rho^{*}_{\delta}=1.7076

, and the equilibrium policy is

\pi^{*}(\cdot\mid G)=(\mathsf{S}\!:\penalty\!0.472,\,\mathsf{C}\!:\penalty\!0.528)

\pi^{*}(\cdot\mid B)=(\mathsf{S}\!:\penalty\!0.481,\,\mathsf{C}\!:\penalty\!0.519)

Figure 1: Model-free learning on the congestion-aware routing instance (

\delta=2

). (a) Mean-square span error of synchronous soft RVI Q-learning (Algorithm 1 ) at the equilibrium population, for the step sizes

\alpha_{n}=a/(n+n_{0})

with

a=4/\varepsilon

and two values of

n_{0}

; the dotted line is proportional to

1/n

. (b) Model-free mean-field iteration (Algorithm 2 ) with a constant number of inner iterations, compared with the exact iteration; the error settles at a floor of order

n^{-1/2}+N^{-1/2}

. (c) The same algorithm with geometrically growing budgets

n_{k}=N_{k}=\lceil b\,\theta^{-2k}\rceil

\theta=0.8

; the dotted line is proportional to

\theta^{k}

Mean Field Games

Mean field games model strategic decisions in large populations through coupled individual policies and aggregate population dynamics.

See all 19 papers

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward
Jun 15, 2026Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı +1Reinforcement Learning Average-Reward RL
Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey
Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
Dec 30, 2025Lars van der Laan, Nathan KallusReinforcement Learning KL-Regularized RL
Department of Statistics, University of Washington · Netflix · Cornell University
Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games
Apr 6, 2026Narim Jeong, Donghwan LeeStackelberg Games Game-Theoretic RL
Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea

Jun 15, 2026J/K move · Enter open · S save

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

Şevket Kaan Alkır, Naci Saldı, Berkay Anahtarcı, Can Deha Karıksız

Department of Mathematics Bilkent University 06800 Ankara, Turkey · Department of Mathematical Engineering Özyeğin University 34794 Istanbul, Turkey

Reinforcement Learning Average-Reward RL Mean Field Games Maximum Entropy RL Game-Theoretic RL

Open paper AskPDF arXiv

We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium under an unknown reward, and the goal is to recover a policy explaining the observed behaviour via the maximum causal entropy principle. We formulate the inverse problem by enforcing consistency with the expert mean-field term and long-run feature expectations, treating two reward classes within a unified occupation-measure framework. For finite-dimensional linear rewards, we give a convex dual reformulation with an explicit log-partition objective, and prove smoothness and curvature properties justifying constant-step-size gradient descent. For infinite-dimensional RKHS rewards, we develop a Lagrangian relaxation whose inner-maximising policy is characterised by a soft Bellman equation. The main obstacle is the absence of a discount-factor contraction. We resolve this by introducing a minorisation-based sub-stochastic kernel that yields a strict contraction of the soft Bellman operator. We establish Fréchet differentiability and Lipschitz smoothness of the log-likelihood score, leading to a gradient ascent algorithm with convergence guarantees. Two numerical examples, a malware-spread MFG and an RKHS-based consumer-choice model, show that the recovered policies closely match expert behaviour.

Abstract

Mean Field Games

Explore similar work

Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing

Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games