Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

Adonis Jamal; Samy Mekkaoui; Yadh Hafsi; Huyên Pham

math.OCOct 8, 2026

Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

Authors: Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huyên Pham

Organizations: École Polytechnique

Abstract

We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.

Figures & tables

Benchmark	$J(\theta^{\star})$	REINFORCE	MF-REINFORCE	at $\lambda_{\star}$	best scale	best scale
				Transport		Transport-Proba
Two-state	$-2.64$	$29.0\pm 0.0$	$0.637\pm 0.093$	$1.36\pm 0.13$	$\mathbf{0.359\pm 0.091}$ $(\lambda_{\star}/4)$	—
Distribution	$-0.05699$	$0.206\pm 0.002$	$0.118\pm 0.024$	$0.0296\pm 0.0012$	$\mathbf{0.0255\pm 0.0011}$ $(\lambda_{\star}/2)$	—
Linear–quadratic	$-7.224$	$0.216\pm 0.002$	—	$0.0390\pm 0.0027$	$\mathbf{0.0196\pm 0.0012}$ $(\lambda_{\star}/4)$	$0.0469\pm 0.0028$ $(\lambda_{\star}/2)$
Portfolio	$-13.15$	$6.87\pm 0.07$	—	$1.22\pm 0.07$	$\mathbf{0.586\pm 0.152}$ $(\lambda_{\star}/4)$	$0.692\pm 0.156$ $(\lambda_{\star}/2)$
Bimodal ( $K=2$ )	$0$	$0.175\pm 0.009$	—	$0.0903\pm 0.0053$	$\mathbf{0.00199\pm 0.00094}$ $(\lambda_{\star}/8)$	—

Table 1: Final optimality gap

|J(\widehat{\theta})-J(\theta^{\star})|

(mean

\pm

std. over five paired seeds; smaller is better). Transport is shown at

\lambda_{\star}=B^{-1/4}

and at its best tested scale in parentheses. Transport-Proba uses its best tested scale. Scale choices and evaluation of the reference policies are detailed in Appendix E.1 .

Error	Swept	$p_{0}$	Measured $p$
Sensitivity MSE	$\eta$ (exact)	$4$	$[3.99,4.02]$
Sensitivity MSE	$n$	$-1$	$[-1.23,-0.51]$
Gradient bias	$\lambda$	$1$	$[1.00,1.28]$
Gradient variance	$B$	$-1$	$[-1.03,-0.90]$
Population MSE	$M$	$-1$	$[-1.07,-0.98]$

Table 2: Predicted powers

p_{0}

and ranges of fitted log-log slopes

p

across the six benchmarks in Table 9 . The exact-flow

\eta

sweep uses the two continuous benchmarks. Fitting details are in Appendix E.3.6 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$T$	$M$	$n$	$B$	Budget	Updates	Step size	$\lambda_{\star}$	$\eta$	$\sigma$	$K$
Two-state	5	200	12	248	2 300	20 000	$10^{-3}$	$0.252$	$0.98$	$0.75$	—
Cybersecurity	3	200	51	153	1 212	20 000	$10^{-3}$	$0.284$	$0.98$	$0.75$	—
Distribution	5	500	280	280	5 300	30 000	$10^{-4}$	$0.244$	$0.98$	$0.5$	—
Advertising	5	200	65	195	2 300	10 000	$10^{-3}$	$0.268$	$0.95$	$1$	—
Linear–quadratic	20	150	10 240	111	210 020	10 000	$10^{-3}$	$0.308$	$0.95$	—	1
Portfolio	10	100	5 120	211	54 310	50 000	$10^{-2}$	$0.262$	$0.95$	—	1

Appendix

Table 3: Transport allocations and optimization settings.

M

n

and

B

are the population, auxiliary and main sample sizes, and the budget is the resulting number of simulated transitions per update,

T(M+n+B)

, which every method of a benchmark receives. Transport is trained at

\lambda\in\lambda_{\star}\{1/8,1/4,1/2,1,2\}

, with

\lambda_{\star}=B^{-1/4}

;

\sigma

is the finite-state simplex randomizer scale and

K

the number of mixture components.

Benchmark	$T$	Reference	REINFORCE	MF-REINFORCE	Transport	$(\lambda,\eta)$
Cybersecurity	3	—	$-0.4887\pm 0.0007$	$-0.4986\pm 0.0142$	$-0.4883\pm 0.0010$	$(0.568666,0.98)$
Advertising	5	$1.0188$	$0.9681\pm 0.0036$	$0.9921\pm 0.0176$	$0.9821\pm 0.0159$	$(0.133802,0.95)$

Appendix

Table 4: Final validation objective on the two benchmarks absent from Table 1 (mean

\pm

standard deviation over five seeds; larger is better). Transport uses the scale with the highest mean final objective. Reference values are computed from the benchmark models; gap evaluation is described in Appendix E.1 .

Estimator	$K$	$\lambda$	$\|J(\widehat{\theta})-J(\theta^{\star})\|$	$\widehat{\theta}$
REINFORCE	—	—	$0.1752\pm 0.0087$	$0.8000\pm 0.0000$
Transport	1	$0.00702927$	$0.1425\pm 0.1239$	$0.8103\pm 0.0339$
Transport	1	$0.0140585$	$0.1425\pm 0.1239$	$0.8103\pm 0.0339$
Transport	1	$0.0281171$	$0.1425\pm 0.1240$	$0.8103\pm 0.0339$
Transport	1	$0.0562341$	$0.1425\pm 0.1241$	$0.8102\pm 0.0339$
Transport	1	$0.112468$	$0.1426\pm 0.1245$	$0.8102\pm 0.0340$

Appendix

Table 5: Bimodal allocation: final gap and policy parameter, initialized at

\theta=0.8

(exact gap

0.1772

). Gaps are Monte Carlo validation estimates. The optimum is

\theta^{\star}=0.868437

. For

K=1

, the projected objective is constant,

J_{1}=-0.923831

; for

K=2

, the representation is exact.

Benchmark	Estimator	Scale	Value change	Opt. error	Final gap
Two-state	Transport	$0.0315$	$0.1191$	$0.3730\pm 0.0617$	$0.5452\pm 0.1134$
		$0.063$	$0.2394$	$0.1032\pm 0.0346$	$0.4427\pm 0.0895$
		$0.126$	$0.4848$	$0.0153\pm 0.0085$	$0.5968\pm 0.0782$
		$0.252$	$0.9970$	$0.0044\pm 0.0027$	$1.4465\pm 0.0889$
		$0.504$	$2.1475$	$0.0074\pm 0.0028$	$3.6452\pm 0.0809$
	MF-REINFORCE	$0.2$	$2.6476$	$0.1892\pm 0.0445$	$0.7098\pm 0.0915$

Appendix

Table 6: Effect of perturbation on objective values and optimization. Value change :

|J^{\lambda}(\theta_{\lambda}^{\star})-J(\theta^{\star})|

; Opt. error :

|J^{\lambda}(\widehat{\theta}_{\lambda})-J^{\lambda}(\theta_{\lambda}^{\star})|

; Final gap :

|J(\widehat{\theta}_{\lambda})-J(\theta^{\star})|

. Objectives are evaluated from the population equations; dispersion is over five seeds. MF-REINFORCE uses its own perturbed objective and scale

\varepsilon

Benchmark	Estimator	Transitions/update	Seconds/run	Ratio
Two-state	REINFORCE	$2\,300$	$70$	$1.0\times$
	MF-REINFORCE	$2\,300$	$387$	$5.5\times$
	Transport	$2\,300$	$220$	$3.1\times$
Cybersecurity	REINFORCE	$1\,212$	$167$	$1.0\times$
	MF-REINFORCE	$1\,212$	$595$	$3.6\times$
	Transport	$1\,212$	$393$	$2.4\times$

Appendix

Table 7: Simulator transitions per update and wall-clock time per complete training run. Each benchmark uses the same simulator budget across methods.

Benchmark	$n$	$\eta$	Sensitivity error		Gradient error
			shared batch	fresh batches	shared batch
Two-state	12	$0.269$	$4.33\cdot 10^{4}$	$3.93\cdot 10^{4}$	$8.46\cdot 10^{3}$
		$0.537$	$31.6$	$32.2$	$15.4$
		$0.85$	$1.26$	$1.31$	$2.91$
		$0.95$	$1.05$	$1.07$	$2.75$
		$0.98$	$1.04$	$1.06$	$2.75$

Appendix

Table 8: Relative sensitivity MSE

\sum_{t}\mathbb{E}\big[\|\widehat{D}_{t}-D_{t}\|_{F}^{2}\big]/\sum_{t}\|D_{t}\|_{F}^{2}

(200 replications) and gradient MSE

\mathbb{E}\big[\|\widehat{G}-\nabla_{\theta}J\|^{2}\big]/\|\nabla_{\theta}J\|^{2}

(100 replications). The population flow is exact; the policy is fixed at initialization and

\lambda=B^{-1/4}

. The first scales are

\eta_{\star}/2

\eta_{\star}

and, when below one,

2\eta_{\star}

, with

\eta_{\star}=n^{-1/4}

Quantity	Swept	LQ	Portfolio	Two-state	Cybersecurity	Distribution	Advertising	$p_{0}$
$\mathbb{E}\\|\widehat{D}-D\\|^{2}$	$\eta$ (exact flows)	$+4.02$	$+3.99$	—	—	—	—	$+4$
$\mathbb{E}\\|\widehat{D}-D\\|^{2}$	$n$	$-1.23$	$-1.14$	$-1.06$	$-1.00$	$-1.02$	$-0.51$	$-1$
$\\|\mathbb{E}\widehat{G}-\nabla J\\|$	$\lambda$	$+1.02$	$+1.02$	$+1.28$	$+1.00$	$+1.00$	$+1.00$	$+1$
$\mathbb{E}\\|\widehat{G}-\mathbb{E}\widehat{G}\\|^{2}$	$B$	$-0.90$	$-0.95$	$-0.99$	$-1.01$	$-1.03$	$-0.92$	$-1$
$\mathbb{E}\\|\widehat{z}-z\\|^{2}$	$M$	$-1.07$	$-0.98$	$-1.02$	$-1.02$	$-1.00$	$-1.06$	$-1$

Appendix

Table 9: Log-log slopes for all benchmarks. Each row varies one quantity, holding the remaining quantities at the benchmark allocation, except that the continuous-state

\lambda

sweep uses

B=4096

. The

\lambda

slopes are fitted on the three smallest scales at which the bias is resolved, as described above.

Explore similar work

Jul 13, 2026math.OC

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

This paper develops a model-free reinforcement learning framework for continuous--time extended mean field control problems, where both the dynamics and reward may depend on the joint distribution of states and controls. We adopt deterministic feedback policies, under which the state--action distribution is induced directly as a push--forward of the state law. This avoids optimization over stochastic kernels and bypasses key limitations of existing approaches in extended mean field settings. We first establish a model--free sensitivity formula for parameterized McKean--Vlasov dynamics and use it to derive a deterministic policy gradient formula expressed through an advantage--rate function on the Wasserstein space. We then refine this formula by introducing local value and advantage--rate representations that depend on the state, action, and joint state--action distribution, yielding a policy gradient that includes both action derivatives and measure--derivative terms with respect to the control distribution. These characterizations lead to a martingale--based learning principle and motivate a continuous--time deep deterministic policy gradient algorithm combining particle approximations, measure--dependent neural networks, temporal--difference learning, and exploration in either action or parameter space. Numerical experiments on stochastic Cucker--Smale consensus control and optimal liquidation with trade crowding demonstrate the efficiency, stability, and robustness of the proposed method, including problems with explicit dependence on the control distribution.

Ziheng Cheng, Xin Guo, Huyên Pham +1

University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London

Policy Gradient Continuous-Time RL Model-Free RL Deep Deterministic Policy Gradient Actor-Critic Methods Policy Learning

Ask PDF

Jun 25, 2026math.OC

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available. In the model-based formulation, policy evaluation is naturally described by a stationary Hamilton-Jacobi-Bellman equation on

\mathcal P_2(\mathbb R^d)

, but this equation involves the drift and diffusion coefficients of the controlled McKean-Vlasov dynamics, which are not identifiable when only discrete-time data are available. On the other hand, a direct reduction to a time-discrete Bellman equation avoids the non-identifiability issue but loses the differential equation structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value with an error of order

Δt

. In the linear-quadratic case, we show our approximation achieves second-order accuracy with only one-step data. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.

Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1

Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.

Reinforcement Learning Stochastic Optimal Control Continuous-Time RL Model-Free RL Actor-Critic Methods

Ask PDF

Jul 1, 2026math.OC

Mean Field Reinforcement Learning

This monograph provides an introduction to mean field reinforcement learning through the lens of Markov decision processes arising from large-population stochastic control with mean field interactions and common noise. Starting from the connection between multi-agent reinforcement learning and mean field control, it develops the probabilistic, mathematical, and control-theoretic framework needed to formulate representative-agent learning problems, analyze their relationship with finite-population systems, and study both general and linear-quadratic models. The presentation includes dynamic programming principles, propagation-of-chaos limits, and theoretical analyses of tabular Q-learning and policy-gradient methods. It also discusses numerical implementations, including tabular schemes and deep reinforcement learning methods such as deep deterministic policy gradient. The goal is to give readers a coherent bridge between mean field control theory and reinforcement learning methodology, emphasizing the mathematical structure of the problems and the design of tractable learning approaches for large stochastic populations.

René Carmona, Mathieu Laurière

Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China

Reinforcement Learning Q-Learning Policy Gradient Methods Policy Learning

Ask PDF