Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

Adonis Jamal; Samy Mekkaoui; Yadh Hafsi; Huyên Pham

math.OCOct 8, 2026

Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

Authors: Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huyên Pham

Organizations: École Polytechnique

Abstract

We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.

Figures & tables

Benchmark	$J(\theta^{\star})$	REINFORCE	MF-REINFORCE	at $\lambda_{\star}$	best scale	best scale
				Transport		Transport-Proba
Two-state	$-2.64$	$29.0\pm 0.0$	$0.637\pm 0.093$	$1.36\pm 0.13$	$\mathbf{0.359\pm 0.091}$ $(\lambda_{\star}/4)$	—
Distribution	$-0.05699$	$0.206\pm 0.002$	$0.118\pm 0.024$	$0.0296\pm 0.0012$	$\mathbf{0.0255\pm 0.0011}$ $(\lambda_{\star}/2)$	—
Linear–quadratic	$-7.224$	$0.216\pm 0.002$	—	$0.0390\pm 0.0027$	$\mathbf{0.0196\pm 0.0012}$ $(\lambda_{\star}/4)$	$0.0469\pm 0.0028$ $(\lambda_{\star}/2)$
Portfolio	$-13.15$	$6.87\pm 0.07$	—	$1.22\pm 0.07$	$\mathbf{0.586\pm 0.152}$ $(\lambda_{\star}/4)$	$0.692\pm 0.156$ $(\lambda_{\star}/2)$
Bimodal ( $K=2$ )	$0$	$0.175\pm 0.009$	—	$0.0903\pm 0.0053$	$\mathbf{0.00199\pm 0.00094}$ $(\lambda_{\star}/8)$	—

Table 1: Final optimality gap

|J(\widehat{\theta})-J(\theta^{\star})|

(mean

\pm

std. over five paired seeds; smaller is better). Transport is shown at

\lambda_{\star}=B^{-1/4}

and at its best tested scale in parentheses. Transport-Proba uses its best tested scale. Scale choices and evaluation of the reference policies are detailed in Appendix E.1 .

Error	Swept	$p_{0}$	Measured $p$
Sensitivity MSE	$\eta$ (exact)	$4$	$[3.99,4.02]$
Sensitivity MSE	$n$	$-1$	$[-1.23,-0.51]$
Gradient bias	$\lambda$	$1$	$[1.00,1.28]$
Gradient variance	$B$	$-1$	$[-1.03,-0.90]$
Population MSE	$M$	$-1$	$[-1.07,-0.98]$

Table 2: Predicted powers

p_{0}

and ranges of fitted log-log slopes

p

across the six benchmarks in Table 9 . The exact-flow

\eta

sweep uses the two continuous benchmarks. Fitting details are in Appendix E.3.6 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$T$	$M$	$n$	$B$	Budget	Updates	Step size	$\lambda_{\star}$	$\eta$	$\sigma$	$K$
Two-state	5	200	12	248	2 300	20 000	$10^{-3}$	$0.252$	$0.98$	$0.75$	—
Cybersecurity	3	200	51	153	1 212	20 000	$10^{-3}$	$0.284$	$0.98$	$0.75$	—
Distribution	5	500	280	280	5 300	30 000	$10^{-4}$	$0.244$	$0.98$	$0.5$	—
Advertising	5	200	65	195	2 300	10 000	$10^{-3}$	$0.268$	$0.95$	$1$	—
Linear–quadratic	20	150	10 240	111	210 020	10 000	$10^{-3}$	$0.308$	$0.95$	—	1
Portfolio	10	100	5 120	211	54 310	50 000	$10^{-2}$	$0.262$	$0.95$	—	1

Appendix

Table 3: Transport allocations and optimization settings.

M

n

and

B

are the population, auxiliary and main sample sizes, and the budget is the resulting number of simulated transitions per update,

T(M+n+B)

, which every method of a benchmark receives. Transport is trained at

\lambda\in\lambda_{\star}\{1/8,1/4,1/2,1,2\}

, with

\lambda_{\star}=B^{-1/4}

;

\sigma

is the finite-state simplex randomizer scale and

K

the number of mixture components.

Benchmark	$T$	Reference	REINFORCE	MF-REINFORCE	Transport	$(\lambda,\eta)$
Cybersecurity	3	—	$-0.4887\pm 0.0007$	$-0.4986\pm 0.0142$	$-0.4883\pm 0.0010$	$(0.568666,0.98)$
Advertising	5	$1.0188$	$0.9681\pm 0.0036$	$0.9921\pm 0.0176$	$0.9821\pm 0.0159$	$(0.133802,0.95)$

Appendix

Table 4: Final validation objective on the two benchmarks absent from Table 1 (mean

\pm

standard deviation over five seeds; larger is better). Transport uses the scale with the highest mean final objective. Reference values are computed from the benchmark models; gap evaluation is described in Appendix E.1 .

Estimator	$K$	$\lambda$	$\|J(\widehat{\theta})-J(\theta^{\star})\|$	$\widehat{\theta}$
REINFORCE	—	—	$0.1752\pm 0.0087$	$0.8000\pm 0.0000$
Transport	1	$0.00702927$	$0.1425\pm 0.1239$	$0.8103\pm 0.0339$
Transport	1	$0.0140585$	$0.1425\pm 0.1239$	$0.8103\pm 0.0339$
Transport	1	$0.0281171$	$0.1425\pm 0.1240$	$0.8103\pm 0.0339$
Transport	1	$0.0562341$	$0.1425\pm 0.1241$	$0.8102\pm 0.0339$
Transport	1	$0.112468$	$0.1426\pm 0.1245$	$0.8102\pm 0.0340$

Appendix

Table 5: Bimodal allocation: final gap and policy parameter, initialized at

\theta=0.8

(exact gap

0.1772

). Gaps are Monte Carlo validation estimates. The optimum is

\theta^{\star}=0.868437

. For

K=1

, the projected objective is constant,

J_{1}=-0.923831

; for

K=2

, the representation is exact.

Benchmark	Estimator	Scale	Value change	Opt. error	Final gap
Two-state	Transport	$0.0315$	$0.1191$	$0.3730\pm 0.0617$	$0.5452\pm 0.1134$
		$0.063$	$0.2394$	$0.1032\pm 0.0346$	$0.4427\pm 0.0895$
		$0.126$	$0.4848$	$0.0153\pm 0.0085$	$0.5968\pm 0.0782$
		$0.252$	$0.9970$	$0.0044\pm 0.0027$	$1.4465\pm 0.0889$
		$0.504$	$2.1475$	$0.0074\pm 0.0028$	$3.6452\pm 0.0809$
	MF-REINFORCE	$0.2$	$2.6476$	$0.1892\pm 0.0445$	$0.7098\pm 0.0915$

Appendix

Table 6: Effect of perturbation on objective values and optimization. Value change :

|J^{\lambda}(\theta_{\lambda}^{\star})-J(\theta^{\star})|

; Opt. error :

|J^{\lambda}(\widehat{\theta}_{\lambda})-J^{\lambda}(\theta_{\lambda}^{\star})|

; Final gap :

|J(\widehat{\theta}_{\lambda})-J(\theta^{\star})|

. Objectives are evaluated from the population equations; dispersion is over five seeds. MF-REINFORCE uses its own perturbed objective and scale

\varepsilon

Benchmark	Estimator	Transitions/update	Seconds/run	Ratio
Two-state	REINFORCE	$2\,300$	$70$	$1.0\times$
	MF-REINFORCE	$2\,300$	$387$	$5.5\times$
	Transport	$2\,300$	$220$	$3.1\times$
Cybersecurity	REINFORCE	$1\,212$	$167$	$1.0\times$
	MF-REINFORCE	$1\,212$	$595$	$3.6\times$
	Transport	$1\,212$	$393$	$2.4\times$

Appendix

Table 7: Simulator transitions per update and wall-clock time per complete training run. Each benchmark uses the same simulator budget across methods.

Benchmark	$n$	$\eta$	Sensitivity error		Gradient error
			shared batch	fresh batches	shared batch
Two-state	12	$0.269$	$4.33\cdot 10^{4}$	$3.93\cdot 10^{4}$	$8.46\cdot 10^{3}$
		$0.537$	$31.6$	$32.2$	$15.4$
		$0.85$	$1.26$	$1.31$	$2.91$
		$0.95$	$1.05$	$1.07$	$2.75$
		$0.98$	$1.04$	$1.06$	$2.75$

Appendix

Table 8: Relative sensitivity MSE

\sum_{t}\mathbb{E}\big[\|\widehat{D}_{t}-D_{t}\|_{F}^{2}\big]/\sum_{t}\|D_{t}\|_{F}^{2}

(200 replications) and gradient MSE

\mathbb{E}\big[\|\widehat{G}-\nabla_{\theta}J\|^{2}\big]/\|\nabla_{\theta}J\|^{2}

(100 replications). The population flow is exact; the policy is fixed at initialization and

\lambda=B^{-1/4}

. The first scales are

\eta_{\star}/2

\eta_{\star}

and, when below one,

2\eta_{\star}

, with

\eta_{\star}=n^{-1/4}

Quantity	Swept	LQ	Portfolio	Two-state	Cybersecurity	Distribution	Advertising	$p_{0}$
$\mathbb{E}\\|\widehat{D}-D\\|^{2}$	$\eta$ (exact flows)	$+4.02$	$+3.99$	—	—	—	—	$+4$
$\mathbb{E}\\|\widehat{D}-D\\|^{2}$	$n$	$-1.23$	$-1.14$	$-1.06$	$-1.00$	$-1.02$	$-0.51$	$-1$
$\\|\mathbb{E}\widehat{G}-\nabla J\\|$	$\lambda$	$+1.02$	$+1.02$	$+1.28$	$+1.00$	$+1.00$	$+1.00$	$+1$
$\mathbb{E}\\|\widehat{G}-\mathbb{E}\widehat{G}\\|^{2}$	$B$	$-0.90$	$-0.95$	$-0.99$	$-1.01$	$-1.03$	$-0.92$	$-1$
$\mathbb{E}\\|\widehat{z}-z\\|^{2}$	$M$	$-1.07$	$-0.98$	$-1.02$	$-1.02$	$-1.00$	$-1.06$	$-1$

Appendix

Table 9: Log-log slopes for all benchmarks. Each row varies one quantity, holding the remaining quantities at the benchmark allocation, except that the continuous-state

\lambda

sweep uses

B=4096

. The

\lambda

slopes are fitted on the three smallest scales at which the bias is resolved, as described above.

Explore similar work

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies
Jul 13, 2026Ziheng Cheng, Xin Guo, Huyên Pham +1Policy Gradient Continuous-Time RL
University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London
Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
Jun 25, 2026Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1Reinforcement Learning Stochastic Optimal Control
Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.
Mean Field Reinforcement Learning
Jul 1, 2026René Carmona, Mathieu LaurièreReinforcement Learning Q-Learning
Department of Operations Research and Financial Engineering & Program in Applied and Computational Mathematics, Princeton NJ 08544, USA · Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning; NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai; NYU Shanghai, 567 West Yangsi Road, Shanghai, 200126, People’s Republic of China

Jul 13, 2026, math.OCJ/K move · Enter open · S save

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

Ziheng Cheng, Xin Guo, Huyên Pham, Yufei Zhang

University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London

Policy Gradient Continuous-Time RL Model-Free RL Deep Deterministic Policy Gradient Actor-Critic Methods Policy Learning

Open paper AskPDF arXiv

This paper develops a model-free reinforcement learning framework for continuous--time extended mean field control problems, where both the dynamics and reward may depend on the joint distribution of states and controls. We adopt deterministic feedback policies, under which the state--action distribution is induced directly as a push--forward of the state law. This avoids optimization over stochastic kernels and bypasses key limitations of existing approaches in extended mean field settings. We first establish a model--free sensitivity formula for parameterized McKean--Vlasov dynamics and use it to derive a deterministic policy gradient formula expressed through an advantage--rate function on the Wasserstein space. We then refine this formula by introducing local value and advantage--rate representations that depend on the state, action, and joint state--action distribution, yielding a policy gradient that includes both action derivatives and measure--derivative terms with respect to the control distribution. These characterizations lead to a martingale--based learning principle and motivate a continuous--time deep deterministic policy gradient algorithm combining particle approximations, measure--dependent neural networks, temporal--difference learning, and exploration in either action or parameter space. Numerical experiments on stochastic Cucker--Smale consensus control and optimal liquidation with trade crowding demonstrate the efficiency, stability, and robustness of the proposed method, including problems with explicit dependence on the control distribution.

Abstract

Explore similar work

Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

Mean Field Reinforcement Learning