Intrinsic Gradient Barriers in Pushforward Deep Generative Models
Authors: Edric Tam, Qingyang Liu, Barbara Engelhardt, Ray Bai
Organizations: Department of Biomedical Data Science, Stanford University · Department of Statistical Sciences, Wake Forest University · Department of Statistics, George Mason University
Broad classes of deep generative models (DGMs) operate by learning to map a simple source distribution, such as a standard Gaussian, to a complex high-dimensional target distribution. Despite their success, DGMs often empirically struggle to accurately represent distributions with heavy tails or well-separated modes. In this work, we show that these difficulties reflect intrinsic properties of the underlying transport problem rather than particular neural network architectures or training procedures. We establish quantitative lower bounds on the gradients and Jacobians of pushforward mappings that transport log-concave source distributions to such targets. For heavy-tailed target distributions, the required gradient magnitude grows as the target develops increasingly large moments. For multimodal target distributions, the low-probability regions between well-separated modes similarly force large gradients in the transport map. We discuss how such large derivatives relate to trade-offs between expressivity and numerical stability. Our results hold under substantially weaker regularity conditions than previous analyses and remain stable under perturbations of the target distribution. Finally, we show that these obstructions can be mitigated when the goal is approximation of a given density rather than generative modeling. Specifically, we propose simple classes of bijective reparametrizations that can alleviate the difficult geometric structure and reduce the gradient burden placed on the learned generative map. Numerical experiments support our theoretical analysis.
Figures & tables
Fig 1: Results from using normalizing flows with a Gaussian source distribution to approximately sample from two target densities before ( left two panels ) and after ( right two panels ) applying bijective reparametrizations. The targets are a heavy-tailed Student- t distribution (top) and a bimodal Gaussian mixture (bottom). Heatmaps show the densities estimated from the generated samples in the original coordinates, while dashed curves show the target contours.
Fig 2: Comparison of Poincaré lower bounds with the expected squared Frobenius norm of the transport Jacobian for different heavy-tailed target distributions under a standard Gaussian source. The red curves with circular markers show the numerically evaluated quantities E[∥JT(Z)∥F2] . The blue dashed curves show the lower bounds.
Fig 3: Log absolute Jacobian determinant heatmaps for Gaussian-to-heavy-tailed transport maps. Note that the Pareto and Fréchet distributions are right-skewed and supported on the positive orthant.
Fig 4: Gradient bounds for two-component Gaussian mixtures. The plots compare E[T′(Z)] with the lower bound in Theorem 5.1 as the width Δ of the low-probability interval increases. The two panels use component standard deviations σ=0.1 and σ=0.5 .
Fig 5: Expected squared Frobenius Jacobian norms before and after the asinh reparametrization for two-dimensional product Student- t , Pareto, and Fréchet targets, using a standard Gaussian source. The shaded regions indicate degrees of freedom or shape parameters less than or equal to 2 .
Fig 6: Expected squared derivatives E[T′(Z)2] and E[S′(Z)2] , for Z∼N(0,1) , before and after transformation for five-component Gaussian mixtures as adjacent means move farther apart. Component widths, transformation parameters, and errors in the reference component locations are held fixed.
Fig S1: Comparison of Poincaré lower bounds with the expected squared Frobenius norm of the transport Jacobian for a product Laplace source and heavy-tailed target distributions. The red curves with circular markers show the numerically evaluated quantities. The blue dashed curves show the lower bounds, using CPI(ν)=2 .
Fig S2: Log absolute Jacobian determinant heatmaps for log-concave-to-heavy-tailed transport maps. As in the Gaussian-source case (Figure 3 ), the magnitude of the Jacobian grows rapidly away from the center as the target becomes increasingly heavy-tailed. Note that the Pareto and Fréchet distributions are right-skewed and supported on the positive orthant.
Fig S3: Comparison of Poincaré lower bounds with the expected squared Frobenius norm of the transport Jacobian for correlated heavy-tailed targets under a standard Gaussian source. The top row, panels (A)–(C), uses ρ=0.5 , and the bottom row, panels (D)–(F), uses ρ=0.9 . Columns correspond to the Student- t , Pareto, and Fréchet target constructions. Each panel shows the numerically evaluated quantity in red and the lower bound computed from the trace of the target covariance matrix as a blue dashed curve. Compare with Figure 2 , where the target has independent coordinates.
Many modern generative modeling methods, including diffusion models, normalizing flows, and flow matching, estimate transport maps or plans between distributions without explicitly targeting an optimal transport (OT) map. In applications like generative modeling, the transport cost itself is irrelevant, and this makes it natural to target maps which are more tractable from either a statistical or computational standpoint. In this short note, we formalize the task of estimating any valid transport map in a rigorous minimax framework. One consequence of this framing is that it yields sample complexity lower bounds for any method whose learned object is evaluated as a transport map or plan, including flow matching and diffusion-based generative models, in settings where direct analysis would be challenging due to the analytic complexity of the methods and their target maps. We observe that, under standard, though strong, stability assumptions from the OT literature, estimating any valid transport map is statistically as hard as estimating the OT map. We complement these results with some examples showing that when these stability assumptions fail, alternative transport maps can be learned substantially more accurately than the OT map. Our minimax framing provides a rigorous foundation for understanding the statistical limits of modern transport-based generative methods and clarifies when targeting sub-optimal maps can provide real statistical advantages.
Sivaraman Balakrishnan
Department of Statistics and Data Science† · Machine Learning Department‡
Despite the remarkable empirical success of generative models, the available theory on their statistical accuracy in scientific computing remains largely pessimistic. This paper develops a theoretical framework for understanding the regularity of transport maps and the generalization properties of one-step Wasserstein-guided generative models for PDE-induced probability measures. We consider normalized target densities associated with linear elliptic and parabolic equations on bounded domains, as well as diffusion and Fokker--Planck equations on the torus. Under standard structural assumptions, we prove that these target measures satisfy doubling conditions. By combining this fact with regularity theory for optimal transport between doubling measures, we show that the optimal transport map from a uniform source measure to the target measure is Hölder continuous. This regularity yields an approximation-theoretic justification for one-step generative models that learn PDE-induced distributions via a single pushforward map. As a representative instance, we study DeepParticle and derive excess-risk bounds characterizing the discrepancy between the learned map and the population-optimal map. We also establish a robustness estimate under target shift and illustrate the theory with experiments which support the derived rates.
Likun Lin, Zhongjian Wang, Jack Xin +1
Department of Mathematics, The University of Hong Kong, Pokfulam Road, Hong Kong SAR, P.R. China. · Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore 637371. · Department of Mathematics, University of California at Irvine, Irvine, CA 92697, USA. +1
Motivated by recent applications in generative modeling and sampling, we introduce a framework for optimal measure transport where cost captures the notion of neural network complexity. In transport-based generative models, samples from a reference distribution (e.g. Gaussian) are mapped to samples of a target distribution along ordinary or stochastic differential equations. These are implemented as deep residual networks when discretized in time, where each hidden layer approximates the associated instantaneous velocity. Thus, given a pair of target and reference measures, a natural question is to search for the most efficient neural representation that implements this transport. Our starting point is the kinetic formulation of OT, due to Benamou and Brenier. We replace the average kinetic L2 energy by the Barron energy \cite{bach2017breaking, ma2022barron}, a natural norm which measures the complexity of representing a given vector field with a neural hidden layer, and which captures the adaptive properties of feature learning. This defines a metric on the space of probability measures, complementing existing Wasserstein and Stein geometries. In this work we examine the properties of this metric in the context of generative modeling. As a first application, we quantify the suboptimality of diffusion generative modeling in the Barron geometry by establishing super-polynomial score approximation lower bounds for data generated by neural network pushforwards of the Gaussian. We then investigate the benefit of adaptivity as a way to study alternative generative models. In a companion paper \cite{companionpaper} we leverage the Barron transport geometry for sampling applications, extending the scope of Stein variational gradient methods via feature adaptation.
Evan Dogariu, Joan Bruna
Courant Institute School of Mathematics, Computing, and Data Science New York University