Organizations: Departments of Mathematics and Computer Science, Emory University, Atlanta, GA, USA · Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer-Sheva, Israel
Learning the underlying probability distribution from samples is a core problem in statistics and applied mathematics. Generative models based on diffusion and flow matching learn a velocity field whose flow transports a known distribution to the data distribution. They produce novel, realistic samples, although the empirical learning problem is ill posed near the data distribution. Its minimizer transports the samples onto the training data, and small perturbations of the data change the velocity field strongly near the end of the flow. The observed generalization is often attributed to implicit regularization by the network parameterization, early stopping of the training, and the time integration during sampling. Explicit regularization has advanced the theory and computation of many inverse problems and has recently been proposed to prevent memorization in generative models. With a regularizer on the magnitude of the velocity, however, the field approaches the memorizing velocity unless the regularization parameter grows to infinity fast enough, and it then stays bounded by a constant that is small only for large parameters. We show that penalizing the derivatives of order k of the velocity renders the D-dimensional empirical problem well posed uniformly in time if 2k>D and the training points are in general position. Our experiments in one and two dimensions discretize the velocity with splines and use direct solvers for the regularized least-squares problems to exclude optimization error. Refining the discretization in space and time changes the error of the Hessian-regularized flows only marginally, whereas without the regularizer the error tends to that of memorization. These flows generalize, with errors comparable to or lower than those of well-tuned kernel density estimates.
Figures & tables
Figure 1: Flows of (a) the empirical velocity vN and (b) the regularized velocity with the Hessian regularizer and hold-out λ(t) for the trimodal target with N=100 . Images show the velocity, scaled by 1−t and plotted on a time axis that stretches late times, with trajectories; the side panels show the generated density, the true density (dashed) and the training data; the strips show the velocity error, the mean squared distance to the population velocity, and, in (b), the selected λ(t) relative to the scale λs of section 4.1 . The narrow panels between the columns magnify t≥0.9 thirtyfold in x near a group of training points: the trajectories of vN end on the training points, those of the regularized flow do not. Panel titles give the total variation error. Training set with the median hold-out error.
Estimator
N=100
N=1000
N=3000
( 10−2 )
( 10−2 )
( 10−2 )
Trimodal (1D)
FM, oracle λ
10.0±0.8
4.1±0.3
2.5±0.2
FM, hold-out
13.4±1.7
4.7±0.3
2.9±0.4
FM, hold-out constant λ
13.0±1.3
4.9±0.2
3.2±0.3
KDE, Sheather–Jones
11.8±1.1
4.3±0.4
2.7±0.2
Inhom. (1D)
FM, oracle λ
14.6±0.7
5.9±0.3
4.0±0.1
Table 1: Total variation error for all targets, mean ± standard error over five training sets, for the flows and for the data-driven KDE with the lowest mean error. Bold: the lowest data-driven mean and the data-driven entries not resolved from it (difference below twice the combined Monte Carlo or set-to-set standard error), per target and N . Tables 2 and 3 list all rules, including the oracle KDEs. FM: flow matching with the Hessian regularizer ( k=2 ); hold-out constant: one λ selected by hold-out for all times ( section 5.7 ). KDE bandwidth rules: Sheather–Jones plug-in [ 33 ] , least-squares cross-validation (LSCV) [ 31 , 8 ] and Abramson’s adaptive rule [ 1 ] ; the fourth-order and product kernels are defined in appendix B .
Figure 2: Total variation error against the spline resolution m for N=1000 , for the limit λ→0 , computed at the floor λ(t)=10−14λs of section 4.1 , and for the hold-out flows of the three regularizers, mean ± standard error over five training sets, for the trimodal target, the noisy circle and the two spirals; tables 5 and 6 list all targets. The limit λ→0 and the flow with the smallness regularizer move toward memorization as m grows, whereas the error of the Hessian flow changes by at most 0.007 .
Figure 3: Distance between the laws generated from the training set and from the training set with every point moved by ε , for (a) the trimodal target with N=100 and (b) the noisy circle with N=1000 , and (c) the corresponding difference of the fields in H2(Ω) , mean over five training sets and two perturbation directions, with ranges, for the Hessian ( k=2 ) and the smallness ( k=0 ) regularizer with λ(t) fixed to the unperturbed selection or re-selected by hold-out.
Figure 4: Densities generated on the two-dimensional targets (rows) with N=1000 by the flow with hold-out λ(t) for the smallness ( k=0 ), gradient ( k=1 ) and Hessian ( k=2 ) regularizers, by the best data-driven KDE, and the true density (columns). Each flow uses the training set with its median error; the KDE uses the median set of the Hessian regularizer. Color scales are shared within a row, with values above 1.1 times the maximum of the true density saturated. Panel tags give the total variation error.
Department of Electrical and Computer Engineering, Ohio State Univeristy · Department of Computer Science and Information Systems, BITS Pilani · Department of Electrical and Computer Engineering, Tufts University +2