Generative Drifting is Secretly Score Matching: a Spectral and Variational Perspective

Erkan Turan; Nicolas Dufour; Maks Ovsjanikov

cs.LGMar 10, 2026

Generative Drifting is Secretly Score Matching: a Spectral and Variational Perspective

Authors: Erkan Turan, Nicolas Dufour, Maks Ovsjanikov

Organizations: LIX, Ecole Polytechnique, IP Paris · Kyutai

Abstract

Generative Modeling via Drifting~\citep{deng2026drifting} has recently achieved state-of-the-art one-step image generation through a kernel-based drift operator, yet its success is largely empirical and its theoretical foundations remain poorly understood. We observe that under a Gaussian kernel, the drift operator is exactly a score difference on smoothed distributions. This answers three questions left open in the original work: (1) whether a vanishing drift guarantees equality of distributions ( $V_{p,q}=0\Rightarrow p=q$ ), (2) how to choose between kernels, and (3) why the stop-gradient operator is indispensable for stable training. Our observations position drifting within the score-matching family. By linearizing the McKean-Vlasov dynamics and probing them in Fourier space, we reveal frequency-dependent convergence timescales comparable to Landau damping in plasma kinetic theory: the Gaussian kernel suffers an exponential high-frequency bottleneck, potentially explaining the empirical preference for the Laplacian kernel. This suggests a fix: an exponential bandwidth annealing schedule $σ(t)=σ_0 e^{-rt}$ that reduces convergence time from $\exp(O(K_{\max}^2))$ to $O(\log K_{\max})$ . Finally, by formalizing drifting as a Wasserstein gradient flow of the smoothed KL divergence, we prove that the stop-gradient operator is not a heuristic but is derived from the frozen-field discretization mandated by the Jordan-Kinderlehrer-Otto (JKO) scheme, and removing it severs training from any gradient-flow guarantee. This variational perspective further provides a general template for constructing novel drift operators, which we demonstrate with a Sinkhorn divergence drift. We validate our analysis on toy datasets and scale it up to ImageNet.

Figures & tables

Figure 1: Numerical confirmation of Theorem 4.1 on a 4-mode Gaussian mixture. (a) Empirical kernel mean-shift drift ( $N=50$ k samples). (b) Analytical score-difference form $\sigma^{2}\nabla\log(p_{\sigma}/q_{\sigma})$ . (c) Overlay: the two fields are visually indistinguishable. (d) Pointwise $\ell_{2}$ error heatmap (mean $4.9\times 10^{-3}$ ). Details are provided in Section F.1 .

Figure 2: Spectral validation. (a) Convergence time

T(k)

vs. frequency: lines are analytical predictions from Theorem 5.2 , markers are measured decay times. The fixed-bandwidth Gaussian exhibits exponential slowdown (Landau damping); the Laplacian kernel yields polynomial scaling; the annealed Gaussian eliminates the bottleneck entirely. (b) Annealing schedules

\sigma(t)

. (c) Total spectral error under different schedules. Details in F.2 .

Figure 3: Loss landscapes with and without stop-gradient, projected onto the top two principal gradient-variation directions. (a,b) Training loss $\|V\|^{2}$ : without SG the minimum is ${\sim}100\times$ deeper. (c,d) Sliced Wasserstein distance: with SG the loss minimum coincides with low distributional error; without SG the deep minimum corresponds to poor sample quality. Red star: trained solution. Details in F.3 .

Figure 4: Drift norm vs. distributional distance during training on synthetic 2D targets. (a) Mean drift norm $\|V\|$ . (b) Sliced Wasserstein distance. (c) Log–log scatter across training steps and seeds. With stop-gradient (solid), the two quantities are strongly correlated ( $r>0.95$ ) and jointly decay. Without stop-gradient (dashed), the drift norm collapses to ${\sim}10^{-8}$ while the Wasserstein distance remains at $0.389$ —a direct demonstration of drift collapse (Theorem 5.6 (ii) ). Details in F.3 .

Figure 5: Sinkhorn-derived drift (bottom row) vs. Laplacian-kernel drift (top row) on the checkerboard distribution. Training snapshots with drift vectors overlaid; rightmost panels show sliced Wasserstein distance over training. Both converge successfully (final SW $1.42\times 10^{-2}$ and $2.07\times 10^{-2}$ respectively), demonstrating that the gradient-flow template of § 6.3 yields practical operators beyond the original kernel family. Details are provided in F.4 .

Drift operator	Bandwidth	NFE	FID $\downarrow$	IS $\uparrow$
Laplacian ( Deng et al., 2026 )	$\tau$ fixed	1	8.55	148.0
Gaussian (ours)	$\sigma$ fixed	1	8.69	157.0
Gaussian (ours)	$\sigma(t)=\sigma_{0}e^{-rt}$	1	8.36	154.2
Sinkhorn (ours)	$\varepsilon$ fixed	1	8.81	135.48

Table 1: Ablation on ImageNet

256{\times}256

class-conditional generation. All variants share the same B/2 architecture, latent-MAE SSL representation, and training schedule of Deng et al. (2026) ; only the drift operator differs. We report FID (

\downarrow

) and IS (

\uparrow

) under one-step (NFE

=

1) generation. Best in bold , second-best underlined .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Optimization
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.95$ , $\epsilon{=}10^{-8}$ )
Weight decay	$0.01$
Peak learning rate	$2\times 10^{-4}$
LR schedule	linear warmup ( $5{,}000$ steps), then constant
Gradient clipping	$\\|g\\|_{2}\leq 2.0$
Total steps	$30{,}000$

Appendix

Table 2: Training hyperparameters for the image-generation ablations ( latent_ablation configuration).

Panel of Figure 6 — Figure 6: Uncurated samples from the exponential baseline ( Deng et al., 2026 ) at $30{,}000$ steps, CFG $w{=}2.2$ . Each cell is an independently generated $256{\times}256$ image; samples from $10$ ImageNet classes (macaw, hummingbird, flamingo, golden retriever, tiger, monarch butterfly, African elephant, sports car, cheeseburger, volcano) are shuffled across the grid.

Panel of Figure 7 — Figure 7: Uncurated samples from the Gaussian-drift model at $30{,}000$ steps, CFG $w{=}1.833$ . Each cell is an independently generated $256{\times}256$ image; samples from $10$ ImageNet classes (macaw, hummingbird, flamingo, golden retriever, tiger, monarch butterfly, African elephant, sports car, cheeseburger, volcano) are shuffled across the grid.

Panel of Figure 8 — Figure 8: Uncurated samples from the Gaussian-drift model with annealing at $30{,}000$ steps, CFG $w{=}1.944$ . Each cell is an independently generated $256{\times}256$ image; samples from $10$ ImageNet classes (macaw, hummingbird, flamingo, golden retriever, tiger, monarch butterfly, African elephant, sports car, cheeseburger, volcano) are shuffled across the grid.

Panel of Figure 9 — Figure 9: Uncurated samples from the Sinkhorn-drift model at $30{,}000$ steps, CFG $w{=}1.167$ . Each cell is an independently generated $256{\times}256$ image; samples from $10$ ImageNet classes (macaw, hummingbird, flamingo, golden retriever, tiger, monarch butterfly, African elephant, sports car, cheeseburger, volcano) are shuffled across the grid.

Shape		Wall-clock per step (ms)
$B$	$d$	softmax	Gaussian	annealed (3R)	Sinkhorn
$64$	$256$	$\mathbf{0.761{\scriptstyle\,\pm\,0.020}}$	$0.775{\scriptstyle\,\pm\,0.012}$	$0.780{\scriptstyle\,\pm\,0.014}$	$1.004{\scriptstyle\,\pm\,0.105}$
$256$	$256$	$1.214{\scriptstyle\,\pm\,0.004}$	$1.265{\scriptstyle\,\pm\,0.005}$	$1.261{\scriptstyle\,\pm\,0.004}$	$\mathbf{1.042{\scriptstyle\,\pm\,0.011}}$
$1024$	$256$	$4.211{\scriptstyle\,\pm\,0.005}$	$4.343{\scriptstyle\,\pm\,0.005}$	$4.341{\scriptstyle\,\pm\,0.005}$	$\mathbf{2.552{\scriptstyle\,\pm\,0.011}}$
$64$	$768$	$\mathbf{0.899{\scriptstyle\,\pm\,0.005}}$	$0.930{\scriptstyle\,\pm\,0.005}$	$0.935{\scriptstyle\,\pm\,0.006}$	$1.040{\scriptstyle\,\pm\,0.088}$

Appendix

Table 3: Per-step drift wall-clock on a single H100, in milliseconds (mean

\pm

std over

100

iterations after

10

warm-ups). The “softmax” column is the Laplacian-kernel softmax baseline; “annealed (3R)” is the Gaussian drift with the

3R

exponential bandwidth schedule of Section 6.2 . Bold marks the fastest variant in each row.

Drifting Models

Drifting models learn transport fields that move generated samples toward a target distribution, often enabling one-step generation.

See all 53 papers

Mar 12, 2026

Sinkhorn-Drifting Generative Models

We establish a theoretical link between the recently proposed "drifting" generative dynamics and gradient flows induced by the Sinkhorn divergence. In a particle discretization, the drift field admits a cross-minus-self decomposition: an attractive term toward the target distribution and a repulsive/self-correction term toward the current model, both expressed via one-sided normalized Gibbs kernels. We show that Sinkhorn divergence yields an analogous cross-minus-self structure, but with each term defined by entropic optimal-transport couplings obtained through two-sided Sinkhorn scaling (i.e., enforcing both marginals). This provides a precise sense in which drifting acts as a surrogate for a Sinkhorn-divergence gradient flow, interpolating between one-sided normalization and full two-sided Sinkhorn scaling. Crucially, this connection resolves an identifiability gap in prior drifting formulations: leveraging the definiteness of the Sinkhorn divergence, we show that zero drift (equilibrium of the dynamics) implies that the model and target measures match. Experiments show that Sinkhorn drifting reduces sensitivity to kernel temperature and improves one-step generative quality, trading off additional training time for a more stable optimization, without altering the inference procedure used by drift methods. These theoretical gains translate to strong low-temperature improvements in practice: on FFHQ-ALAE at the lowest temperature setting we evaluate, Sinkhorn drifting reduces mean FID from 187.7 to 37.1 and mean latent EMD from 453.3 to 144.4, while on MNIST it preserves full class coverage across the temperature sweep. Project page: https://mint-vu.github.io/SinkhornDrifting/

Ping He, Om Khangaonkar, Hamed Pirsiavash +2

Department of Computer Science, Vanderbilt University · Department of Computer Science, University of California, Davis · Department of Computer Science, Purdue University

Sinkhorn Divergence Drifting Models Flow-Based Generative Modeling Optimal Transport One-Step Generative Modeling

Ask PDF

Mar 8, 2026

A Unified View of Score-Based and Drifting Models

Drifting models train one-step generators by optimizing a kernel-induced mean-shift discrepancy between the data and model distributions, with Laplace kernels used by default in practice. At each point, this discrepancy compares the kernel-weighted displacement toward nearby data samples with the corresponding displacement toward nearby model samples, thereby defining a transport direction for generated samples. In this paper, we show that drifting is more closely connected to score-based generative modeling than it may first appear, establishing a precise link to the score-matching principle underlying diffusion models. For Gaussian kernels, the population mean-shift field exactly equals the difference between the scores (i.e., the gradient-log-densities) of the Gaussian-smoothed data and model distributions. This identity follows from Tweedie's formula, which links the score of a Gaussian-smoothed density to its conditional mean, and implies that Gaussian-kernel drifting is exactly a score-matching objective on smoothed distributions. More generally, we derive an exact decomposition for radial kernels in which mean shift equals a score-based field plus a residual term. For the practical Laplace kernel, we further show theoretically and empirically that this residual is negligible in high dimension, implying that the transport field used in practice is nearly score-based. Our results reveal a structural connection to diffusion models: both methods use score-mismatch transport directions, but drifting realizes the score nonparametrically through kernel-based estimates, whereas diffusion models learn it parametrically with neural networks.

Chieh-Hsin Lai, Bac Nguyen, Naoki Murata +5

Sony AI · Sony Group Corporation · Stanford University +1

Generative Modeling Drifting Models Score-Based Generative Modeling Score Matching Diffusion Models

Ask PDF

May 11, 2026

Kernel-Gradient Drifting Models

We propose kernel-gradient drifting, a one-step generative modeling framework that replaces the fixed Euclidean displacement direction in drifting models with directions induced by the kernel itself. Standard drifting is attractive because it enables fast, high-quality generation without distilling a large pretrained diffusion model, but its theory is currently understood mainly for Gaussian kernels, where the drift coincides with smoothed score matching and is identifiable. Our gradient-based reformulation exposes this score-based structure for general kernels: the resulting drift is the score difference between kernel-smoothed data and model distributions, yielding identifiability for characteristic kernels and a smoothed-KL descent interpretation of the drifting dynamics. Since kernel gradients are intrinsic tangent vectors, the same construction extends naturally to Riemannian manifolds and to discrete data via the Fisher-Rao geometry of the probability simplex. Across spherical geospatial data, promoter DNA and molecule generation, kernel-gradient drifting enables state-of-the-art one-step generation beyond the Euclidean setting without distillation.

Maria Esteban-Casadevall, Jorge Carrasco-Pollo, Max Welling +3

AMLab · University of Amsterdam · CuspAI +1

Synthetic Data Generation One-Step Generative Modeling Generative Modeling Drifting Models

Ask PDF

Abstract

Drifting Models

Explore similar work

Sinkhorn-Drifting Generative Models

A Unified View of Score-Based and Drifting Models

Kernel-Gradient Drifting Models