Generalizing Score-based generative models for Heavy-tailed Distributions

Tiziano Fassina; Gabriel Cardoso; Sylvan Le Corff; Thomas Romary

stat.MLFeb 28, 2026

Generalizing Score-based generative models for Heavy-tailed Distributions

Authors: Tiziano Fassina, Gabriel Cardoso, Sylvan Le Corff, Thomas Romary

Organizations: STIM, Mines Paris Paris, France · LPSM, Sorbonne Université Paris, France

Abstract

Score-based generative models (SGMs) have achieved remarkable empirical success, motivating their application to a broad range of data distributions. However, extending them to heavy-tailed targets remains a largely open problem. Although dedicated models for heavy-tailed distributions have been proposed, their generative fidelity remains unclear and they lack solid theoretical foundations, leaving important questions open in this regime. In this paper, we address this gap through two theoretical contributions. First, we show that combining early stopping with a suitable initialization is sufficient to extend the diffusion framework to any target distribution; in particular, we establish the well-posedness of the backward process and prove convergence of the approximated diffusion in KL divergence. Second, we derive novel theoretical guarantees for generation with normalizing flows, obtaining convergence results that hold under mild conditions on the flow family and without any assumption on the tail behavior of the target distribution. Building on these results, we propose a unified generative framework for heavy-tailed distributions: a normalizing flow is first trained to capture the tail behavior and is then used as an initialization prior for an SGM, which refines the samples by recovering fine-grained structural details. This design leverages the complementary strengths of the two model classes within a theoretically principled pipeline, overcoming the limitations of existing approaches.

Figures & tables

Panel of Figure 1 — Figure 1: GMM (left) and HT (right) targets.

Panel of Figure 2 — Figure 2: Tail quantiles (0.9–0.9999). $p_{\infty}$ , $p_{T}$ , $p_{\theta}$ , $\mu_{\text{data}}$ . NUTS denoiser.

Target	Method	Init	$\sigma_{T}{=}0.801$	$\sigma_{T}{=}1.055$	$\sigma_{T}{=}1.244$
GMM	Analytic	$\pi_{\infty}$	$0.169\,{\pm}\,0.013$	$0.105\,{\pm}\,0.007$	$0.081\,{\pm}\,0.005$
		$p_{0}^{\theta}$	$0.027\,\pm\,0.004$	$0.031\,\pm\,0.003$	$0.034\,\pm\,0.003$
		$\vec{p}_{T}$	$0.024\,{\pm}\,0.004$	$0.026\,{\pm}\,0.003$	$0.030\,{\pm}\,0.003$
	Flow	–	$0.067\,{\pm}\,0.005$
HT	NUTS	$\pi_{\infty}$	$0.074\,{\pm}\,0.001$	$0.037\,{\pm}\,0.006$	$0.025\,{\pm}\,0.003$
		$p_{\theta}$	$0.014\,{\pm}\,0.002$	$0.014\,{\pm}\,0.002$	$0.012\,{\pm}\,0.004$

Table 1: MSW (mean

\pm

std). Global for GMM, bulk for HT. Flow baseline does not depend on

\sigma_{T}

Panel of Figure 3 — Figure 3: $\operatorname{D}_{\operatorname{KL}}\left(\vec{p}_{T}||p_{0}^{\widehat{\theta}^{n}}\right)$ for different $n$ and $\sigma_{T}$ .

Panel of Figure 4 — Figure 4: Tail quantiles (0.9–0.9999). $p_{\infty}$ , $p_{T}$ , $p_{\theta}$ , $\mu_{\text{data}}$ . Trained $\text{NN}_{\theta}$ denoiser, $10^{4}$ samples.

Figure 5: Denoising quality for the NUTS sampler at $\sigma_{t}=1.0$ . The heatmap reports the $\text{diff}_{\text{MCMC}}$ metric, which compares posterior means across configurations (rows: input norms $\|x\|$ ; columns: $(n_{\text{mcmc}},n_{\text{warmup}})$ pairs). Warmer colors (yellow/red) indicate larger deviations from the oracle, while cooler colors (blue) indicate smaller deviations. The selected configuration $(1000,\,200)$ attains accuracy comparable to the oracle at a markedly lower computational cost.

Learning rate	Batch size	Dataset size	Blocks	Channel Multipliers	Normalization	Activation	Positional Embedding
$10^{-4}$	20000	$10^{3}-10^{4}-10^{5}$	4	[2,4,4,2]	GroupNorm	SiLU	FNet-style

Table 2: Configuration of the MLP denoiser used for the HT case. The model uses four fully connected blocks with GroupNorm and SiLU activations. The noise schedule is continuous with log-normal sampling. FNet refers to the denoiser of Karras et al. [2022] and we used Adam optimizer Kingma and Ba [2014] .

Dimension	Flow Layers	NN Width	NN Depth	Activation	Learning Rate	Batch Size	N. Epochs
2	5	50	3	ReLU6	$10^{-3}$	1000	3000

Table 3: Configuration of the normalizing flow models used for heavy-tail and GMM cases. Both models use coupling flows with triangular affine transformations. Training employs Adam optimizer with dynamic or fixed noise strategies. The flow operates in a noise-conditioned setting where samples are corrupted with Gaussian noise during training.

	Init	$\sigma=0.8012415$	$\sigma=1.05527906$	$\sigma=1.24404052$
NUTS	$\pi_{\infty}$	0.074 $\pm$ 0.001	0.037 $\pm$ 0.006	0.025 $\pm$ 0.003
	$p_{\theta}$	0.014 $\pm$ 0.002	0.014 $\pm$ 0.002	0.012 $\pm$ 0.004
	$\vec{p}_{T}$	0.012 $\pm$ 0.002	0.012 $\pm$ 0.003	0.012 $\pm$ 0.003
$\text{NN}_{\theta}$	$\pi_{\infty}$	0.074 $\pm$ 0.001	0.035 $\pm$ 0.009	0.027 $\pm$ 0.001
	$p_{\theta}$	0.012 $\pm$ 0.001	0.011 $\pm$ 0.002	0.012 $\pm$ 0.001
	$\vec{p}_{T}$	0.011 $\pm$ 0.002	0.011 $\pm$ 0.002	0.011 $\pm$ 0.002

Table 4: Bulk MSW (mean

\pm

std) for all configurations.

Panel of Figure 6 — Figure 6: Tail quantiles (0.9–0.9999). $p_{\infty}$ , $p_{T}$ , $p_{\theta}$ , Reference . We test different MCMC-based denoisers

Panel of Figure 7 — Figure 7: Tail quantiles (0.9–0.9999). $p_{\infty}$ , $p_{T}$ , $p_{\theta}$ , Reference . We test Neural Network denoisers trained on different size of training datasets.

Figure 8: KL divergence between $p_{0}^{\theta}$ and $\vec{p}_{T}$ for the flow trained under the fixed-noise regime of Algorithm 1 . Rows index the training set size $n_{\text{train}}$ , columns the noise level $\sigma_{T}$ . The cubic root of the KL divergence is shown for readability; warmer colors correspond to larger divergences. The fixed-noise strategy yields low KL only when the dataset is sufficiently large.

Figure 9: KL divergence between $p_{0}^{\theta}$ and $\vec{p}_{T}$ for the flow trained under the dynamic-noise regime of Algorithm 1 . Rows index the training set size $n_{\text{train}}$ , columns the noise level $\sigma_{T}$ . The cubic root of the KL divergence is shown for readability; warmer colors correspond to larger divergences. Compared to the fixed-noise regime ( Figure 8 ), the dynamic strategy produces an improved performance the bigger $\sigma_{T}$ it is, confirming the benefit of training the flow on noised data.

Model	FID	DINO FD	KID	SWD	MSW
$\pi_{\infty}(\sigma_{T}=80)$ (standard)	5.90	67.13	0.0018	0.042 $\pm$ 0.001	5.095 $\pm$ 0.129
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=0.5$ (fixed)	3.38	58.72	0.0009	0.016 $\pm$ 0.000	1.955 $\pm$ 0.112
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=1.0$ (fixed)	3.30	63.26	0.0009	0.023 $\pm$ 0.000	1.984 $\pm$ 0.041
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=1.5$ (fixed)	3.11	69.58	0.0008	0.035 $\pm$ 0.000	4.781 $\pm$ 0.124
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=2.0$ (fixed)	2.93	79.57	0.0006	0.048 $\pm$ 0.001	6.992 $\pm$ 0.145
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=0.5$ (dynamical)	4.35	59.56	0.0013	0.027 $\pm$ 0.000	2.999 $\pm$ 0.062

Table 5: Aggregated Results for ImageNet

{}_{\text{birds}}

Panel of Figure 10 — Figure 10: Nearest-neighbour grid comparison for ImageNet ${}_{\text{birds}}$ . Each row corresponds to a different initialization strategy; columns correspond to the queried training classes.

Model	Channel size	# blocks	Layers per block	Epochs	Batch size	Learning rate	Training Factor	Guidance
Original Model	$512$	$8$	$8$	$100$	$128$ (default)	$10^{-4}$ (default)	,	$0$ uncond / $1$ cond
$p_{0}^{\theta}$ , FFHQ	$128$	$1$	$8$	$700$	$500$	$3\cdot 10^{-4}$	$14$	$0$ (uncond)
$p_{0}^{\theta}$ , ImageNet ${}_{\text{birds}}$	$128$	$1$	$8$	$700$	$1000$	$3\cdot 10^{-4}$	$14$	$0.5$ – $2$ (cond)
$p_{0}^{\theta}$ , ImageNet ${}_{\text{dogs}}$	$128$	$1$	$8$	$700$	$1000$	$3\cdot 10^{-4}$	$14$	$0.5$ – $2$ (cond)

Table 6: TarFlow Zhai et al. [2025] configurations: original model versus the lightweight variant we use as

p_{0}^{\theta}

Model	FID	DINO FD	KID	SWD	MSW
$\pi_{\infty}(\sigma_{T}=80)$ (standard)	2.53	198.34	0.0013	0.041 $\pm$ 0.003	5.380 $\pm$ 0.282
$p_{0}^{\theta}$ (fixed)	3.00	210.86	0.0019	0.036 $\pm$ 0.002	3.295 $\pm$ 0.331
$p_{0}^{\theta}$ (dynamical)	3.54	221.39	0.0024	0.062 $\pm$ 0.002	6.646 $\pm$ 0.274
$\vec{p}_{T}$	2.47	188.70	0.0014	0.011 $\pm$ 0.001	0.085 $\pm$ 0.289
$\pi_{\infty}(\sigma_{T}=7)$	16.80	322.57	0.0152	0.255 $\pm$ 0.001	25.138 $\pm$ 0.134

Table 7: Aggregated Results for FFHQ

Panel of Figure 11 — Figure 11: Nearest-neighbour grid comparison for FFHQ-64. Each row corresponds to a different initialization strategy.

Model	FID	DINO FD	KID	SWD	MSW
$\pi_{\infty}(\sigma_{T}=80)$ (standard)	10.46	95.30	0.0050	0.040 $\pm$ 0.001	3.739 $\pm$ 0.088
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=0.5$ (fixed)	5.53	70.35	0.0022	0.029 $\pm$ 0.001	2.911 $\pm$ 0.073
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=1.0$ (fixed)	5.25	69.59	0.0021	0.034 $\pm$ 0.001	3.034 $\pm$ 0.071
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=1.5$ (fixed)	4.76	70.36	0.0018	0.041 $\pm$ 0.001	4.359 $\pm$ 0.171
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=2.0$ (fixed)	4.25	71.51	0.0014	0.050 $\pm$ 0.001	6.146 $\pm$ 0.175
$p_{0}^{\theta}$ - CFG ${}_{\text{flow}}=0.5$ (dynamical)	6.66	73.08	0.0029	0.033 $\pm$ 0.001	2.810 $\pm$ 0.068

Table 8: Aggregated Results for ImageNet

{}_{\text{dogs}}

Panel of Figure 12 — Figure 12: Nearest-neighbour grid comparison for ImageNet ${}_{\text{dogs}}$ . Each row corresponds to a different initialization strategy; columns correspond to the queried training classes.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 493 papers

May 13, 2026

Do Heavy Tails Help Diffusion? On the Subtle Trade-off Between Initialization and Training

Recent works have proposed incorporating heavy-tailed (HT) noise into diffusion- and flow-based generative models, with the goals of better recovering the tails of target distributions and improving generative diversity. This motivation is intuitive: if the data are heavy-tailed, HT noise may appear better matched than light-tailed (LT) Gaussian noise. However, replacing Gaussian noise by HT noise also changes the underlying estimation problem. In this paper, we revisit this paradigm through a combined theoretical and empirical study, establishing sampling-error bounds for two representative diffusion models driven by HT and LT noise. We show that HT noise makes the statistical estimation problem harder, leading to less favorable sampling-error bounds. We support these findings with experiments on synthetic and real-world datasets, empirically recovering the predicted error trade-off. Our results call into question a growing design trend in generative modeling and challenge the use of HT noise to improve rare-region exploration.

Hamza Cherkaoui, Hélène Halconruy, Antonio Ocello

SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France · Modal’X, Université Paris Nanterre, Nanterre, France · CREST, ENSAE Paris, Institut Polytechnique de Paris, Palaiseau, France

Diffusion Model Sampling Heavy-Tailed Noise Heavy-Tailed Distributions Diffusion Models

Ask PDF

May 19, 2026

Tail Annealing for Heavy-Tailed Flow Matching

Standard generative models struggle with heavy-tailed data: Lipschitz architectures cannot produce power-law tails from Gaussian noise, and interpolating between heavy-tailed data and Gaussians is ill-posed. We propose a simple fix: apply the soft-log transform

φ(x) = \mathrm{sign}(x) \cdot \log(1 + |x|)

coordinate-wise to data before training, then exponentiate samples after generation. A Hill diagnostic decides per-coordinate whether to transform, leaving light-tailed margins untouched at no added complexity. This compresses heavy tails into a range where standard flow matching succeeds, without heavy-tailed base distributions or architectural modifications. We provide theoretical intuition for why this works: the log-transform maps Pareto tails to exponentials, and the induced dynamics implement a form of tail annealing via power transformations. On a 144-configuration multivariate benchmark (3 copulas,

d

up to 100, 4 tail indices), Log-FM dominates specialized baselines on

W_1

, CVaR

_{99}

, and extreme-quantile metrics, and is the only method with zero severe divergences across 2{,}880 runs.

Jean Pachebat

1CMAP, Ecole Polytechnique, Institut Polytechnique de Paris, France.

Flow Matching Generative Modeling Flow-Based Generative Modeling Heavy-Tailed Distributions

Ask PDF

Oct 14, 2026

From Global to Tail Convergence: Dynamics of $α$ -Stable Score-Based Generative Models

Score-based generative models driven by

α

-stable L{é}vy processes have been proposed as an alternative to Brownian diffusion models, with empirical evidence suggesting improved sampling and better modeling of heavy-tailed data. Yet theoretical guarantees for these models remain limited. We develop a non-asymptotic convergence analysis for scale-preserving

α

-stable generative dynamics, covering both compactly supported and power-law targets. In p-Wasserstein distance, we quantify the effects of early stopping, initialization, score approximation, and time discretization, and show exponential decay of the initialization error. A comparison with Brownian dynamics reveals that global Wasserstein convergence alone does not fundamentally distinguish the two models: both initialization errors decay exponentially with the forward horizon on the common class of heavy-tailed targets. A sharp separation emerges instead in the tails. At any positive generation time, stable dynamics produces power-law tails with index

α

, whereas Brownian dynamics remains light-tailed. For a target with tail index

ν

, asymptotically correct extreme-quantile scaling is possible precisely when

α

ν

. These results identify tail-index matching as a principled mechanism behind the ability of stable generative models to capture extreme events.

Ianis Hammani, Claire Boyer

ENS Paris Saclay, LMO · LMO, IUF

Score-Based Generative Modeling Diffusion Models Heavy-Tailed Distributions Stochastic Differential Equations

Ask PDF

Abstract

Generative Modeling

Explore similar work

Do Heavy Tails Help Diffusion? On the Subtle Trade-off Between Initialization and Training

Tail Annealing for Heavy-Tailed Flow Matching

From Global to Tail Convergence: Dynamics of ααα-Stable Score-Based Generative Models

From Global to Tail Convergence: Dynamics of $α$ -Stable Score-Based Generative Models