Phase-Type Variational Autoencoders for Heavy-Tailed Data

Abdelhakim Ziani; András Horváth; Paolo Ballarini

cs.LGMar 2, 2026

Phase-Type Variational Autoencoders for Heavy-Tailed Data

Authors: Abdelhakim Ziani, András Horváth, Paolo Ballarini

Organizations: Université Paris Saclay, Lab. MICS, CentraleSupélec, Gif-sur-Yvette, France · Università di Torino, Torino, Italy

Abstract

Heavy-tailed distributions are ubiquitous in real-world data, where rare but extreme events dominate risk and variability. However, standard Variational Autoencoders (VAEs) employ simple decoder distributions, such as Gaussian distributions, that fail to capture heavy-tailed behavior, while existing heavy-tail-aware extensions remain restricted to predefined parametric families whose tail behavior is fixed a priori. We propose the Phase-Type Variational Autoencoder (PH-VAE), whose decoder distribution is a latent-conditioned Phase-Type (PH) distribution, defined as the absorption time of a continuous-time Markov chain (CTMC). This formulation composes multiple exponential time scales, yielding a flexible and analytically tractable decoder that adapts its finite-range tail behavior directly from the observed data. Experiments on synthetic and real-world benchmarks demonstrate that PH-VAE accurately approximates diverse heavy-tailed distributions, significantly outperforming Gaussian, Student-t, and extreme-value-based VAE decoders in modeling observed tail behavior and extreme quantiles. In multivariate settings, PH-VAE captures realistic cross-dimensional tail dependence through its shared latent representation. To our knowledge, this is the first work to integrate Phase-Type distributions into deep generative modeling, bridging applied probability and representation learning.

Figures & tables

Figure 1: Architecture of the Phase-Type Variational Autoencoder (PH-VAE). A shared latent variable $z$ conditions an acyclic Phase-Type decoder, enabling heavy-tailed likelihoods with cross-dimensional dependence.

Figure 2: A Phase-Type distribution with three transient states.

Panel of Figure 3 — Figure 3: Synthetic Weibull data: true samples, Gaussian VAE generations, and PH-VAE generations.

	Weibull ( $k=0.8,\lambda=1.0)$		Pareto ( $\alpha=2.4,\ x_{m}=1.0$ )		Lognormal ( $\mu=0,\ \sigma=1.5$ )		Burr ( $c=1.5,\ k=0.8$ )
Model	KS ${}_{\text{tail}}$ ↓	Q99 Error ↓	KS ${}_{\text{tail}}$ ↓	Q99 Error ↓	KS ${}_{\text{tail}}$ ↓	Q99 Error ↓	KS ${}_{\text{tail}}$ ↓	Q99 Error ↓
VAE	$0.190\pm 0.012$	$0.235\pm 0.003$	$0.259\pm 0.005$	$0.336\pm 0.002$	$0.498\pm 0.007$	$0.582\pm 0.002$	$0.433\pm 0.002$	$0.724\pm 0.003$
t-VAE	$0.225\pm 0.090$	$0.878\pm 0.000$	$0.233\pm 0.025$	$0.632\pm 0.006$	$0.650\pm 0.001$	$100.783\pm 1.683$	$0.518\pm 0.002$	$39.845\pm 1.837$
xVAE	$0.187\pm 0.008$	$0.062\pm 0.005$	$0.161\pm 0.002$	$0.131\pm 0.008$	$0.032\pm 0.005$	$0.209\pm 0.020$	$N/A$	$1.000\pm 0.000$
LogNormal-VAE	$0.326\pm 0.023$	$2.088\pm 0.381$	$0.072\pm 0.046$	$0.197\pm 0.079$	$0.026\pm 0.010$	$0.082\pm 0.051$	$0.143\pm 0.010$	$0.379\pm 0.056$
Gamma-VAE	$0.056\pm 0.004$	$0.082\pm 0.012$	$0.236\pm 0.100$	$0.342\pm 0.030$	$0.136\pm 0.048$	$0.306\pm 0.050$	$0.083\pm 0.015$	$\mathbf{0.166\pm 0.087}$

Table 1: Quantitative evaluation of tail reconstruction on synthetic datasets.

\mathit{KS}_{tail}

is computed on the conditional CDF above the 95th percentile, and Q99 is the relative deviation of the 99th percentile. Across all four distributions, PH-VAE achieves the strongest tail approximation on nearly all datasets and metrics, highlighting the flexibility of the PH decoder relative to fixed-family decoders.

Figure 4: Log-log CCDF of Danish Fire Insurance losses for real data, Gaussian VAE, and PH-VAE.

		Danish Fire		Word Frequency
Metric	$q$	PH-VAE $\downarrow$	VAE $\downarrow$	PH-VAE $\downarrow$	VAE $\downarrow$
QErr	0.950	$0.130\pm 0.052$	$0.423\pm 0.009$	$0.065\pm 0.029$	$0.435\pm 0.016$
	0.990	$0.456\pm 0.082$	$0.654\pm 0.018$	$0.096\pm 0.047$	$0.515\pm 0.036$
	0.995	$0.334\pm 0.050$	$0.702\pm 0.015$	$0.070\pm 0.056$	$0.558\pm 0.040$
CVaRErr	0.950	$0.263\pm 0.087$	$0.677\pm 0.007$	$0.219\pm 0.039$	$0.636\pm 0.034$
	0.990	$0.200\pm 0.137$	$0.809\pm 0.005$	$0.332\pm 0.043$	$0.720\pm 0.043$

Table 2: Relative tail-risk errors on real-world heavy-tailed datasets. Lower is better. PH-VAE consistently reduces both extreme quantile and CVaR errors compared with the Gaussian VAE baseline.

Variant	$m$	$\beta$	KS ${}_{\text{tail}}\downarrow$	Q99 error $\downarrow$
Default (PH-VAE)	10	1.0	$0.084\pm 0.006$	$\mathbf{0.030\pm 0.036}$
PH-VAE (small)	5	1.0	$0.093\pm 0.027$	$0.217\pm 0.177$
PH-VAE (medium)	15	1.0	$\mathbf{0.055\pm 0.008}$	$0.149\pm 0.254$
PH-VAE ( $\beta=0.5$ )	10	0.5	$0.110\pm 0.008$	$0.859\pm 0.642$
PH-VAE ( $\beta=2.0$ )	10	2.0	$0.083\pm 0.008$	$0.078\pm 0.067$

Table 3: Ablation study of PH-VAE on Pareto data. We vary the number of phases

m

and the KL weight

\beta

. The default configuration used in all main experiments is

m=10,\ \beta=1

Figure 5: Log–log CCDF of word-frequency counts from the Google Web Trillion Word Corpus, comparing real data, Gaussian VAE, and PH-VAE.

Figure 6: Illustration of correlation matrices of synthetic data, comparing ground truth (left), PH-VAE (middle), and Independent PH-VAEs (right).

Model	Corr. error $\downarrow$	Kendall $\tau$ error $\downarrow$	Tail CoExErr@99 $\downarrow$
Gaussian VAE	0.5798	0.1429	$1.38{\times}10^{-3}$
Independent PH-VAEs	1.8824	0.2679	$2.04{\times}10^{-3}$
PH-VAE	$\mathbf{0.5659}$	$\mathbf{0.0672}$	$\mathbf{5.44{\times}10^{-4}}$

Table 4: Synthetic multivariate (5D) data dependence metrics.

Model	Corr. error $\downarrow$	Kendall $\tau$ error $\downarrow$	Tail CoExErr@99 $\downarrow$
Gaussian VAE	0.8848	0.2667	$1.38{\times}10^{-3}$
Independent PH-VAEs	1.8645	0.0359	$2.05{\times}10^{-3}$
PH-VAE	$\mathbf{0.1716}$	$\mathbf{0.0293}$	$\mathbf{9.79{\times}10^{-4}}$

Table 5: Real financial returns dependence metrics.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

\left\{\begin{array}[]{ll}s\leftarrow j&\text{with prob. }A_{sj}/(-A_{ss})\ ,~~j=1,2,...,m\\ s\leftarrow absorbing&\text{with prob. }t_{s}/(-A_{ss})\end{array}\right.

Appendix

Algorithm 2 Sampling from a Single Phase-Type Distribution

Panel of Figure 8 — Figure 8: Practical illustration of approximating a PDF with Phase-Type distributions with different number of phases $m$ .

Panel of Figure 9 — Figure 9: Approximation of a heavy-tailed target distribution, a Burr in this scenario, using finite Phase-Type models.

Data	$m$	Time / epoch (s)	Total time (s)	Final NLL $\downarrow$
Weibull ( $k=0.8$ )	2	5.40 $\pm$ 0.25	72.5	1.0895
	4	5.18 $\pm$ 0.16	67.6	1.0863
	8	5.29 $\pm$ 0.24	69.1	1.0866
	16	5.23 $\pm$ 0.17	68.1	1.0863
	32	5.20 $\pm$ 0.11	67.3	1.0877
Lognormal ( $\sigma=1.5$ )	2	5.23 $\pm$ 0.19	68.5	1.8255

Appendix

Table 6: Wall-clock training time and final Negative Log Likelihood (NLL) of PH-VAE as a function of the number of phases

m

across heavy-tailed distributions. Training time per epoch remains essentially constant as

m

increases, indicating that uniformization enables stable likelihood evaluation without introducing a noticeable computational overhead.

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 208 papers

Markov Chain Decoders Overcome the Heavy-Tail Limitations of Lipschitz Generative Models
May 18, 2026Abdelhakim Ziani, Andras Horvath, Paolo BallariniVariational Autoencoders Generative Modeling
Université Paris Saclay, Lab. MICS, CentraleSupélec, Gif-sur-Yvette, France · Università di Torino, Torino, Italy
Tail Annealing for Heavy-Tailed Flow Matching
May 19, 2026Jean PachebatFlow Matching Generative Modeling
1CMAP, Ecole Polytechnique, Institut Polytechnique de Paris, France.
Fine-Tuning Generative Models for Extreme Events via CVaR-Penalized Wasserstein Gradient Flows
Aug 12, 2026Thejani Gamage, Hyemin Gu, Zhizhen Zhang +3Wasserstein Gradient Flows Generative Modeling
Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, MA · School of Data and Information Sciences (SDIS) and Department of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC

May 18, 2026J/K move · Enter open · S save

Markov Chain Decoders Overcome the Heavy-Tail Limitations of Lipschitz Generative Models

Abdelhakim Ziani, Andras Horvath, Paolo Ballarini

Université Paris Saclay, Lab. MICS, CentraleSupélec, Gif-sur-Yvette, France · Università di Torino, Torino, Italy

Variational Autoencoders Generative Modeling Heavy-Tailed Distributions

Open paper AskPDF arXiv

Heavy-tailed distributions are prevalent in performance evaluation, network traffic, and risk modeling. This behavior poses a fundamental challenge for modern deep generative models. Standard Variational Autoencoders (VAEs) employ Gaussian decoder likelihoods and Lipschitz-constrained neural networks, a combination that is structurally incapable of producing heavy-tailed outputs: the Gaussian tail decays exponentially, and Lipschitz continuity prevents the decoder from amplifying rare events from the latent space input to sufficiently overcome this decay. We provide both a theoretical characterization of this limitation and a controlled empirical demonstration using synthetic Pareto data across a grid of tail indices

α

\in

{2, 3, 5, 30} and dimensions d

\in

{1, 5, 10}. As a solution, we replace the Gaussian decoder with a Phase-Type (PH) distribution based on Markov chains, while keeping the encoder, latent space, and training procedure identical. PH distributions allow for arbitrarily precise approximations of any positive-valued distributions, including heavy-tailed families. Experiments showed that the PH-based model reduces tail Kolmogorov-Smirnov distance by up to x6 and extreme quantile error by up to x10 compared to the Gaussian baseline for heavy-tailed data. These results demonstrate that integrating Markov chain-based distributions into the decoder of a generative model institutes a principled and practically effective solution to the heavy-tail generation problem.

Abstract

Variational Autoencoders

Explore similar work

Markov Chain Decoders Overcome the Heavy-Tail Limitations of Lipschitz Generative Models

Tail Annealing for Heavy-Tailed Flow Matching

Fine-Tuning Generative Models for Extreme Events via CVaR-Penalized Wasserstein Gradient Flows