PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models
Authors: Tadeusz Dziarmaga, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur
Organizations: Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland · Jagiellonian University, Doctoral School of Exact and Natural Sciences, Kraków, Poland
Deep generative models (DGMs) have caused a paradigm shift in the field of machine learning, yielding noteworthy advancements in domains such as image synthesis, natural language processing, and other related areas. However, a comprehensive evaluation of these models that effectively captures the interplay among fidelity, diversity, and novelty in generated samples remains a persistent challenge. A recently proposed approach, the Feature Likelihood Divergence (FLD), offers a theoretically grounded and practical tool in this regard but faces notable computational limitations. In this paper, we propose PALATE, a novel framework designed to enhance the evaluation of DGMs by addressing the efficiency constraints of FLD. Our approach is based on a peculiar application of the law of total expectation to random variables representing accessible data. When integrated with the MMD baseline metric and the recently introduced DINOv3 feature extractor, PALATE provides a comprehensive evaluation framework that matches or exceeds state-of-the-art methods while offering superior computational efficiency and scalability to large-scale datasets. Through a series of experiments, we demonstrate that the PALATE enhancement constitutes a practical and computationally efficient framework for holistic DGM assessment, particularly in detecting sample memorization and evaluating generalization capabilities. The source code is provided as supplementary material.
Figures & tables
Figure 1: We propose PALATE, a memorization-aware enhancement to deep generative model assessment. Grounded in a peculiar application of the law of total expectation and leveraging DINOv3 representations, it yields MPALATE , an MMD-based holistic evaluation metric that captures fidelity, diversity, and memorization in a single score.
Holistic metrics
Fidelity/diversity metrics
Fidelity metrics
Model
MPALATE (our) ↓
FLD ↓
FD ↓
KD ↓
CMMD ↓
Precision ↑
Density ↑
CIFAR-10
LSGM-ODE
0.2618
4.25
148.27
0.18
0.025
0.74
0.76
PFGM++
0.2614
4.51
141.65
0.17
0.022
0.75
0.80
iDDPM-DDIM
0.2666
5.61
212.30
0.26
0.072
0.74
0.78
StyleGAN-XL
0.2652
5.52
204.60
0.24
0.018
0.72
0.69
StyleGAN2-ada
0.2721
6.91
305.92
0.45
0.037
0.71
0.70
Table 1: Evaluation of various DGMs on CIFAR-10 and ImageNet. Metrics implementations and experimental setups follow [ 18 ] (FLD and Gen. Gap FLD), [ 17 ] (CMMD), and [ 37 ] (all others). Calculations use the data examples from [ 37 ] ; mean per-class Vendi Score is omitted for some models due to missing class labels in the generated samples.
Figure 2: Spearman correlation heatmaps among evaluation metrics on CIFAR-10 (top row) and ImageNet (bottom row). For proper comparison, metric scales are aligned so that higher values consistently indicate better performance.
Figure 3: Comparison of MPALATE (our) and FLD under transformations applied to samples generated by PFGM++ on CIFAR-10. Results for other models are shown for reference. Top : Nearly imperceptible transformations. Bottom : Large transformations.
Figure 4: Capability of MPALATE (our) and FLD to capture increasing sample diversity as the number of classes grows, while keeping the total number of generated samples fixed at 10,000 . Top: CIFAR-10. Bottom : ImageNet.
Figure 5: Capability of MPALATE (our) and FLD to detect diversity degradation under increasing class imbalance on CIFAR-10. We progressively duplicate 1,000 fixed samples from a single (first) class while leaving all other classes unchanged.
Figure 6: Performance of MPALATE (our) and FLD on mixtures of generated and training images, with the proportion of training samples varying from 0% to 100% . FLD becomes numerically unstable at 100% (reaching values up to ∼107 ) and is therefore shown only up to 99% . Top : CIFAR-10. Bottom : ImageNet.
Figure 7: Memorization ratios vs. metric values for CIFAR-10-trained models. Swapped train/test memorization metrics are shown with low opacity. Results for CT , CT mod., and AuthPct are taken from [ 37 ] . The x -axis of PALATE (our) is inverted.
Figure 8: Stability of MPALATE (our) and FLD as a function of sample size (mean ± standard deviation over 10 random seeds). Top: CIFAR-10. Bottom: ImageNet.
Figure 9: Performance of PALATE (our) and Generalization Gap FLD on prompts interpolating between MS-COCO training captions and PartiPrompts, with the proportion of PartiPrompts varying from 0% to 100% .
Figure 10: Evolution of MPALATE (our) across LoRA fine-tuning checkpoints of Stable Diffusion v1.5 on 5,000 MS-COCO caption–image pairs.
Figure 11: Computation time of MPALATE (our) and FLD on ImageNet as a function of sample size. FLD is shown only up to 20,000 samples due to memory limitations at larger scales. All runtimes are averaged over 10 random seeds.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Capability of Recall and Coverage metrics to capture increasing sample diversity as the number of classes grows, while keeping the total number of generated samples fixed at 10,000 . Top : CIFAR-10. Bottom : ImageNet.
Figure 2: Performance of MPALATE (our) computed on DINOv2 embeddings on mixtures of generated and training images, with the proportion of training samples varying from 0% to 100% on CIFAR-10.
Figure 3: Stability of MPALATE (our) computed on DINOv2 embeddings as a function of sample size (mean ± standard deviation over 10 random seeds) on CIFAR-10.
Figure 4: Performance of MPALATE (our) computed on DINOv3 embeddings on mixtures of generated and training images, with the proportion of training samples varying from 0% to 100% on CIFAR-10.
SCALE(DMMD)
PALATE
CIFAR-10
10.5
3.5
ImageNet
11.6
3.9
MS-COCO
10.9
3.6
Appendix
Table 1: Bandwidth σ values used in our experiments.
Model
Hum. Err. Rate ↑
Mem. Ratio ↓
Model
Hum. Err. Rate ↑
CIFAR-10
LSGM-ODE
0.437 ± 0.010
0.7293
ImageNet 256 × 256
DiT-XL-2-guided
0.330 ± 0.013
PFGM++
0.436 ± 0.011
0.9730
LDM
0.309 ± 0.017
iDDPM-DDIM
0.400 ± 0.013
1.5160
DiT-XL-2
0.286 ± 0.016
StyleGAN-XL
0.399 ± 0.012
0.7439
ADMG-ADMU
0.269 ± 0.016
StyleGAN2-ada
0.393 ± 0.012
1.0339
ADM
0.266 ± 0.016
BigGAN-Deep
0.387 ± 0.014
0.7970
ADMG
0.248 ± 0.012
Appendix
Table 2: Human Error Rate and Memorization Ratio of various DGMs on CIFAR-10 and ImageNet. Implementations follow [ 18 ] . Due to the substantial memory requirements of ImageNet and prior findings indicating that memorization is not a primary concern for this dataset [ 37 ] , the Memorization Ratio is not computed for ImageNet.
Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d. test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and Rényi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.
Shashaank Aiyer, Yishay Mansour, Shay Moran +1
University of Maryland · Tel Aviv University and Google Research · Technion and Google Research
Deep generative models have achieved great success in producing high-quality samples, making them a central tool across machine learning applications. Beyond sample quality, an important yet less systematically studied question is whether trained generative models faithfully capture the diversity of the underlying data distribution. In this work, we address this question by directly comparing the diversity of samples generated by state-of-the-art models with that of test samples drawn from the target data distribution, using recently proposed reference-free entropy-based diversity scores, Vendi and RKE. Across multiple benchmark datasets, we find that test data consistently attains substantially higher Vendi and RKE diversity scores than the generated samples, suggesting a systematic downward diversity bias in modern generative models. To understand the origin of this bias, we analyze the finite-sample behavior of entropy-based diversity scores and show that their expected values increase with sample size, implying that diversity estimated from finite training sets could inherently underestimate the diversity of the true distribution. As a result, optimizing the generators to minimize divergence to empirical data distributions would induce a loss of diversity. Finally, we discuss potential diversity-aware regularization and guidance strategies based on Vendi and RKE as principled directions for mitigating this bias, and provide empirical evidence suggesting their potential to improve the results.
Farzan Farnia, Mohammad Jalali, Azim Ospanov
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
We propose the Monge Inception Distance (MIND), a metric for evaluating generative models that addresses key limitations of the widely adopted Fréchet Inception Distance (FID). The MIND metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. This approach circumvents the estimation of high-dimensional means and covariance matrices, which underlie FID's poor sample complexity and vulnerability to adversarial attacks. We empirically demonstrate three primary advantages: (i) it is more sample-efficient by one order of magnitude, (ii) it is faster to compute by two orders of magnitude, (iii) it is more robust to adversarial attacks such as moment-matching. We show that MIND with 5k samples can replace the evaluation performance of FID with 50k samples, providing high correlation with this standard benchmark and superior discriminative performance. We further demonstrate that even smaller sample sizes (e.g., 1k or 2k) remain highly informative for rapid model iteration.