PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models

Tadeusz Dziarmaga; Marcin Kądziołka; Laura Musioł; Patryk Marszałek; Artur Kasymov; Piotr Kościelniak; Marcin Mazur

cs.LGMar 24, 2025

PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models

Authors: Tadeusz Dziarmaga, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur

Organizations: Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland · Jagiellonian University, Doctoral School of Exact and Natural Sciences, Kraków, Poland

Abstract

Deep generative models (DGMs) have caused a paradigm shift in the field of machine learning, yielding noteworthy advancements in domains such as image synthesis, natural language processing, and other related areas. However, a comprehensive evaluation of these models that effectively captures the interplay among fidelity, diversity, and novelty in generated samples remains a persistent challenge. A recently proposed approach, the Feature Likelihood Divergence (FLD), offers a theoretically grounded and practical tool in this regard but faces notable computational limitations. In this paper, we propose PALATE, a novel framework designed to enhance the evaluation of DGMs by addressing the efficiency constraints of FLD. Our approach is based on a peculiar application of the law of total expectation to random variables representing accessible data. When integrated with the MMD baseline metric and the recently introduced DINOv3 feature extractor, PALATE provides a comprehensive evaluation framework that matches or exceeds state-of-the-art methods while offering superior computational efficiency and scalability to large-scale datasets. Through a series of experiments, we demonstrate that the PALATE enhancement constitutes a practical and computationally efficient framework for holistic DGM assessment, particularly in detecting sample memorization and evaluating generalization capabilities. The source code is provided as supplementary material.

Figures & tables

Figure 1: We propose PALATE, a memorization-aware enhancement to deep generative model assessment. Grounded in a peculiar application of the law of total expectation and leveraging DINOv3 representations, it yields $M_{\text{PALATE}}$ , an MMD-based holistic evaluation metric that captures fidelity, diversity, and memorization in a single score.

		Holistic metrics		Fidelity/diversity metrics			Fidelity metrics
	Model	$M_{\text{PALATE{}}}$ (our) $\downarrow$	FLD $\downarrow$	FD $\downarrow$	KD $\downarrow$	CMMD $\downarrow$	Precision $\uparrow$	Density $\uparrow$
CIFAR-10	LSGM-ODE	0.2618	4.25	148.27	0.18	0.025	0.74	0.76
	PFGM++	0.2614	4.51	141.65	0.17	0.022	0.75	0.80
	iDDPM-DDIM	0.2666	5.61	212.30	0.26	0.072	0.74	0.78
	StyleGAN-XL	0.2652	5.52	204.60	0.24	0.018	0.72	0.69
	StyleGAN2-ada	0.2721	6.91	305.92	0.45	0.037	0.71	0.70

Table 1: Evaluation of various DGMs on CIFAR-10 and ImageNet. Metrics implementations and experimental setups follow [ 18 ] (FLD and Gen. Gap FLD), [ 17 ] (CMMD), and [ 37 ] (all others). Calculations use the data examples from [ 37 ] ; mean per-class Vendi Score is omitted for some models due to missing class labels in the generated samples.

Figure 2: Spearman correlation heatmaps among evaluation metrics on CIFAR-10 (top row) and ImageNet (bottom row). For proper comparison, metric scales are aligned so that higher values consistently indicate better performance.

Panel of Figure 3 — Figure 3: Comparison of $M_{\text{PALATE{}}}$ (our) and FLD under transformations applied to samples generated by PFGM++ on CIFAR-10. Results for other models are shown for reference. Top : Nearly imperceptible transformations. Bottom : Large transformations.

Panel of Figure 4 — Figure 4: Capability of $M_{\text{PALATE{}}}$ (our) and FLD to capture increasing sample diversity as the number of classes grows, while keeping the total number of generated samples fixed at $10{,}000$ . Top: CIFAR-10. Bottom : ImageNet.

Figure 5: Capability of $M_{\text{PALATE{}}}$ (our) and FLD to detect diversity degradation under increasing class imbalance on CIFAR-10. We progressively duplicate $1{,}000$ fixed samples from a single (first) class while leaving all other classes unchanged.

Panel of Figure 6 — Figure 6: Performance of $M_{\text{PALATE}}$ (our) and FLD on mixtures of generated and training images, with the proportion of training samples varying from $0\%$ to $100\%$ . FLD becomes numerically unstable at $100\%$ (reaching values up to $\sim 10^{7}$ ) and is therefore shown only up to $99\%$ . Top : CIFAR-10. Bottom : ImageNet.

Panel of Figure 7 — Figure 7: Memorization ratios vs. metric values for CIFAR-10-trained models. Swapped train/test memorization metrics are shown with low opacity. Results for $C_{T}$ , $C_{T}$ mod., and AuthPct are taken from [ 37 ] . The $x$ -axis of PALATE (our) is inverted.

Panel of Figure 8 — Figure 8: Stability of $M_{\text{PALATE}}$ (our) and FLD as a function of sample size (mean $\pm$ standard deviation over 10 random seeds). Top: CIFAR-10. Bottom: ImageNet.

Figure 9: Performance of PALATE (our) and Generalization Gap FLD on prompts interpolating between MS-COCO training captions and PartiPrompts, with the proportion of PartiPrompts varying from $0\%$ to $100\%$ .

Figure 10: Evolution of $M_{\text{PALATE}}$ (our) across LoRA fine-tuning checkpoints of Stable Diffusion v1.5 on $5{,}000$ MS-COCO caption–image pairs.

Figure 11: Computation time of $M_{\text{PALATE}}$ (our) and FLD on ImageNet as a function of sample size. FLD is shown only up to $20{,}000$ samples due to memory limitations at larger scales. All runtimes are averaged over 10 random seeds.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Capability of Recall and Coverage metrics to capture increasing sample diversity as the number of classes grows, while keeping the total number of generated samples fixed at $10{,}000$ . Top : CIFAR-10. Bottom : ImageNet.

Figure 2: Performance of $M_{\text{PALATE}}$ (our) computed on DINOv2 embeddings on mixtures of generated and training images, with the proportion of training samples varying from $0\%$ to $100\%$ on CIFAR-10.

Figure 3: Stability of $M_{\text{PALATE}}$ (our) computed on DINOv2 embeddings as a function of sample size (mean $\pm$ standard deviation over 10 random seeds) on CIFAR-10.

Figure 4: Performance of $M_{\text{PALATE}}$ (our) computed on DINOv3 embeddings on mixtures of generated and training images, with the proportion of training samples varying from $0\%$ to $100\%$ on CIFAR-10.

	SCALE(DMMD)	PALATE
CIFAR-10	10.5	3.5
ImageNet	11.6	3.9
MS-COCO	10.9	3.6

Appendix

Table 1: Bandwidth

\sigma

values used in our experiments.

	Model	Hum. Err. Rate $\uparrow$	Mem. Ratio $\downarrow$		Model	Hum. Err. Rate $\uparrow$
CIFAR-10	LSGM-ODE	0.437 $\pm$ 0.010	0.7293	ImageNet 256 $\times$ 256	DiT-XL-2-guided	0.330 $\pm$ 0.013
	PFGM++	0.436 $\pm$ 0.011	0.9730		LDM	0.309 $\pm$ 0.017
	iDDPM-DDIM	0.400 $\pm$ 0.013	1.5160		DiT-XL-2	0.286 $\pm$ 0.016
	StyleGAN-XL	0.399 $\pm$ 0.012	0.7439		ADMG-ADMU	0.269 $\pm$ 0.016
	StyleGAN2-ada	0.393 $\pm$ 0.012	1.0339		ADM	0.266 $\pm$ 0.016
	BigGAN-Deep	0.387 $\pm$ 0.014	0.7970		ADMG	0.248 $\pm$ 0.012

Appendix

Table 2: Human Error Rate and Memorization Ratio of various DGMs on CIFAR-10 and ImageNet. Implementations follow [ 18 ] . Due to the substantial memory requirements of ImageNet and prior findings indicating that memorization is not a primary concern for this dataset [ 37 ] , the Memorization Ratio is not computed for ImageNet.

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 309 papers

A Theoretical Framework for Statistical Evaluability of Generative Models
Apr 7, 2026Shashaank Aiyer, Yishay Mansour, Shay Moran +1Generative AI Evaluation Statistical Learning Theory
University of Maryland · Tel Aviv University and Google Research · Technion and Google Research
Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Error
Feb 16, 2026Farzan Farnia, Mohammad Jalali, Azim OspanovGenerative Modeling Generative AI Evaluation
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
MIND: Monge Inception Distance for Generative Models Evaluation
May 7, 2026Quentin Berthet, Yu-Han Wu, Clement Crepy +3Image Generation Evaluation Sliced Wasserstein Distance
Google DeepMind · LPSM, Sorbonne Université

Apr 7, 2026J/K move · Enter open · S save

A Theoretical Framework for Statistical Evaluability of Generative Models

Shashaank Aiyer, Yishay Mansour, Shay Moran, Han Shao

University of Maryland · Tel Aviv University and Google Research · Technion and Google Research

Generative AI Evaluation Statistical Learning Theory Integral Probability Metrics

Open paper AskPDF arXiv

Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d. test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and Rényi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.

Abstract

Generative AI Evaluation

Explore similar work

A Theoretical Framework for Statistical Evaluability of Generative Models

Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Error

MIND: Monge Inception Distance for Generative Models Evaluation