When Marginals Match but Structure Fails: Covariance Fidelity in Generative Models

Nazia Riasat

stat.MLMar 17, 2026

When Marginals Match but Structure Fails: Covariance Fidelity in Generative Models

Authors: Nazia Riasat

Organizations: North Dakota State University

Abstract

Generative models are increasingly deployed as substitutes for real data in downstream scientific workflows, yet standard evaluation criteria remain focused on marginal distribution matching. We argue that this represents a fundamental gap: downstream inference is rarely a marginal operation, and a model that passes every univariate diagnostic can still produce structurally unreliable synthetic data. We introduce covariance-level dependence fidelity, measured by D_Sigma(P,Q) = ||Sigma_P - Sigma_Q||_F, as a principled, computable criterion for evaluating whether a generative model preserves the joint structure of data beyond its univariate marginals. Three results formalise this criterion. First, marginal fidelity provides no constraint on dependence structure: D_Sigma can be made arbitrarily large while all univariate marginals match exactly. Second, covariance divergence induces quantifiable downstream instability, including sign reversals in population regression coefficients. Third, bounding D_Sigma provides positive stability guarantees for dependence-sensitive procedures such as PCA via Davis-Kahan-type bounds. Empirical validation across three domains, image data (Fashion-MNIST VAE, n = 60,000), bulk RNA-seq (TCGA-BRCA, n = 1,111), and a small-sample stress test (Alzheimer's gene expression, n = 113), shows that D_Sigma/delta consistently distinguishes structure-discarding from structure-preserving generators in cases where standard marginal diagnostics show little separation, confirming that covariance-level fidelity provides information orthogonal to existing evaluation metrics across domains and sample sizes.

Figures & tables

Figure 1: Covariance structures for two bivariate normal distributions with identical marginals but opposite correlation signs. The change in dependence structure reverses the population regression slope, illustrating inferential instability under dependence divergence.

Figure 2: PCA subspace instability under covariance perturbation. Left: Subspace angle

\sin\angle(u_{P},u_{Q_{\varepsilon}})

as a function of perturbation

\varepsilon

. The dashed line shows the Theorem 3 Davis–Kahan bound; the shaded region marks the stable regime (

\varepsilon<\sqrt{2}

). The bound is nearly tight throughout and becomes vacuous at the eigengap threshold (dotted vertical line). Right: Leading eigenvectors at

\varepsilon=0.5

(stable) and

\varepsilon=2.5

(unstable), illustrating the qualitative transition in subspace alignment that marginal diagnostics cannot detect.

Figure 3: Diagnostic summary across all generator–dataset comparisons. (a) Each generator plotted by $D_{\Sigma}/\delta$ versus PCA subspace angle at $r=1$ ; bubble area encodes the RV-coefficient. The dashed vertical line marks the instability threshold ( $D_{\Sigma}/\delta=1$ , Theorem 3). (b) Median KS distance (hatched, lighter) versus $D_{\Sigma}/\delta$ (solid) per generator. KS distances cluster in $[0.02,0.26]$ with no structural separation; $D_{\Sigma}/\delta$ immediately identifies Splatter as unstable. Together, panels (a) and (b) visualize the central claim of Table 7 : marginal diagnostics cannot see what $D_{\Sigma}/\delta$ sees.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Theorem	Claim	Key quantity
Theorem 1	Marginal fidelity does not imply dependence fidelity	$D_{\Sigma}(P,Q)$ can be arbitrarily large with $P_{j}=Q_{j}$
Theorem 2	Dependence divergence induces inferential instability	$\|\beta(P)-\beta(Q)\|\leq\frac{1}{\sqrt{2}\,\sigma^{2}_{X}}\\|\Sigma_{P}-\Sigma_{Q}\\|_{F}$
Theorem 3	Dependence fidelity yields stability guarantees	$\\|\sin\Theta(U_{P},U_{Q})\\|_{2}\leq 2D_{\Sigma}(P,Q)/\gamma$

Appendix

Table 1: Summary of the three main theoretical results.

Figure 4: Joint extreme-event probabilities for Gaussian and $t$ copulas. Despite identical marginals, the $t$ copula exhibits substantially higher joint tail risk due to heavy-tail dependence.

Figure 5: Empirical marginal CDFs for Gaussian and $t$ -copula samples. The marginals are indistinguishable, demonstrating that marginal fidelity alone cannot detect dependence differences.

Figure 6: Population regression slope under two dependence structures with identical marginal distributions but opposite correlations. Dependence divergence induces a sign reversal in the regression coefficient, illustrating instability of downstream inference.

Figure 7: Heatmaps of the covariance difference matrix $\hat{\Sigma}_{\rm real}-\hat{\Sigma}_{\rm syn}$ for the VAE (left, range $[-0.35,0.35]$ ) and the Gaussian baseline (right, range $[-0.016,0.016]$ ) on Fashion-MNIST (top-50 PCA dimensions). Despite broadly comparable marginal KS profiles, the VAE exhibits substantially larger and more structured covariance residuals concentrated in the leading PC dimensions, reflecting the systematic elimination of off-diagonal covariance by the diagonal posterior assumption ( $D_{\Sigma}=4.92$ vs. $0.25$ ).

Figure 8: Distribution of KS distances across 50 PCA dimensions comparing real Fashion-MNIST data against VAE-generated (left, median = 0.090) and Gaussian baseline (right, median = 0.020) synthetic samples. Both generators achieve similar and small KS profiles, confirming that marginal diagnostics substantially understate the structural gap between the two generators ( $D_{\Sigma}(\text{VAE})=4.92$ vs. $D_{\Sigma}(\text{Gaussian})=0.25$ ).

	VAE ( $D_{\Sigma}/\delta=0.64$ )		Gaussian ( $D_{\Sigma}/\delta=0.03$ )
$r$	$\\|\sin\Theta\\|_{2}$	DK bound	$\\|\sin\Theta\\|_{2}$	DK bound
1	0.1940	1.278	0.0049	0.0652
2	0.1266	1.2294	0.0086	0.0628
3	0.6900	—	0.0236	0.6932
5	0.4817	—	0.0402	1.9028
10	0.4385	—	0.0284	—

Appendix

Table 2: PCA subspace angles on Fashion-MNIST across subspace dimension

r

(

n=60{,}000

p=50

PCA dimensions,

n/p=1{,}200

). The Theorem 3 Davis–Kahan bound uses the local eigengap

\delta_{r}=\lambda_{r}-\lambda_{r+1}

; “—” denotes a vacuous bound (

D_{\Sigma}/\delta_{r}\geq 1

), where

\|\sin\Theta\|_{2}

denotes the principal angle between the leading

r

-dimensional subspaces.

Figure 9: PCA subspace angles $\|\sin\Theta(\hat{U}_{\rm real},\hat{U}_{\rm syn})\|_{2}$ as a function of subspace dimension $r$ for the VAE ( $D_{\Sigma}/\delta=0.64$ , blue) and the Gaussian baseline ( $D_{\Sigma}/\delta=0.03$ , green) on Fashion-MNIST ( $n=60{,}000$ , $p=50$ PCA dimensions). The VAE angle jumps sharply at $r=3$ when the perturbation crosses the local eigengap, consistent with Theorem 3; the Gaussian baseline remains stable throughout. The dotted line at 1.0 marks the vacuous-bound threshold.

Figure 10: Population regression coefficients (PC 1 on PC 2 through PC 10 ) estimated from real vs. synthetic Fashion-MNIST data. The VAE (left) produces coefficients scattered far from the identity line with sign reversals, consistent with $D_{\Sigma}/\delta=0.64$ and Theorem 2. The Gaussian baseline (right) tracks the identity closely within a narrow band, consistent with $D_{\Sigma}/\delta=0.03$ and the stability guarantee of Theorem 3.

Figure 11: $D_{\Sigma}$ versus $1-\text{RV}$ across 200 random PC subsets of size 20 for the VAE (left) and Gaussian baseline (right) on Fashion-MNIST. The VAE shows moderate Spearman correlation ( $r=-0.383$ , $p<0.001$ ) with KS $p=0.142$ , indicating the two metrics are concordant but not identical in distribution. The Gaussian baseline shows significant distributional divergence between $D_{\Sigma}$ and $1-\text{RV}$ (KS $p=0.000$ ) despite concordant rank ordering, reflecting the scale sensitivity of raw $D_{\Sigma}$ versus the bounded RV-coefficient. Together these results confirm that $D_{\Sigma}/\delta$ and RV are complementary rather than redundant diagnostics.

Figure 12: Bootstrap distributions of $\hat{D}_{\Sigma}$ ( $B=500$ resamples) for the VAE (left) and Gaussian baseline (right) on Fashion-MNIST. Black vertical lines show the observed values; red dashed lines mark the 95% confidence intervals. The non-overlapping CIs ( $[4.83,5.04]$ vs. $[0.33,0.56]$ ) confirm that the difference in $D_{\Sigma}$ between the two generators, reflecting severe bootstrap upward bias: the interval lies entirely above the observed value, yielding zero empirical coverage of the point estimate. This is a known consequence of resampling instability when $n\ll d$ (see Appendix B.3); the two generators remain clearly separated.

Model	$D_{\Sigma}$	$D_{\Sigma}/\delta$	$\tilde{D}_{\Sigma}$	RV	95% CI	Spearman $(D_{\Sigma}$ , $1$ -RV )	KS $p$
VAE (diagonal posterior)	4.92	0.64	3.125	0.9835	[4.83, 5.04]	-0.383	0.142
Gaussian (structure-preserving)	0.25	0.03	0.210	0.9999	[0.33, 0.56]	-0.427	0.000

Appendix

Table 3: Complete dependence fidelity diagnostics on Fashion-MNIST (

n=60{,}000

p=50

PCA dimensions,

n/p=1{,}200

). Bootstrap 95% CIs based on

B=500

resamples. Spearman correlation and KS test compare

D_{\Sigma}

and

1-\text{RV}

across 200 random PC subsets of size 20. Note: † Bootstrap CI for the Gaussian baseline reflects right-skew of the bootstrap distribution at near-zero

D_{\Sigma}

; see Appendix B.3 for discussion.

Metric	Value	Metric	Value
$D_{\Sigma}$	49.35	Eigengap $\delta$	279.87
$\tilde{D}_{\Sigma}$	3.69	$D_{\Sigma}/\delta$	0.18
RV-coefficient	0.9929	$1-\text{RV}$	0.0071
Median KS	0.26	KS $p$ -value	0.581
Spearman $(D_{\Sigma},1-\text{RV})$	$-0.010$
Subspace diagnostics

Appendix

Table 4: Dependence fidelity diagnostics: Alzheimer’s dataset (

n=113

p=50

genes,

n/p=2.26

). All values should be interpreted as qualitative directional estimates given the small sample size. Bootstrap CI is not reported as it is unreliable in this regime; see text for discussion.

Figure 13: Distribution of KS distances across genes (Alzheimer’s dataset, $n=113$ ). Most KS values are small, indicating approximate marginal preservation. This marginal similarity does not preclude covariance-level discrepancy, as shown in Figure 14 .

Figure 14: Covariance difference matrix $\Sigma_{\rm real}-\Sigma_{\rm syn}$ for the Alzheimer’s dataset. Despite similar marginal KS profiles, visible off-diagonal residuals indicate that dependence structure is not preserved by the marginal-only Gaussian generator. Given $n/p=2.26$ , these patterns should be interpreted as qualitative directional evidence rather than precise numerical estimates.

Figure 15: Regression coefficients estimated from real versus synthetic Alzheimer’s data. Deviations from the identity reflect sensitivity to dependence structure, consistent with Theorem 2, though numerical magnitudes should be interpreted cautiously given the small sample size.

Figure 16: PCA subspace angle between real and synthetic gene expression data as a function of subspace dimension

r

. Despite similar marginal distributions (Figure 13 ), the leading principal subspaces diverge substantially, consistent with the covariance distortion shown in Figure 14 and the instability predicted by Theorem 3 when

D_{\Sigma}

exceeds the eigengap.

Figure 17:

D_{\Sigma}

versus

1-\text{RV}

across 100 random gene subsets of size 20 (Alzheimer’s dataset,

n=113

p=50

). Spearman

r=-0.010

(

p=0.92

), KS

p=0.581

. The near-zero correlation indicates that

D_{\Sigma}

and

1-\text{RV}

are effectively decoupled in this small-sample regime, in contrast to the concordant behaviour observed in Appendix B.2 (Fashion-MNIST, Spearman

r=-0.383

) and Appendix B.6 (TCGA-BRCA,

r=0.748

for Splatter). The decoupling reflects estimation noise at

n/p=2.26

rather than a structural property of the metrics.

Figure 18: Dependence collapse in a diagonal-covariance VAE with $\rho=0.8$ . Left two panels: Covariance matrices of the true distribution $P$ and the diagonal approximation $Q$ . The off-diagonal structure present in $\Sigma_{P}$ is eliminated by the independence assumption, yielding $D_{\Sigma}(P,Q)=\sqrt{2}\,|\rho|\approx 1.131$ . Right panel: Samples from $P$ (correlated, blue) and $Q$ (independent, red) with population regression lines overlaid. Despite identical marginals $\mathcal{N}(0,1)$ , the regression slope collapses from $\beta(P)=0.8$ to $\beta(Q)=0$ under the diagonal constraint, illustrating inferential instability consistent with Theorem 2 .

	Splatter ( $D_{\Sigma}=249.79$ )		Poisson thinning ( $D_{\Sigma}=88.35$ )
$r$	$\\|\sin\Theta\\|_{2}$	DK bound	$\\|\sin\Theta\\|_{2}$	DK bound
1	0.9985	—	0.2099	1.5242
2	0.9978	—	0.4371	—
3	0.9943	—	0.4085	—
5	1.0000	—	0.8749	—
10	0.9983	—	0.9579	—

Appendix

Table 5: PCA subspace angles between real and synthetic TCGA-BRCA data across subspace dimension

r

. The Theorem 3 bound

D_{\Sigma}/\delta

uses the local eigengap

\delta_{r}=\lambda_{r}-\lambda_{r+1}

. For Splatter all bounds are vacuous (

D_{\Sigma}/\delta>1

, shown as “—”); for Poisson thinning the

r=1

bound are vacuous; stability is confirmed empirically. Both share the same real covariance (

\delta=121.67

n=1{,}111

p=100

Model	$D_{\Sigma}$	$D_{\Sigma}/\delta$	$\tilde{D}_{\Sigma}$	RV	95% CI	SE	Spearman $(D_{\Sigma},1\text{-RV})$	KS $p$
Splatter	249.79	2.05	26.58	0.299	[237.10, 265.62]	7.28	0.748	0.906
Poisson thinning	88.35	1.53	6.54	0.961	[82.56, 107.78]	6.60	$-$ 0.195	0.699

Appendix

Table 6: Extended dependence fidelity diagnostics for TCGA-BRCA (

n=1{,}111

p=100

HVGs,

n/p=11.1

). Bootstrap 95% CI based on

B=500

resamples. Spearman correlation and KS test compare

D_{\Sigma}

and

1-\text{RV}

across 100 random gene subsets of size 30, where

\tilde{D}_{\Sigma}=\|C_{P}-C_{Q}\|_{F}

is the normalized (correlation-matrix) form of the divergence.

Figure 19: Distribution of KS distances across 100 HVGs comparing real TCGA-BRCA data against Splatter-generated (left) and Poisson-thinned (right) synthetic data. Both simulators exhibit broadly similar KS profiles, confirming approximate marginal fidelity in both cases. Marginal diagnostics alone cannot distinguish the two simulators, despite their substantially different covariance divergences (

D_{\Sigma}=249.79

vs.

88.35

Figure 20: Heatmaps of the covariance difference matrix

\hat{\Sigma}_{\text{real}}-\hat{\Sigma}_{\text{syn}}

for Splatter (top, range

[-15,15]

) and Poisson thinning (bottom, range

[-5,5]

). Despite similar marginal KS distances, Splatter exhibits substantially larger and more structured covariance residuals, reflecting the absence of joint modelling.

Figure 21: Population regression coefficients (gene 1 on genes 2–10) estimated from real vs. synthetic TCGA-BRCA data. Splatter (top) produces coefficients with near-zero correlation to the real values and frequent sign reversals, consistent with

D_{\Sigma}/\delta=2.05

. Poisson thinning (bottom) tracks the real coefficients closely along the identity line with few sign flips, consistent with

D_{\Sigma}/\delta=1.53

and Theorem 2 .

Figure 22: PCA subspace angles $\|\sin\Theta(\hat{U}_{\text{real}},\hat{U}_{\text{syn}})\|_{2}$ as a function of subspace dimension $r$ for Splatter (blue, $D_{\Sigma}/\delta=2.05$ ) and Poisson thinning (green, $D_{\Sigma}/\delta=1.53$ ) on TCGA-BRCA ( $n=1{,}111$ , $p=100$ HVGs). Splatter angles range from $0.9943$ to $1.0000$ across all $r$ , indicating near-complete subspace misalignment. Poisson thinning remains below $1.0$ for all $r$ and satisfies the Theorem 3 Davis–Kahan bound at $r=1$ ( $0.2099<1.53$ ). The degradation at higher $r$ in both cases reflects decreasing local eigengaps, as predicted by Theorem 3 .

Figure 23:

D_{\Sigma}

versus

1-\text{RV}

across 100 random gene subsets of size 30 (TCGA-BRCA, Splatter,

n=1{,}111

p=100

). Spearman

r=0.748

(

p<0.001

), KS

p=0.906

. The strong positive concordance and indistinguishable distributions after standardisation confirm that both metrics detect the same widespread covariance failure in the large-divergence unstable regime (

D_{\Sigma}/\delta=2.05

). This is the strongest concordance result across all datasets in this study.

Figure 25: Comparison of three evaluation metrics, median KS distance, $D_{\Sigma}/\delta$ , and MMD (Gaussian kernel, median heuristic bandwidth [ Gretton et al., 2012 ] ), for Splatter and Poisson thinning on TCGA-BRCA ( $n=1{,}111$ , $p=100$ HVGs). The dashed red line marks the $D_{\Sigma}/\delta$ instability threshold of $1$ from Theorem 3. KS distances are identical across generators ( $0.11$ each). MMD shows a modest separation ( $0.338$ vs. $0.09$ ) but provides no actionable stability criterion. Only $D_{\Sigma}/\delta$ places Splatter clearly in the unstable regime ( $2.053$ ) and Poisson thinning in the closer stable regime ( $1.53$ ), directly predicting the downstream consequences documented in Tables 6 and 7 .

		Marginal	Covariance-level			Downstream
Dataset	Model	Median KS	$D_{\Sigma}/\delta$	Regime	RV	$\\|\sin\Theta\\|_{2}\ (r{=}1)$	Sign flips
Fashion-MNIST	VAE (diagonal)	0.090	0.64	stable	0.984	0.194	many
Fashion-MNIST	Gaussian ‡	0.020	0.03	stable	1.000	0.005	none
TCGA-BRCA	Splatter ∗	0.127	2.05	unstable	0.299	0.999	many
TCGA-BRCA	Poisson thinning ∗	0.110	1.53	unstable	0.961	0.210	few
Alzheimer’s †	Gaussian	0.260	0.18	stable	0.993	0.070	—

Appendix

Table 7: Empirical summary across all datasets and generator comparisons. Columns are organised by diagnostic depth: marginal fidelity (KS), covariance divergence (

D_{\Sigma}/\delta

, RV), and downstream instability (subspace angle at

r=1

, regression sign flips).

\dagger

Alzheimer’s values are qualitative only (

n/p=2.26

); sign flips are not reported because regression magnitudes are unreliable at this sample size (see Appendix B.3), not because reversals are absent. Figure 14 shows substantial coefficient divergence.

\ddagger

Gaussian baseline preserves covariance by construction. Both

*

Splatter and

*

Poisson thinning are in unstable regime (

D_{\Sigma}/\delta>1

); DK bound is vacuous. Sign flips: number of reversed regression coefficients out of 9. For Fashion-MNIST, PC1 regressed on PC2–PC10; for TCGA-BRCA, gene 1 regressed on genes 2–10. Here ‘many’ indicates

\geq 5

reversals; ‘few’ indicates

\leq 2

. For Regime: stable if (

D_{\Sigma}/\delta<1

) and unstable if (

D_{\Sigma}/\delta>=1

Theorem	Claim	Empirical evidence	Status
Theorem 1	Marginal fidelity does not imply dependence fidelity; $D_{\Sigma}$ can be large while KS $\approx 0$	KS distances are similar across all generator pairs within each dataset, yet $D_{\Sigma}/\delta$ and RV differ by factors of $10$ – $165\times$	✓
Theorem 2	Covariance divergence induces regression instability bounded by $\frac{1}{\sqrt{2}\sigma_{X}^{2}}\\|\Sigma_{P}-\Sigma_{Q}\\|_{F}$	Regression sign flips present for all generators with $D_{\Sigma}/\delta>0.5$ ; absent for all generators with $D_{\Sigma}/\delta<0.1$	✓
Theorem 3	$\\|\sin\Theta\\|_{2}\leq 2D_{\Sigma}/\delta$ at $r=1$ when $D_{\Sigma}<\delta$	Bound is vacuous for Poisson thinning and Alzheimer’s; VAE bound confirmed at $r=1$ (observed 0.194 < 1.278). Empirical stability confirmed across all datasets.	✓
Remark 1	$D_{\Sigma}$ does not capture tail or nonlinear dependence	t-copula example (Appendix B.1) shows joint extreme-event failures invisible to $D_{\Sigma}$ by design	✓

Appendix

Table 8: Theoretical claims and empirical confirmation. Each theorem is listed with the empirical evidence that confirms it across datasets. “Confirmed” means the predicted direction holds in every comparison examined. The

D_{\Sigma}/\delta

thresholds in the Theorem 2 row (0.5 and 0.1) are empirical observations from this study, not theoretically derived cutoffs.

Generator	Median KS	MMD	$D_{\Sigma}/\delta$
Poisson thinning	0.11	0.09	1.53
Splatter	0.11	0.338	2.053
Instability threshold	—	—	1.000

Appendix

Table 9: KS, MMD, and

D_{\Sigma}/\delta

across generators on TCGA-BRCA (

n=1{,}111

p=100

HVGs). The dashed threshold marks

D_{\Sigma}/\delta=1

(Theorem 3). Only

D_{\Sigma}/\delta

provides an actionable stability criterion.

Explore similar work

Jun 8, 2026

Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records

Synthetic healthcare data are widely proposed as privacy-preserving substitutes for real patient data, yet their evaluation remains dominated by statistical similarity and predictive performance that do not reflect clinical validity. We introduce a multi-dimensional evaluation framework grounded in epidemiology, assessing descriptive fidelity, clinical utility, and structural validity, corresponding to descriptive, predictive, and causal questions. We evaluate four representative generative paradigms - GAN-based, VAE-boosted, diffusion-based, and masked modelling - using PRIME-CVD, a 50,000-person cohort with known ground-truth structure. While all models reproduce marginal distributions, none simultaneously preserve subgroup structure, effect estimates, and dependency structure. Notably, models with strong distributional fidelity can exhibit poor calibration and distorted relationships, leading to unreliable inference. These results show that current evaluation practices can overestimate synthetic data quality and motivate domain-informed assessment based on the ability to support valid clinical and scientific conclusions.

Nicholas I-Hsien Kuo, Blanca Gallego, Louisa Jorm

Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia

Synthetic Tabular Data Generation Electronic Health Records Synthetic Data Evaluation

Ask PDF

Apr 7, 2026

A Theoretical Framework for Statistical Evaluability of Generative Models

Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d. test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and Rényi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.

Shashaank Aiyer, Yishay Mansour, Shay Moran +1

University of Maryland · Tel Aviv University and Google Research · Technion and Google Research

Generative AI Evaluation Statistical Learning Theory Integral Probability Metrics

Ask PDF

Sep 27, 2026

StatD2GAN: When Calibration Masks Generator Quality in Held-Out Evaluation of Synthetic Weather Sequences

Generative models for multivariate weather series are routinely evaluated with pooled distributional metrics computed after marginal calibration. We show this practice can invalidate architectural conclusions, and rebuild the evaluation of StatD2GAN, a three-discriminator GAN with evolutionary weight adaptation, around a held-out protocol: the final two calendar years of each dataset are held out behind a 168 hour embargo, calibration is fitted on the training block only, and all metrics are computed on the held-out block. Evidence comes from 25 matched (location, seed) pairs across five Koppen-Geiger climates, tested with Wilcoxon signed-rank tests under Holm correction. Four results follow. First, isotonic calibration drives the Kolmogorov-Smirnov distance to within 2% of a per-location noise-and-shift floor for every architecture tested, including a deliberately weak RCGAN baseline, so calibrated marginal metrics cannot discriminate between architectures. Second, the sorted-representation discriminator is the only component whose removal significantly degrades cross-variable dependence (Kendall tau MAE +0.080, Holm p = 0.009), with a regime-dependent effect: near zero in Ankara, above 115% in Dubai and Yakutsk. A rank-transformed variant isolates the mechanism as quantile supervision of the marginals rather than copula matching. Third, physical constraint violations are injected by calibration, not the generator; projection removes them at negligible cost (deltaKS <= 0.003). Fourth, pooled metrics conceal a collapse of between-sequence weekly-mean variability, a proxy for seasonal and regime diversity, in TimeGAN that only sequence-level statistics expose. We recommend floor-referenced marginal evaluation, matched-pair testing, and sequence-level variance decomposition as minimum requirements for calibrated generative pipelines.

Mustafa Ozaytac, Ozge Karadag Atas

Department of Statistics, Graduate School of Science and Engineering, Hacettepe University, Ankara, 06800, Turkey · Department of Statistics, Faculty of Science, Hacettepe University, Ankara, 06800, Turkey

Time Series Generation Model Calibration Generative Adversarial Networks Synthetic Data Evaluation

Ask PDF