stat.MLMar 17, 2026
SaveWhen Marginals Match but Structure Fails: Covariance Fidelity in Generative Models
Organizations: North Dakota State University
Abstract
Generative models are increasingly deployed as substitutes for real data in downstream scientific workflows, yet standard evaluation criteria remain focused on marginal distribution matching. We argue that this represents a fundamental gap: downstream inference is rarely a marginal operation, and a model that passes every univariate diagnostic can still produce structurally unreliable synthetic data. We introduce covariance-level dependence fidelity, measured by D_Sigma(P,Q) = ||Sigma_P - Sigma_Q||_F, as a principled, computable criterion for evaluating whether a generative model preserves the joint structure of data beyond its univariate marginals. Three results formalise this criterion. First, marginal fidelity provides no constraint on dependence structure: D_Sigma can be made arbitrarily large while all univariate marginals match exactly. Second, covariance divergence induces quantifiable downstream instability, including sign reversals in population regression coefficients. Third, bounding D_Sigma provides positive stability guarantees for dependence-sensitive procedures such as PCA via Davis-Kahan-type bounds. Empirical validation across three domains, image data (Fashion-MNIST VAE, n = 60,000), bulk RNA-seq (TCGA-BRCA, n = 1,111), and a small-sample stress test (Alzheimer's gene expression, n = 113), shows that D_Sigma/delta consistently distinguishes structure-discarding from structure-preserving generators in cases where standard marginal diagnostics show little separation, confirming that covariance-level fidelity provides information orthogonal to existing evaluation metrics across domains and sample sizes.
Figures & tables
Figure 1: Covariance structures for two bivariate normal distributions with identical marginals but opposite correlation signs. The change in dependence structure reverses the population regression slope, illustrating inferential instability under dependence divergence.
Figure 2: PCA subspace instability under covariance perturbation. Left: Subspace angle as a function of perturbation . The dashed line shows the Theorem 3 Davis–Kahan bound; the shaded region marks the stable regime ( ). The bound is nearly tight throughout and becomes vacuous at the eigengap threshold (dotted vertical line). Right: Leading eigenvectors at (stable) and (unstable), illustrating the qualitative transition in subspace alignment that marginal diagnostics cannot detect.
Figure 3: Diagnostic summary across all generator–dataset comparisons. (a) Each generator plotted by versus PCA subspace angle at ; bubble area encodes the RV-coefficient. The dashed vertical line marks the instability threshold ( , Theorem 3). (b) Median KS distance (hatched, lighter) versus (solid) per generator. KS distances cluster in with no structural separation; immediately identifies Splatter as unstable. Together, panels (a) and (b) visualize the central claim of Table 7 : marginal diagnostics cannot see what sees.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Theorem | Claim | Key quantity |
|---|---|---|
| Theorem 1 | Marginal fidelity does not imply dependence fidelity | can be arbitrarily large with |
| Theorem 2 | Dependence divergence induces inferential instability | |
| Theorem 3 | Dependence fidelity yields stability guarantees |
Table 1: Summary of the three main theoretical results.
Figure 4: Joint extreme-event probabilities for Gaussian and copulas. Despite identical marginals, the copula exhibits substantially higher joint tail risk due to heavy-tail dependence.
Figure 5: Empirical marginal CDFs for Gaussian and -copula samples. The marginals are indistinguishable, demonstrating that marginal fidelity alone cannot detect dependence differences.
Figure 6: Population regression slope under two dependence structures with identical marginal distributions but opposite correlations. Dependence divergence induces a sign reversal in the regression coefficient, illustrating instability of downstream inference.
Figure 7: Heatmaps of the covariance difference matrix for the VAE (left, range ) and the Gaussian baseline (right, range ) on Fashion-MNIST (top-50 PCA dimensions). Despite broadly comparable marginal KS profiles, the VAE exhibits substantially larger and more structured covariance residuals concentrated in the leading PC dimensions, reflecting the systematic elimination of off-diagonal covariance by the diagonal posterior assumption ( vs. ).
Figure 8: Distribution of KS distances across 50 PCA dimensions comparing real Fashion-MNIST data against VAE-generated (left, median = 0.090) and Gaussian baseline (right, median = 0.020) synthetic samples. Both generators achieve similar and small KS profiles, confirming that marginal diagnostics substantially understate the structural gap between the two generators ( vs. ).
| VAE ( ) | Gaussian ( ) | |||
|---|---|---|---|---|
| DK bound | DK bound | |||
| 1 | 0.1940 | 1.278 | 0.0049 | 0.0652 |
| 2 | 0.1266 | 1.2294 | 0.0086 | 0.0628 |
| 3 | 0.6900 | — | 0.0236 | 0.6932 |
| 5 | 0.4817 | — | 0.0402 | 1.9028 |
| 10 | 0.4385 | — | 0.0284 | — |
Table 2: PCA subspace angles on Fashion-MNIST across subspace dimension ( , PCA dimensions, ). The Theorem 3 Davis–Kahan bound uses the local eigengap ; “—” denotes a vacuous bound ( ), where denotes the principal angle between the leading -dimensional subspaces.
Figure 9: PCA subspace angles as a function of subspace dimension for the VAE ( , blue) and the Gaussian baseline ( , green) on Fashion-MNIST ( , PCA dimensions). The VAE angle jumps sharply at when the perturbation crosses the local eigengap, consistent with Theorem 3; the Gaussian baseline remains stable throughout. The dotted line at 1.0 marks the vacuous-bound threshold.
Figure 10: Population regression coefficients (PC 1 on PC 2 through PC 10 ) estimated from real vs. synthetic Fashion-MNIST data. The VAE (left) produces coefficients scattered far from the identity line with sign reversals, consistent with and Theorem 2. The Gaussian baseline (right) tracks the identity closely within a narrow band, consistent with and the stability guarantee of Theorem 3.
Figure 11: versus across 200 random PC subsets of size 20 for the VAE (left) and Gaussian baseline (right) on Fashion-MNIST. The VAE shows moderate Spearman correlation ( , ) with KS , indicating the two metrics are concordant but not identical in distribution. The Gaussian baseline shows significant distributional divergence between and (KS ) despite concordant rank ordering, reflecting the scale sensitivity of raw versus the bounded RV-coefficient. Together these results confirm that and RV are complementary rather than redundant diagnostics.
Figure 12: Bootstrap distributions of ( resamples) for the VAE (left) and Gaussian baseline (right) on Fashion-MNIST. Black vertical lines show the observed values; red dashed lines mark the 95% confidence intervals. The non-overlapping CIs ( vs. ) confirm that the difference in between the two generators, reflecting severe bootstrap upward bias: the interval lies entirely above the observed value, yielding zero empirical coverage of the point estimate. This is a known consequence of resampling instability when (see Appendix B.3); the two generators remain clearly separated.
| Model | RV | 95% CI | Spearman , -RV ) | KS | |||
|---|---|---|---|---|---|---|---|
| VAE (diagonal posterior) | 4.92 | 0.64 | 3.125 | 0.9835 | [4.83, 5.04] | -0.383 | 0.142 |
| Gaussian (structure-preserving) | 0.25 | 0.03 | 0.210 | 0.9999 | [0.33, 0.56] | -0.427 | 0.000 |
Table 3: Complete dependence fidelity diagnostics on Fashion-MNIST ( , PCA dimensions, ). Bootstrap 95% CIs based on resamples. Spearman correlation and KS test compare and across 200 random PC subsets of size 20. Note: † Bootstrap CI for the Gaussian baseline reflects right-skew of the bootstrap distribution at near-zero ; see Appendix B.3 for discussion.
| Metric | Value | Metric | Value |
| 49.35 | Eigengap | 279.87 | |
| 3.69 | 0.18 | ||
| RV-coefficient | 0.9929 | 0.0071 | |
| Median KS | 0.26 | KS -value | 0.581 |
| Spearman | |||
| Subspace diagnostics | |||
Table 4: Dependence fidelity diagnostics: Alzheimer’s dataset ( , genes, ). All values should be interpreted as qualitative directional estimates given the small sample size. Bootstrap CI is not reported as it is unreliable in this regime; see text for discussion.
Figure 13: Distribution of KS distances across genes (Alzheimer’s dataset, ). Most KS values are small, indicating approximate marginal preservation. This marginal similarity does not preclude covariance-level discrepancy, as shown in Figure 14 .
Figure 14: Covariance difference matrix for the Alzheimer’s dataset. Despite similar marginal KS profiles, visible off-diagonal residuals indicate that dependence structure is not preserved by the marginal-only Gaussian generator. Given , these patterns should be interpreted as qualitative directional evidence rather than precise numerical estimates.
Figure 15: Regression coefficients estimated from real versus synthetic Alzheimer’s data. Deviations from the identity reflect sensitivity to dependence structure, consistent with Theorem 2, though numerical magnitudes should be interpreted cautiously given the small sample size.
Figure 16: PCA subspace angle between real and synthetic gene expression data as a function of subspace dimension . Despite similar marginal distributions (Figure 13 ), the leading principal subspaces diverge substantially, consistent with the covariance distortion shown in Figure 14 and the instability predicted by Theorem 3 when exceeds the eigengap.
Figure 17: versus across 100 random gene subsets of size 20 (Alzheimer’s dataset, , ). Spearman ( ), KS . The near-zero correlation indicates that and are effectively decoupled in this small-sample regime, in contrast to the concordant behaviour observed in Appendix B.2 (Fashion-MNIST, Spearman ) and Appendix B.6 (TCGA-BRCA, for Splatter). The decoupling reflects estimation noise at rather than a structural property of the metrics.
Figure 18: Dependence collapse in a diagonal-covariance VAE with . Left two panels: Covariance matrices of the true distribution and the diagonal approximation . The off-diagonal structure present in is eliminated by the independence assumption, yielding . Right panel: Samples from (correlated, blue) and (independent, red) with population regression lines overlaid. Despite identical marginals , the regression slope collapses from to under the diagonal constraint, illustrating inferential instability consistent with Theorem 2 .
| Splatter ( ) | Poisson thinning ( ) | |||
| DK bound | DK bound | |||
| 1 | 0.9985 | — | 0.2099 | 1.5242 |
| 2 | 0.9978 | — | 0.4371 | — |
| 3 | 0.9943 | — | 0.4085 | — |
| 5 | 1.0000 | — | 0.8749 | — |
| 10 | 0.9983 | — | 0.9579 | — |
Table 5: PCA subspace angles between real and synthetic TCGA-BRCA data across subspace dimension . The Theorem 3 bound uses the local eigengap . For Splatter all bounds are vacuous ( , shown as “—”); for Poisson thinning the bound are vacuous; stability is confirmed empirically. Both share the same real covariance ( , , ).
| Model | RV | 95% CI | SE | Spearman | KS | |||
|---|---|---|---|---|---|---|---|---|
| Splatter | 249.79 | 2.05 | 26.58 | 0.299 | [237.10, 265.62] | 7.28 | 0.748 | 0.906 |
| Poisson thinning | 88.35 | 1.53 | 6.54 | 0.961 | [82.56, 107.78] | 6.60 | 0.195 | 0.699 |
Table 6: Extended dependence fidelity diagnostics for TCGA-BRCA ( , HVGs, ). Bootstrap 95% CI based on resamples. Spearman correlation and KS test compare and across 100 random gene subsets of size 30, where is the normalized (correlation-matrix) form of the divergence.
Figure 19: Distribution of KS distances across 100 HVGs comparing real TCGA-BRCA data against Splatter-generated (left) and Poisson-thinned (right) synthetic data. Both simulators exhibit broadly similar KS profiles, confirming approximate marginal fidelity in both cases. Marginal diagnostics alone cannot distinguish the two simulators, despite their substantially different covariance divergences ( vs. ).
Figure 20: Heatmaps of the covariance difference matrix for Splatter (top, range ) and Poisson thinning (bottom, range ). Despite similar marginal KS distances, Splatter exhibits substantially larger and more structured covariance residuals, reflecting the absence of joint modelling.
Figure 21: Population regression coefficients (gene 1 on genes 2–10) estimated from real vs. synthetic TCGA-BRCA data. Splatter (top) produces coefficients with near-zero correlation to the real values and frequent sign reversals, consistent with . Poisson thinning (bottom) tracks the real coefficients closely along the identity line with few sign flips, consistent with and Theorem 2 .
Figure 22: PCA subspace angles as a function of subspace dimension for Splatter (blue, ) and Poisson thinning (green, ) on TCGA-BRCA ( , HVGs). Splatter angles range from to across all , indicating near-complete subspace misalignment. Poisson thinning remains below for all and satisfies the Theorem 3 Davis–Kahan bound at ( ). The degradation at higher in both cases reflects decreasing local eigengaps, as predicted by Theorem 3 .
Figure 23: versus across 100 random gene subsets of size 30 (TCGA-BRCA, Splatter, , ). Spearman ( ), KS . The strong positive concordance and indistinguishable distributions after standardisation confirm that both metrics detect the same widespread covariance failure in the large-divergence unstable regime ( ). This is the strongest concordance result across all datasets in this study.
Figure 25: Comparison of three evaluation metrics, median KS distance, , and MMD (Gaussian kernel, median heuristic bandwidth [ Gretton et al., 2012 ] ), for Splatter and Poisson thinning on TCGA-BRCA ( , HVGs). The dashed red line marks the instability threshold of from Theorem 3. KS distances are identical across generators ( each). MMD shows a modest separation ( vs. ) but provides no actionable stability criterion. Only places Splatter clearly in the unstable regime ( ) and Poisson thinning in the closer stable regime ( ), directly predicting the downstream consequences documented in Tables 6 and 7 .
| Marginal | Covariance-level | Downstream | |||||
| Dataset | Model | Median KS | Regime | RV | Sign flips | ||
| Fashion-MNIST | VAE (diagonal) | 0.090 | 0.64 | stable | 0.984 | 0.194 | many |
| Fashion-MNIST | Gaussian ‡ | 0.020 | 0.03 | stable | 1.000 | 0.005 | none |
| TCGA-BRCA | Splatter ∗ | 0.127 | 2.05 | unstable | 0.299 | 0.999 | many |
| TCGA-BRCA | Poisson thinning ∗ | 0.110 | 1.53 | unstable | 0.961 | 0.210 | few |
| Alzheimer’s † | Gaussian | 0.260 | 0.18 | stable | 0.993 | 0.070 | — |
Table 7: Empirical summary across all datasets and generator comparisons. Columns are organised by diagnostic depth: marginal fidelity (KS), covariance divergence ( , RV), and downstream instability (subspace angle at , regression sign flips). Alzheimer’s values are qualitative only ( ); sign flips are not reported because regression magnitudes are unreliable at this sample size (see Appendix B.3), not because reversals are absent. Figure 14 shows substantial coefficient divergence. Gaussian baseline preserves covariance by construction. Both Splatter and Poisson thinning are in unstable regime ( ); DK bound is vacuous. Sign flips: number of reversed regression coefficients out of 9. For Fashion-MNIST, PC1 regressed on PC2–PC10; for TCGA-BRCA, gene 1 regressed on genes 2–10. Here ‘many’ indicates reversals; ‘few’ indicates . For Regime: stable if ( ) and unstable if ( ).
| Theorem | Claim | Empirical evidence | Status |
|---|---|---|---|
| Theorem 1 | Marginal fidelity does not imply dependence fidelity; can be large while KS | KS distances are similar across all generator pairs within each dataset, yet and RV differ by factors of – | ✓ |
| Theorem 2 | Covariance divergence induces regression instability bounded by | Regression sign flips present for all generators with ; absent for all generators with | ✓ |
| Theorem 3 | at when | Bound is vacuous for Poisson thinning and Alzheimer’s; VAE bound confirmed at (observed 0.194 < 1.278). Empirical stability confirmed across all datasets. | ✓ |
| Remark 1 | does not capture tail or nonlinear dependence | t-copula example (Appendix B.1) shows joint extreme-event failures invisible to by design | ✓ |
Table 8: Theoretical claims and empirical confirmation. Each theorem is listed with the empirical evidence that confirms it across datasets. “Confirmed” means the predicted direction holds in every comparison examined. The thresholds in the Theorem 2 row (0.5 and 0.1) are empirical observations from this study, not theoretically derived cutoffs.
| Generator | Median KS | MMD | |
|---|---|---|---|
| Poisson thinning | 0.11 | 0.09 | 1.53 |
| Splatter | 0.11 | 0.338 | 2.053 |
| Instability threshold | — | — | 1.000 |
Table 9: KS, MMD, and across generators on TCGA-BRCA ( , HVGs). The dashed threshold marks (Theorem 3). Only provides an actionable stability criterion.