Finite-Sample Sensitivity of Kernel Metrics for Generative Evaluation

Tomasz Radzikowski; Mateusz Modrzejewski

cs.LGOct 25, 2026

Finite-Sample Sensitivity of Kernel Metrics for Generative Evaluation

Authors: Tomasz Radzikowski, Mateusz Modrzejewski

Organizations: Institute of Computer Science Warsaw University of Technology Warsaw, Poland

Abstract

Kernel distances such as the Kernel Inception Distance (KID) and the Kernel Audio Distance (KAD) compare generated and reference samples in an embedding space, and model comparisons rest on their values. Each ships with one fixed kernel, and a reported value does not say which changes that kernel detects at the sample size used. We run each default as a permutation two-sample test on real embeddings of symbolic music, audio and images, comparing a sample with an independent sample changed by a mean shift, an isotropic contraction, or a rotation that keeps the mean and the covariance trace and is matched to the shift in transport distance. At 780 samples per side, both defaults rarely detect the rotations in five evaluation spaces, while they detect the shifts in every run. The median-bandwidth Gaussian kernel of KAD detects the contraction and regains power against rotations on image embeddings with an order of magnitude more samples. The cubic kernel of KID contains second- and third-moment information but misses the contraction in most runs, and our experiments do not establish a high-power rotation regime for it over the tested sample sizes. Narrower Gaussian bandwidths restore sensitivity in music and audio, an $\ell_1$ Laplace kernel does so for images, and aggregated tests combine both at nominal level. We recommend that a reported kernel distance state its sample budget and the changes its configuration detects at that budget.

Figures & tables

Figure 1: Gaussian-kernel MMD rejection rates for a rotation that keeps the mean and the covariance trace (left) and for a mean shift (right), at

n{=}780

per side and level

0.05

, against the bandwidth as a multiple of the median squared distance. Hollow markers show the median heuristic. Bars are descriptive Wilson

95\%

intervals over

20

draws, and over

50

for the Inception point at

0.05

(

26/50

). The shift has no greater transport distance than the rotation.

Alternative	Spaces	median	cubic	window	MMDAgg	Fréchet
mean shift, $s{=}0.1$	music, audio	20/20	20/20	20/20	–	5–20/20
	Inception	20/20	20/20	20/20	20/20	13/20
contraction, $s{=}0.1$	music, audio	20/20	0–3/20	–	–	5–20/20
	Inception	20/20	4/20	20/20	–	13/20
rotation, $s{=}0.1$	music, audio	1–4/20	0–2/20	20/20	18/20–50/50	10–20/20
	Inception	3/20	3/20	26/50	20/20	16/20

Table 1: Rejections at

n{=}780

per side. Music and audio cells give the range over CLaMP 2, VGGish and PANNs. The window is the Gaussian kernel at

0.05\times

the median squared distance. A dash marks a cell the battery did not run. Fréchet uses

n{=}390

in the 2048-dimensional spaces (PANNs, Inception).

Figure 2: Rejection rates against samples per side on Inception features of CIFAR-10, with descriptive Wilson

95\%

intervals. Draws per point:

20

up to

5{,}000

10

10{,}000

5

20{,}000

. (a) A rotation matched to a mean shift in transport distance. (b) DDPM samples against CIFAR-10 with each side centered once (Section 3 ). Panel (b) is descriptive: centering each side before permutation treats the observed and permuted labels differently, so its rates do not estimate calibrated power. MMDAgg was not run on (a) at

20{,}000

. The dotted line marks the nominal level

0.05

Question	Quantity	Disclosure	Limitation
Do the distributions differ?	a two-sample test chosen in advance (MMDAgg at package defaults)	kernel collection, preprocessing, samples per side, calibration, $p$ -value	non-rejection is not equivalence; rejection does not measure quality
Do the locations differ?	mean displacement in embedding units	normalization, uncertainty	says nothing about spread or support
How do covariances differ?	Bures term on shrinkage covariances, or a normalized covariance difference	estimator, shrinkage, rank, uncertainty	hard to estimate when $n<d$
Does spectral diversity differ?	effective rank and $n_{90}$ against the reference	matched sample sizes, reference definition	misses rotations and many coverage changes
Are training samples copied?	a dedicated copying test against held-out training data	reference and training sets, statistic, threshold, calibration	no detection is not proof of no memorization

Table 2: A reporting protocol. Each row is a separate question with its own quantity and calibration.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Distortion	Space	$.01$	$.02$	$.05$	$.1$	$.2$	$.5$	$1$	$2$	med
rotation $s{=}.1$	C2	1	6	20	20	15	5	2	1	2
	VG	4	17	20	18	11	4	4	3	4
	PN	7	20	20	20	10	4	1	0	1
mean shift $s{=}.1$	C2	0	4	20	20	20	20	20	20	20
	VG	4	8	20	20	20	20	20	20	20
	PN	4	16	20	20	20	20	20	20	20

Appendix

Table 3: Cells give rejections out of

20

draws by Gaussian bandwidth, in multiples of the median

d^{2}

; “med” is the classical median heuristic

\sigma{=}\mathrm{med}(d)

. C2 is CLaMP 2, VG is VGGish, and PN is PANNs. The main text quotes the med column of this table.

	CLaMP 2	VGGish	PANNs
cubic (KID): rotation	1/20	2/20	0/20
cubic: contraction	4/20	2/20	2/20
cubic: mean shift	20/20	20/20	20/20
cubic: control	0/20	0/20	1/20
MMDAgg (defaults): rotation	18/20	50/50	20/20
MMDAgg, Gaussian-only: rotation	18/20	42/50	20/20

Appendix

Table 4: Fixed-kernel arms and aggregation at weak strength

s{=}0.1

unless noted, with

20

draws per cell and

50

where two counts appear or where marked. Median-baseline rotation cells with two counts report seeds

0

;

1

\dagger

: a second run of this cell; the real-comparisons battery gives

25/50

for the same configuration (§ 7 ).

$\gamma$	$n$	$.02$	$.05$	$.1$	$.5$	$1$	$2$	MMDAgg
$0.05$	$500$	1.00	1.00	.92	.24	.10	.06	.95
	$2000$	1.00	1.00	1.00	.96	.46	.24	1.00
	$6000$	1.00	1.00	1.00	1.00	1.00	.72	1.00
$0.15$	$500$	1.00	1.00	1.00	1.00	.96	.66	1.00
	$2000$	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	$6000$	1.00	1.00	1.00	1.00	1.00	1.00	1.00

Appendix

Table 5: Synthetic power on 768d Gaussians with equal means and

25\%

of dimensions rescaled by

1{+}\gamma

, with

50

repetitions per cell and

20

for MMDAgg. The empirical level of the “

1

” arm at

\gamma{=}0

0.06

. Headers are multiples of the median squared distance, so column “

1

” (

\sigma^{2}{=}\mathrm{med}\,d^{2}

) is the synthetic proxy for the classical heuristic, which the real-data tables report separately as

\sigma{=}\mathrm{med}(d)

Space	Variant	median	cubic	window	MMDAgg
CLaMP 2	control	1/20	1/20	2/20	1/20
( $f{=}0.10$ )	clean, mean-matched	4/50	2/50	34/50	24/50
	resampled, mean-matched	15/50	4/50	50/50	50/50
	duplicates only	6/50	4/50	50/50	50/50
VGGish	control	1/20	1/20	1/20	1/20
( $f{=}0.084$ )	clean, mean-matched	1/20	0/20	2/20	0/20

Appendix

Table 6: Mode drop battery with

20

draws,

50

where a count fell between

3

and

17

; half of the largest of

k{=}10

k

-means clusters is removed. “clean” replaces deleted points from a disjoint reserve of the corpus; “resampled” replaces them by drawing from the remainder, which injects duplicates at the measured rate

f

; “duplicates only” applies that rate with no mode removed. “mean-matched” re-subtracts the induced mean displacement.

Space	Alternative	median	cubic	window	MMDAgg
CLaMP 2	control	1/20	0/20	2/20	1/20
	rank-2 rotation	7/50	2/50	39/50	36/50
	rank-5 rotation	16/50	3/50	50/50	50/50
	rank-1 inflation	50/50	8/50	50/50	50/50
	rank-1 deflation	50/50	5/50	50/50	50/50
	random direction	2/20	1/20	1/20	1/20

Appendix

Table 7: Low-rank alternatives at

\gamma{=}0.3

, with

50

draws where a count fell between

3

and

17

and

20

otherwise; each construction keeps the mean of the transformed sample (

\|\Delta\hat{\mu}\|/r=0.000

between that sample before and after the transformation). Rank-2 and rank-5 are rotations confined to the leading principal components; inflation and deflation are rank-one scalings of the first component; “random direction” applies the same scaling along a random direction; contamination displaces

2\%

of points.

proj.	$d_{\mathrm{eff}}$	$.02$	$.05$	$.1$	median	MMDAgg	shift $.05$	ctrl
PCA-16	$12.8$	18	16	11	4	18	19	$\leq 3$
PCA-32	$23.0$	18	17	14	3	15	20	$\leq 3$
PCA-64	$40.2$	13	16	13	3	12	20	$\leq 3$
PCA-128	$69.0$	7	16	11	3	9	20	$\leq 3$
PCA-256	$113.1$	4	13	12	3	10	20	$\leq 3$
PCA-512	$167.9$	4	13	10	3	15	20	$\leq 3$

Appendix

Table 8: Rotation power along the within-space

d_{\mathrm{eff}}

ladder on Inception/CIFAR-10 at

s{=}0.1

, with

20

draws unless marked. Columns

.02

–

.1

give the Gaussian bandwidth as a multiple of the median

d^{2}

; “shift

.05

” is the mean-shift arm at the

0.05\times

bandwidth; “ctrl” is the maximum control-cell count across the row’s arms.

	Plain	Recentered
Pair	all four configs	median	cubic	window / MMDAgg
GPT-2 vs. training corpus	$20/20$	$25/50$	$0/50$	$50/50$
FolkRNN vs. training corpus	$20/20$	$20/20$	$20/20$	$20/20$
$T{=}0.85$ vs. $T{=}1.0$	$20/20$	$20/20$	$20/20$	$20/20$
$T{=}0.7$ vs. $T{=}1.0$	$20/20$	$20/20$	$20/20$	$20/20$
text2midi vs. MidiCaps (its training set)	$20/20$	$20/20$	$20/20$	$20/20$

Appendix

Table 9: Real comparisons through the full battery at

n{=}780

per side with

20

draws; cells with mid-range counts were increased to

50

draws. “Recentered” centers each sample separately on its own mean before permutation. This changes the raw higher moments as well as the means, and the recentered counts are descriptive (Section 3 ). Controls are independent splits of the corpus. text2midi ( Bhandari et al., 2025 ) is sampled on

2{,}000

held-out MidiCaps captions; FolkRNN is the model of Sturm et al. (2016) . Audio rows: MusicGen ( Copet et al., 2023 ) and Stable Audio ( Evans et al., 2026 ) ,

2{,}000

ten-second clips each against

2{,}000

FMA clips, both sides through one provenance pipeline.

\dagger

: for the Aria row ( Bradshaw et al., 2025 ) the plain arm is

20/20

at the median, cubic and MMDAgg but

0/20

at the window bandwidth.

Space	Copies	Narrow	Median	FMD	MMDAgg
VGGish	$1\%$	$-0.7$	$+0.1$	$-0.4$	$2/20$
	$5\%$	$-3.8$	$-0.7$	$-0.8$	$0/20$
	$10\%$	$-8.0$	$-0.9$	$-1.6$	$0/20$
PANNs	$1\%$	$-0.6$	$-0.2$	$-0.5$	$0/20$
	$5\%$	$-3.3$	$-0.5$	$-1.0$	$0/20$
	$10\%$	$-6.8$	$-0.9$	$-1.9$	$0/20$

Appendix

Table 10: Cells give the mean response to duplicate injection in control-normalized units (

z

against independent splits;

20

draws per cell, exact copies; perturbed copies agree within

\pm 0.5

) and MMDAgg rejections (

5/360

pooled across all cells and both variants,

{\approx}1.4\%

). Copying lowers cross-set distances, so responsive statistics go negative; the text discusses magnitudes

|z|

. Narrow is the Gaussian kernel at

0.05\times

median

d^{2}

, Median the median heuristic, and FMD the Fréchet distance on embeddings.

$T$	eff ratio	$n_{90}$ ratio	tr ratio	Vendi ratio
$0.50$	$0.79\pm.01$	$0.80\pm.01$	$0.84\pm.00$	$0.75\pm.01$
$0.70$	$0.88\pm.00$	$0.87\pm.01$	$0.95\pm.01$	$0.83\pm.01$
$0.85$	$0.92\pm.01$	$0.89\pm.01$	$1.02\pm.01$	$0.90\pm.01$
$1.00$	$1.08\pm.01$	$1.07\pm.01$	$1.00\pm.01$	$0.99\pm.02$
$1.20$	$1.04\pm.01$	$0.97\pm.01$	$1.21\pm.00$	$1.15\pm.01$

Appendix

Table 11: Temperature dose-response of a GPT-2-class model on the folk corpus. Cells are ratios to the training reference, means

\pm

s.d. over five random

5{,}000

-sample subsets of each set.

\mathrm{eff}

is the exponentiated spectral entropy of the covariance,

n_{90}

the number of components covering

90\%

of the variance,

\mathrm{tr}

the trace, and Vendi the score of Friedman & Dieng (2023) with a cosine kernel.

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 313 papers

MIND: Monge Inception Distance for Generative Models Evaluation
May 7, 2026Quentin Berthet, Yu-Han Wu, Clement Crepy +3Image Generation Evaluation Sliced Wasserstein Distance
Google DeepMind · LPSM, Sorbonne Université
Consistent Distributed Ranking of Generative Models via Kernel Distances
Oct 18, 2023Zixiao Wang, Farzan Farnia, Zhenghao Lin +2Kernel Methods Image Generation Evaluation
What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation
Aug 25, 2026Hao ChenGenerative AI Evaluation Image Generation Evaluation
Department of Statistics University of California, Davis

May 7, 2026J/K move · Enter open · S save

MIND: Monge Inception Distance for Generative Models Evaluation

Quentin Berthet, Yu-Han Wu, Clement Crepy, Romuald Elie+2

Google DeepMind · LPSM, Sorbonne Université

Image Generation Evaluation Sliced Wasserstein Distance Generative AI Evaluation

Open paper AskPDF arXiv

We propose the Monge Inception Distance (MIND), a metric for evaluating generative models that addresses key limitations of the widely adopted Fréchet Inception Distance (FID). The MIND metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. This approach circumvents the estimation of high-dimensional means and covariance matrices, which underlie FID's poor sample complexity and vulnerability to adversarial attacks. We empirically demonstrate three primary advantages: (i) it is more sample-efficient by one order of magnitude, (ii) it is faster to compute by two orders of magnitude, (iii) it is more robust to adversarial attacks such as moment-matching. We show that MIND with 5k samples can replace the evaluation performance of FID with 50k samples, providing high correlation with this standard benchmark and superior discriminative performance. We further demonstrate that even smaller sample sizes (e.g., 1k or 2k) remain highly informative for rapid model iteration.

Abstract

Generative AI Evaluation

Explore similar work

MIND: Monge Inception Distance for Generative Models Evaluation

Consistent Distributed Ranking of Generative Models via Kernel Distances

What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation