M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Models

Sathiyamohan Nishankar; Pubudu Sanjeewani; Asanka Perera

cs.CVSep 27, 2026

M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Models

Authors: Sathiyamohan Nishankar, Pubudu Sanjeewani, Asanka Perera

Organizations: Faculty of Engineering, University of Peradeniya, Sri Lanka · School of Computing Technologies, RMIT University, Melbourne, Australia · School of Engineering & Digital Technologies, University of Southern Queensland, Brisbane, Australia · School of Engineering & Technologies, UNSW, Canberra, Australia

Abstract

Quantitative evaluation of generative models for radiology remains challenging. Clinically relevant structures are often small and infrequent, feature spaces learned from natural images may represent them poorly, and a single summary score cannot distinguish limited fidelity from limited diversity. This study proposes the Medical Multi-axis Maximum Mean Discrepancy score (M3-Score), an evaluation framework based on RadioDINO-s16, a frozen vision transformer pretrained on radiology images. M3-Score reports three complementary axes computed at pre-specified encoder depths: \emph{fidelity}, measured by an unbiased multi-bandwidth radial basis function (RBF) MMD $^2$ at the final block; \emph{memorization}, measured by nearest-neighbor distances at 75% depth; and \emph{coverage}, defined as the fraction of real images with a generated neighbor within their $k$ -nearest-neighbor radius at 33% depth. Reference sets are sampled across subjects to limit the influence of correlated slices. On BraTS brain MRI, the fidelity axis ordered five comparison sets of increasing severity (Spearman $ρ= 1.00$ ), and a subject-disjoint real set yielded $\mathrm{MMD}^2 = 0$ (permutation $p = 1$ ). An unconditional denoising diffusion probabilistic model achieved $\mathrm{MMD}^2 = 0.073$ (95% confidence interval $[0.071, 0.080]$ ) but covered only 38% of the real distribution. Under progressive mode dropping, $k$ -NN manifold recall increased at all twelve encoder blocks, whereas the proposed coverage estimator decreased monotonically ( $ρ= -1.00$ ). RadioDINO-s16 features separated real brain MRI from generated samples with a ROC-AUC of 0.819, compared with 0.555 for InceptionV3 and 0.582 for CLIP. Across a twentyfold range of sample sizes, the mean M3 value varied by a factor of 1.05, compared with 2.52 for the Fréchet Inception Distance.

Figures & tables

Category	Metric	Dir.	Measured property
Distributional	FID [ 20 ]	$\downarrow$	Fréchet distance between the feature distributions of real and generated images.
	KID [ 7 ]	$\downarrow$	Unbiased kernel-based distance between distributions, with improved small-sample behavior.
	IS [ 39 ]	$\uparrow$	Classifier-based measure of sample quality and diversity.
Perceptual	SSIM [ 43 ]	$\uparrow$	Structural similarity based on luminance, contrast, and structure.
	PSNR	$\uparrow$	Pixel-level reconstruction quality expressed as peak signal-to-noise ratio.
	MS-SSIM	$\uparrow$	Structural similarity computed at multiple scales.

Table 1: Commonly used evaluation metrics for generative medical image models and the proposed M3-Score. The column Dir. indicates whether lower (

\downarrow

) or higher (

\uparrow

) values are preferable. M3-Score reports fidelity, memorization, and coverage as separate axes, each computed at a pre-specified encoder block.

Figure 1: Overview of the $M^{3}$ -Score framework. Real and generated images are preprocessed and encoded by a frozen RadioDINO-s16 encoder. The [CLS] token features from three pre-specified blocks are used to compute fidelity (L12), memorization (L9), and coverage (L4). The dashed path denotes the layer-redundancy analysis based on centered kernel alignment (CKA), which is used solely for analysis and does not contribute to the reported axes.

		Fidelity (L12)				Memorization (L9)	Coverage (L4)
Comparison set	Content	$\widehat{\mathrm{MMD}}^{2}$	95% CI	$p$	$Z$	Rate	Cov.
Held-out real	real vs. real	$0$	$[0.0010,0.0026]^{\ddagger}$	$1$	$-0.6$	$0.478$	$0.970$
DDPM	in-domain diffusion	$0.0732$	$[0.0705,0.0801]$	$0.002$	$342.4$	$0.010$	$0.384$
WDM-3D (MRI)	weaker same-modality generator	$0.2159$	$[0.2102,0.2250]$	$0.002$	$696.9$	$0$	$0.002$
LIDC CT	cross-modality shift	$0.3984$	$[0.3909,0.4117]$	$0.002$	$718.4$	$0$	$0$
Retinal	far-domain shift	$0.4585$	$[0.4525,0.4678]$	$0.002$	$805.3$	$0$	$0$

Table 2: Three-axis evaluation of five comparison sets against a reference of

N=500

BraTS slices from

500

subjects. Rows are ordered by expected severity. Fidelity is reported with a permutation test (

B=500

) and a bootstrap

95\%

CI (

500

resamples). ‡ Not interpreted, because the ordinary bootstrap is inconsistent for a degenerate U-statistic under

H_{0}

Figure 2: Three-axis evaluation of five comparison sets of increasing expected severity against a reference of

N=500

BraTS slices from

500

subjects. The leftmost (hatched) bar denotes the subject-disjoint held-out real set. (a) Fidelity with bootstrap

95\%

intervals; the interval of the held-out real set is not interpreted (see Table 2 ). (b) Coverage: the DDPM has the lowest fidelity distance among the generated sets but covers approximately

38\%

of the real distribution. (c) Memorization: rates are close to zero for all generated sets and high for the held-out real set.

Figure 3: Representative axial BraTS slices: real (top) and generated by the unconditional DDPM (bottom). Both sets exhibit plausible tissue boundaries, ventricles, and FLAIR-hyperintense lesions, and the two sets are difficult to distinguish by visual inspection. Images are shown at the $224\times 224$ resolution used as encoder input.

Metric	Expected	Spearman $\rho$
Metric	Expected	Noise	Blur
M3-Score	$\uparrow$	$1.000$	$1.000$
FID (Inception)	$\uparrow$	$1.000$	$1.000$
KID (Inception)	$\uparrow$	$1.000$	$1.000$
CMMD (CLIP)	$\uparrow$	$1.000$	$1.000$
SSIM	$\downarrow$	$-1.000$	$-1.000$

Table 3: Spearman rank correlation

\rho

between degradation level and metric value (seven noise levels; five blur radii).

\uparrow

: the metric is expected to increase with degradation;

\downarrow

: the metric is expected to decrease.

Figure 4: Gaussian noise sweep ( $\sigma\in[0,0.5]$ ; $N=500$ BraTS images). Top left: distributional metrics; top right: perceptual metrics; bottom left: all metrics rescaled to $[0,1]$ ; bottom right: $|\mathrm{Spearman}\ \rho|$ for M3, FID, KID, CMMD, $\alpha$ -Precision, and $\beta$ -Recall (labeled PRECISION and RECALL).

Figure 5: Gaussian blur sweep ( $r\in\{0,1,2,3,4\}$ pixels; $N=500$ BraTS images). Top left: distributional metrics; top right: perceptual metrics; bottom left: all metrics rescaled to $[0,1]$ ; bottom right: $|\mathrm{Spearman}\ \rho|$ for M3, FID, KID, CMMD, $\alpha$ -Precision, and $\beta$ -Recall (labeled PRECISION and RECALL). $\alpha$ -Precision decreases to $0.006$ at $r=2$ , indicating that the BraTS manifold is strongly dependent on fine texture.

	Isolation Forest fitted in
Per-image distance	RadioDINO-s16	InceptionV3
RadioDINO-s16 score (M3)	$\mathbf{+0.736}\,[+0.69,+0.77]$	$+0.144\,[+0.06,+0.23]$
InceptionV3 score	$+0.155\,[+0.07,+0.24]$	$\mathbf{+0.649}\,[+0.60,+0.70]$
Pixel MSE	$+0.288\,[+0.21,+0.37]$	$+0.364\,[+0.29,+0.44]$

Table 4: Spearman correlation between per-image distances and Isolation Forest anomaly scores, with the detector fitted in each feature space (

N=500

generated images;

95\%

CI by Fisher

z

-transformation). Bold entries denote cases in which the distance and the detector share a feature space.

	Discriminative		Saturated
Feature space	DDPM	WDM-3D	LIDC CT	Retinal
RadioDINO-s16 L12	$\mathbf{0.819}$	$0.986$	$1.000$	$1.000$
InceptionV3 2048-d (FID)	$0.555$	$0.967$	$1.000$	$0.999$
CLIP ViT-L/14 (CMMD)	$0.582$	$\mathbf{0.999}$	$0.998$	$0.999$

Table 5: Per-image OOD discrimination by feature space. ROC-AUC for separating a subject-disjoint held-out real set from each comparison set, using the distance to the real-reference centroid on normalized features.

N=500

per set; the reference contains

500

slices from

500

subjects.

Figure 6: Per-image OOD discrimination using a centroid-distance score in three feature spaces. The positive class is a subject-disjoint held-out real set. (a) ROC curves for real images versus DDPM samples: RadioDINO-s16 attains AUC

=0.819

, compared with

0.555

for InceptionV3 (FID) and

0.582

for CLIP (CMMD). (b) AUC for each comparison set. The cross-modality and far-domain sets (shaded) saturate at

\geq 0.998

in all feature spaces.

Figure 7: Examples of the comparison sets evaluated against real BraTS MRI, with two examples in each column. From left: real MRI; DDPM samples; WDM-3D samples, which are of visibly lower quality but remain brain MRI; and chest CT slices generated by the WDM-3D model trained on LIDC-IDRI, which differ in both modality and anatomy. M3 and FID place the CT set furthest from real brain MRI, whereas CMMD places it closer than the WDM-3D set (Table 6 ).

Comparison vs. BraTS real	M3 (fidelity) [95% CI]	FID	CMMD [95% CI]
DDPM (in-domain)	$0.073\,[0.071,0.080]$	$68.4$	$0.104\,[0.102,0.111]$
WDM-3D (weaker same-modality)	$0.216\,[0.210,0.225]$	$173.4$	$0.737\,[0.719,0.754]$
LIDC CT (cross-modality)	$\mathbf{0.398}\,[0.391,0.412]$	$319.9$	$0.664\,[0.652,0.682]$
$\Delta$ (CT $-$ WDM-3D)	$\mathbf{+0.182}$	$+146.5$	$-0.073$

Table 6: Distances from real BraTS images (

N=500

) for three comparison sets, with bootstrap

95\%

CIs for M3 and CMMD (

500

resamples). Higher values indicate a larger distance, and the expected order is DDPM

<

WDM-3D

<

CT. M3 and FID follow this order, whereas CMMD reverses the last two sets. FID is reported as a point estimate.

Injection rate	Mem. rate (L9)	Fidelity MMD 2 (L12)	Coverage (L4)
$0$	$0.010$	$0.0732$	$0.384$
$0.1$	$0.108$	$0.0597$	$0.674$
$0.2$	$0.210$	$0.0470$	$0.810$
$0.3$	$0.308$	$0.0346$	$0.906$
$0.5$	$0.506$	$0.0173$	$0.982$

Table 7: Near-duplicate injection on the standard reference (

N=500

from

500

subjects). A known fraction of the DDPM set is replaced by jittered copies of real images. The memorization rate follows the injected fraction (mean absolute error

0.008

), while fidelity decreases and coverage increases.

	$k$ -NN recall [ 29 ]	Coverage [ 34 ] , Eq. ( 6 )
Block	$\rho(\downarrow,\text{recall})$	$\rho(\downarrow,\text{cov.})$	Range	Criterion
L1	$+1.00$	$-1.00$	$0.376$	✓
L2	$+0.70$	$-1.00$	$0.196$	✓
L3	$+0.90$	$-1.00$	$0.228$	✓
L4	$\mathbf{+0.90}$	$\mathbf{-1.00}$	$\mathbf{0.232}$	✓
L5	$+1.00$	$-1.00$	$0.192$	✓

Table 8: Comparison of mode dropping responses between two coverage estimators across all twelve encoder blocks. We report the Spearman rank correlation (

\rho

) between the mode drop fraction (

\downarrow

) and the respective metric value. A theoretically valid coverage estimator must exhibit a strongly negative

\rho

. Range denotes the absolute difference between the maximum and minimum values across all perturbation levels. Our pre-specified extraction depth (L4) is highlighted in bold. The evaluation utilized

N=500

real slices from 500 distinct subjects in a nested design using

k=5

nearest neighbors.

Figure 8: Mode-drop response at all twelve encoder blocks (gray lines), with the pre-specified depth L4 highlighted. (a)

k

-NN manifold recall increases as diversity is removed, because its hyperspheres are fitted to the subsampled generated set. (b) Real-radius coverage decreases monotonically at every depth.

$\tau$	$M$	Retained layers	$S_{\text{RG}}$	$S_{\text{RR}}$
$0.70$	$3$	$\{2,5,\mathbf{12}\}$	$0.0915$	$1.2{\times}10^{-4}$
$0.80$	$4$	$\{1,2,6,\mathbf{12}\}$	$0.0866$	$0.6{\times}10^{-4}$
$0.85$	$5$	$\{1,2,4,7,\mathbf{12}\}$	$0.0907$	$0.2{\times}10^{-4}$
$0.90$	$6$	$\{1,2,4,6,8,\mathbf{12}\}$	$0.0879$	$0.4{\times}10^{-4}$
$0.92$	$6$	$\{1,2,4,6,8,\mathbf{12}\}$	$0.0879$	$0.4{\times}10^{-4}$
$0.95$	$7$	$\{1,2,4,5,7,9,\mathbf{12}\}$	$0.0900$	$0.7{\times}10^{-4}$

Table 9: CKA-based layer-redundancy analysis on the standard reference (

N=500

from

500

subjects).

M

is the number of retained layers.

S_{\text{RG}}

is the mean MMD 2 between the real and generated sets over the retained layers, and

S_{\text{RR}}

is the corresponding value between two subject-disjoint real sets. L12 (bold) is retained at every threshold.

Figure 9: Area-matched lesion specificity design applied to a real LGG slice. From left to right: the original image slice; the expert FLAIR abnormality mask highlighted in red; the lesion condition with Gaussian noise ( $\sigma=45$ ) injected strictly within the tumor region; and the control condition with identical noise injected within a mirrored, contralateral region of equal area highlighted in green. Because both regions share an identical shape and area, any observed differences in the metric response are purely attributable to the anatomical location of the perturbation.

$\sigma$	$M^{3}$ -Score (RadioDINO-s16, RBF)	InceptionV3, RBF	InceptionV3, polynomial (KID)
20	N/A	$1.13\ [0.85,1.32]$	$1.12\ [0.85,1.34]$
30	N/A	$1.09\ [0.88,1.26]$	$1.08\ [0.88,1.25]$
45	$\mathbf{1.71}\ [0.94,2.14]$	$1.10\ [0.92,1.26]$	$1.08\ [0.92,1.24]$
60	$\mathbf{1.57}\ [1.03,1.97]$	$1.10\ [0.95,1.24]$	$1.08\ [0.93,1.22]$
90	$\mathbf{1.39}\ [1.06,1.71]$	$1.08\ [0.96,1.20]$	$1.06\ [0.93,1.19]$

Table 10: Lesion-to-healthy specificity ratio

\kappa

with bootstrap 95% confidence intervals for the mirror control condition (

N=200

). The kernel-matched baseline (RBF computed on InceptionV3) differs from

M^{3}

-Score exclusively in its backbone architecture, whereas KID differs in both the backbone and the kernel. The “N/A” designation indicates a noise level falling below the metric’s detection threshold. Values for

\sigma=10

are excluded due to lack of signal.

Figure 10: Lesion-to-healthy specificity ratio $\kappa$ plotted as a function of the region-local noise level $\sigma$ using real LGG expert masks ( $N=200$ ). $M^{3}$ -Score (red and orange lines) attains $\kappa=1.4$ to $1.8$ for both the mirror and texture-matched controls. In contrast, the Inception-based KID (dark blue and gray lines) remains stagnant near 1.05. Shaded bands denote the bootstrap 95% confidence intervals. Note that $M^{3}$ -Score values falling below its lower detection threshold ( $\sigma<45$ ) are omitted.

Condition	$M^{3}$ -Score	FID	CMMD
A: Real vs. generated (baseline)	$0.0732$	$68.4$	$0.1040$
B: Real vs. masked real	$0.0440$	$212.7$	$0.3339$
C: Real vs. masked generated	$0.1192$	$249.0$	$0.4772$
Relative change (B $-$ A) / A	$-39.9\%$	$+211.0\%$	$+221.0\%$
Relative change (C $-$ A) / A	$+63.0\%$	$+264.2\%$	$+358.7\%$

Table 11: Impact of a central occlusion artifact (

64\times 64

zero-filled square) applied to the standard reference dataset (

N=500

across 500 subjects). Condition B masks the real images, while condition C masks the generated images. Relative changes are computed with respect to the unmasked baseline (condition A, corresponding to Tables 2 and 6 ).

	CV (%)			Mean
$N$	M3	FID	CMMD	M3	FID	CMMD
$25$	$12.99$	$\mathbf{8.98}$	$11.97$	$0.0734$	$169.5$	$0.1019$
$50$	$9.71$	$\mathbf{4.32}$	$6.57$	$0.0717$	$132.6$	$0.1003$
$100$	$6.73$	$\mathbf{3.63}$	$7.44$	$0.0750$	$112.0$	$0.1038$
$200$	$5.60$	$2.25$	$\mathbf{2.10}$	$0.0722$	$88.2$	$0.1001$
$500$	$1.27$	$1.25$	$\mathbf{1.17}$	$0.0732$	$67.3$	$0.1028$

Table 12: Sample-size dependence based on ten independent draws for each

N

, with real images resampled at the subject level. CV quantifies repeatability at fixed

N

(lower is better; the lowest value in each row is shown in bold). Mean quantifies comparability across

N

Explore similar work

MedForj: An open, large-scale foundational generative prior for high-resolution 3D brain MRI
Oct 29, 2025Samuel W. Remedios, Aaron Carass, Jerry L. Prince +1Medical Image Generation Diffusion Priors
Image Analysis and Communications Laboratory, Johns Hopkins University · Department of Neurology, Johns Hopkins Hospital
Tokenizer-Generator Coupling in Medical Image Generation
Aug 7, 2026Liam ChalcroftMedical Image Generation Visual Tokenizers
University College London London, United Kingdom
Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers
Jun 17, 2026Fabio De Sousa Ribeiro, Emma A. M. Stanley, Charles Jones +7Digitally Reconstructed Radiographs Medical Image Generation
Imperial College London · Causality in Healthcare AI Hub · Cleveland Clinic London +4

Oct 29, 2025, eess.IVJ/K move · Enter open · S save

MedForj: An open, large-scale foundational generative prior for high-resolution 3D brain MRI

Samuel W. Remedios, Aaron Carass, Jerry L. Prince, Blake E. Dewey

Image Analysis and Communications Laboratory, Johns Hopkins University · Department of Neurology, Johns Hopkins Hospital

Medical Image Generation Diffusion Priors

Open paper AskPDF arXiv

This work introduces MedForj, a suite of 3D foundational generative priors based on diffusion models. The MedForj models were trained on

72{,}659

1~mm isotropic 3D

T_1

-weighted MRI human brain image volumes from

38{,}174

subjects, drawn from a curated corpus of

80{,}675

volumes from

42{,}506

subjects spanning

38

publicly available datasets. These training images were manually inspected to exclude those with poor quality and excessive pathology, and otherwise were minimally processed. The models include six different diffusion training strategies: rectified flow, latent diffusion rectified flow, flow matching, velocity prediction, clean prediction, and noise prediction. Image samples produced by each of these models were compared to each other and against real, ground truth data under downstream segmentation distributions, FID, five inverse problems, and blind human inspection in an observer study. Flow matching was the strongest strategy overall, achieving the best inverse problem solving results at

28.80

~dB PSNR and

0.874

SSIM averaged over the five forward problems, the highest rate of reconstructions judged real by blind human raters at

72.6\%

, and the closest per-structure match to real segmented anatomy in a permutation test. It was not best everywhere: rectified flow produced the most convincing unconditional samples in the observer study and the best FID, and the latent rectified-flow model achieved the smallest joint distributional distance to real anatomy. No other strategy, however, performed consistently well across all four evaluations. We therefore recommend flow matching as the default MedForj prior, while releasing every strategy so that the choice can be revisited per application. All model weights and corresponding code are publicly available at https://github.com/piksl-research/medforj.

Abstract

Explore similar work

MedForj: An open, large-scale foundational generative prior for high-resolution 3D brain MRI

Tokenizer-Generator Coupling in Medical Image Generation

Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers