What Does Fréchet Distance Measure? A Directional Decomposition

Yunghee Lee; Jaeyeon Kim

cs.AIOct 4, 2026

What Does Fréchet Distance Measure? A Directional Decomposition

Authors: Yunghee Lee, Jaeyeon Kim

Organizations: Agency for Defense Development, South Korea · Harvard University

Abstract

The Fréchet distance is a de facto standard for evaluating generative models across domains, appearing as FID for images and FVD for videos. It summarizes the discrepancy between generated and reference distributions in a single scalar, with lower values typically interpreted as better generation quality. However, this scalar view can obscure what drives the comparison. For example, in COCO dataset, increasing the number of diffusion sampling steps improves ImageReward scores yet worsens (increases) FID. Motivated by this mismatch, we seek to make the Fréchet distance more interpretable by uncovering where the discrepancy lies. To this end, we introduce directional Fréchet distance, the expected squared projection of the optimal transport displacement onto a given direction. Across our image, video, and protein case studies, we find that a small number of interpretable directions account for much of the distance. We use these directions to explain the FID increase in terms of semantic concepts represented by CLIP embeddings, quantify FVD's bias toward per-frame appearance, and revisit the interpretation of Protein FID. We open-source our codebase at https://github.com/yhlee-add/directional-fd.

Figures & tables

Figure 1: Sign reversal on SD1.5/COCO under two standard knobs. As the classifier-free guidance scale (left) or the number of sampling steps (right) increases, per-sample quality rises while FID worsens. A lower FID therefore does not track higher per-sample quality.

	share	dictionary	surviving concepts (weight)	cos
$e_{1}$	$24.0\%$	CLIP	composition simplicity ( $0.55$ ), depth of field ( $0.12$ ), furniture ( $0.08$ )	$0.74$
$e_{2}$	$3.8\%$	CLIP	brightness ( $-0.15$ ), weather ( $-0.12$ ), crowd density ( $0.08$ ), indoor/outdoor ( $0.08$ )	$0.49$
$e_{3}$	$2.5\%$	ImageNet	Dungeness crab ( $-0.09$ ), mashed potato ( $-0.06$ ), burrito ( $-0.05$ )	$0.40$

Table 1: Sparse concept fit of the top eigenvectors of

M

over the two concept dictionaries, for SD1.5,

50

steps, and COCO. Share is

\mathrm{FD}(e_{i})/\mathrm{FD}

, the weights are the largest surviving Lasso coefficients, and cos is the reconstruction cosine.

Figure 2: Reference and generated features projected onto the top three eigenvectors of $M$ , using SD1.5, $50$ steps, and COCO. Each point pairs a caption’s reference and generated projection. The reference (red) and generated (blue) marginal distributions are drawn inside, and the dashed line is $y=x$ . For $e_{1}$ , the points are slightly above $y=x$ , indicating that the generated set is shifted toward $+e_{1}$ . For $e_{2}$ , the points concentrate vertically, demonstrating that the generated set has less variance, and vice versa for $e_{3}$ .

Figure 3: Left : directional FD of the top three fixed eigenvectors against total FID across the eight-step COCO sweep, in the frozen

50

-step basis. Each point is one step setting, the solid line is the least-squares fit, and its slope is the fraction of the FID change that eigenvector carries, and the dashed line is the

\mathrm{slope}=1

an eigenvector carrying the entire change would follow. The top eigenvector

e_{1}

carries most of it (slope

0.66

R^{2}=0.96

), while

e_{2}

and

e_{3}

stay nearly flat (slopes

0.03

and

0.02

). Right : cumulative slope

s_{1:i}

, the summed slopes of the top

i

eigenvectors, which rises above the whole change (dashed) over the head and falls back to it over the tail (the eigenvectors before and after the curve’s peak).

	flicker set, $M(R,T)$				generator, $M(R,G)$
	$k=1$		$k=10$		$k=1$		$k=10$
embedding, family	appearance	flicker	appearance	flicker	appearance	flicker	appearance	flicker
I3D, motion blur	$0.325$	$0.023$	$0.663$	$0.273$	$0.022$	$0.005$	$0.288$	$0.150$
I3D, elastic	$0.372$	$0.211$	$0.648$	$0.495$	$0.027$	$0.012$	$0.288$	$0.107$
VideoMAE, motion blur	$0.061$	$0.470$	$0.181$	$0.612$	$0.013$	$0.019$	$0.095$	$0.096$
VideoMAE, elastic	$0.163$	$0.389$	$0.282$	$0.553$	$0.012$	$0.009$	$0.148$	$0.084$

Table 2: Share of directional FD on the appearance and flicker bands (the top eigenvectors of

M(R,S)

and

M(S,T)

), at the top

k=1

and

k=10

eigenvectors. The flicker-set panel evaluates

M(R,T)

at severity

3

, with bands averaged over the other four severities. The generator panel evaluates ModelScope’s generated set, zero-shot on the UCF-101 class names, on bands averaged over all five severities. The two bands overlap, so the shares of a row can sum past one.

	share	surviving concepts (weight)	cos	$d$	var
$e_{1}$	$66\%$	coil ( $-0.97$ ), contact order ( $+0.53$ ), length ( $+0.16$ ), globularity ( $-0.16$ )	$0.72$	$+3.2$	$0.55$
$e_{2}$	$14\%$	backbone-bond deviation ( $-0.58$ ), clash ( $+0.50$ )	$0.92$	$+0.0$	$0.05$
$e_{3}$	$4\%$	contact order ( $-0.90$ ), clash ( $+0.31$ )	$0.83$	$+0.1$	$0.16$

Table 3: Sparse concept fit of the top eigenvectors of

M

over the geometry concept dictionary, and how the two sets differ along each, for La-Proteina at

400

steps against the ESM3 reference, in the

32

-dimensional PCA space of Protein FID. Share is

\mathrm{FD}(e_{i})/\mathrm{FD}

, the weights are Lasso coefficients, cos is the reconstruction cosine,

d

is the Cohen’s

d

of the reference and generated projections onto

e_{i}

, and var is their variance ratio. We show the top surviving concepts here. Refer to Appendix G for details.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	dictionary	surviving concepts (weight)	cos
$e_{1}$	CLIP	composition simplicity ( $+0.55$ ), depth of field ( $+0.12$ ), furniture ( $+0.08$ ), electronics ( $+0.05$ ), daytime ( $+0.04$ ), portrait versus scene ( $-0.03$ )	$0.74$
$e_{1}$	ImageNet	notebook ( $+0.02$ )	$0.16$
$e_{2}$	CLIP	brightness ( $-0.15$ ), weather ( $-0.12$ ), crowd density ( $+0.08$ ), indoor/outdoor ( $+0.08$ ), depth of field ( $+0.05$ ), electronics ( $+0.01$ ), centered composition ( $-0.01$ ), vehicles ( $+0.01$ )	$0.49$
$e_{2}$	ImageNet	hand blower ( $+0.08$ ), horse cart ( $-0.06$ ), china cabinet ( $+0.06$ ), Crock Pot ( $+0.03$ ), space heater ( $+0.02$ ), crane ( $-0.01$ ), jinrikisha ( $-0.01$ ), tank ( $-0.01$ )	$0.43$
$e_{3}$	CLIP	indoor/outdoor ( $+0.03$ ), sports equipment ( $+0.01$ )	$0.17$
$e_{3}$	ImageNet	Dungeness crab ( $-0.09$ ), mashed potato ( $-0.06$ ), burrito ( $-0.05$ ), spiny lobster ( $-0.02$ ), orange ( $+0.02$ ), ice cream ( $-0.02$ )	$0.40$

Appendix

Table 4: Every concept whose weight rounds to at least

0.01

in the sparse concept fits of the top eigenvectors of

M

, for SD1.5,

50

steps, and COCO. Each weight is positive when the concept, or class logit, grows toward

+e_{i}

, the generated end. The last column is the reconstruction cosine.

Figure 4: Cumulative slope

\beta_{1:i}=\sum_{j\leq i}\beta_{j}

of the top

i

fixed eigenvectors against

i

. Here

\beta_{i}

is the slope of the directional FD

\mathrm{FD}(e_{i})

against the step count, in each reference’s own fixed

50

-step basis. The curves of COCO, Flickr, ImageNet, and MJHQ rise over the head before falling over the tail, while the face curves descend throughout. Each endpoint is the total slope

\beta_{\mathrm{FD}}

, which is positive for COCO, Flickr, and ImageNet, negative for the faces, and near zero for MJHQ.

embedding, family	set	cos	union	flicker $\mid$ appearance	appearance $\mid$ flicker
I3D, motion blur	flicker set	$0.74$	$0.746$	$0.083$	$0.473$
I3D, elastic	flicker set	$0.80$	$0.833$	$0.185$	$0.338$
VideoMAE, motion blur	flicker set	$0.61$	$0.786$	$0.605$	$0.174$
VideoMAE, elastic	flicker set	$0.69$	$0.741$	$0.459$	$0.188$
I3D, motion blur	generated	$0.74$	$0.384$	$0.096$	$0.234$
I3D, elastic	generated	$0.80$	$0.371$	$0.083$	$0.265$

Appendix

Table 5: Overlap of the

k=10

appearance and flicker bands. The flicker set is

M(R,T)

at severity

3

with bands computed from the other severities, and the generated set is

M(R,G)

with bands computed from all severities. Columns give the largest principal-angle cosine between the bands, the share of their union, and the share of the flicker band orthogonalized against the appearance band (flicker

\mid

appearance) and the reverse.

severity	$1$	$2$	$3$	$4$	$5$
I3D ratio	$1.41$	$1.29$	$1.18$	$1.11$	$1.07$
VideoMAE ratio	$3.56$	$3.63$	$3.23$	$2.87$	$2.77$
I3D flicker share, $k=1$	$0.053$	$0.028$	$0.023$	$0.028$	$0.031$
VideoMAE flicker share, $k=1$	$0.235$	$0.371$	$0.470$	$0.475$	$0.374$

Appendix

Table 6: Scalar ratio and top flicker-eigenvector share across the five motion blur severities, with bands computed from the other severities.

space		share	surviving concepts (weight)	cos	$d$	var
PCA- $32$	$e_{1}$	$66\%$	coil ( $-0.97$ ), contact order ( $+0.53$ ), length ( $+0.16$ ), globularity ( $-0.16$ ), radius of gyration ( $-0.10$ )	$0.72$	$+3.2$	$0.55$
	$e_{2}$	$14\%$	backbone-bond deviation ( $-0.58$ ), clash ( $+0.50$ ), contact order ( $+0.18$ ), segments per residue ( $+0.17$ )	$0.92$	$+0.0$	$0.05$
	$e_{3}$	$4\%$	contact order ( $-0.90$ ), clash ( $+0.31$ ), segments per residue ( $+0.18$ ), backbone-bond deviation ( $+0.17$ ), radius of gyration ( $+0.02$ )	$0.83$	$+0.1$	$0.16$
raw	$e_{1}$	$56\%$	coil ( $-0.99$ ), length ( $+0.51$ ), contact order ( $+0.45$ ), radius of gyration ( $-0.36$ ), alpha-versus-beta ( $+0.06$ ), segments per residue ( $-0.04$ )	$0.71$	$+3.2$	$0.56$
	$e_{2}$	$12\%$	backbone-bond deviation ( $-0.58$ ), clash ( $+0.50$ ), contact order ( $+0.19$ ), segments per residue ( $+0.18$ ), strand ( $+0.06$ ), coil ( $-0.06$ ), globularity ( $+0.04$ )	$0.90$	$+0.0$	$0.04$
	$e_{3}$	$4\%$	contact order ( $-0.82$ ), clash ( $+0.30$ ), backbone-bond deviation ( $+0.17$ ), segments per residue ( $+0.13$ ), radius of gyration ( $+0.02$ ), strand ( $-0.01$ ), length ( $-0.01$ )	$0.76$	$+0.1$	$0.16$

Appendix

Table 7: Full sparse concept fits of the top eigenvectors of

M

for La-Proteina at

400

steps, in the

32

-dimensional PCA space of Protein FID (Table 3 , penalty

10^{-3}

) and in the raw

1536

-dimensional ESM3 space (penalty

10^{-5}

). Every surviving concept is listed with its weight, and the other columns are as in Table 3 .

Explore similar work

Representation Fréchet Loss for Visual Generation
Apr 30, 2026Jiawei Yang, Zhengyang Geng, Xuan Ju +2Fréchet Inception Distance Visual Generation
USC · CMU · CUHK +1
Rethinking FID Through the Geometry of the Reference Dataset
May 28, 2026Yunghee Lee, Byeonghyun PakFréchet Inception Distance Real-World Dataset
AdvFD: Boosting Visual Generation via Adversarial Fr'echet Distance Loss
Aug 11, 2026Mingju Gao, Jingkai Zhou, Kun Gai +2Fréchet Inception Distance Visual Generation
Peking University · KlingAI Research

Apr 30, 2026, cs.CVJ/K move · Enter open · S save

Representation Fréchet Loss for Visual Generation

Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian+1

USC · CMU · CUHK+1

Fréchet Inception Distance Visual Generation Adversarial Training

Open paper AskPDF arXiv

We show that Fréchet Distance (FD), long considered impractical as a training objective, can in fact be effectively optimized in the representation space. Our idea is simple: decouple the population size for FD estimation (e.g., 50k) from the batch size for gradient computation (e.g., 1024). We term this approach FD-loss. Optimizing FD-loss reveals several surprising findings. First, post-training a base generator with FD-loss in different representation spaces consistently improves visual quality. Under the Inception feature space, a one-step generator achieves0.72 FID on ImageNet 256x256. Second, the same FD-loss repurposes multi-step generators into strong one-step generators without teacher distillation, adversarial training or per-sample targets. Third, FID can misrank visual quality: modern representations can yield better samples despite worse Inception FID. This motivates FDr

^k

, a multi-representation metric. We hope this work will encourage further exploration of distributional distances in diverse representation spaces as both training objectives and evaluation metrics for generative models.

Abstract

Explore similar work

Representation Fréchet Loss for Visual Generation

Rethinking FID Through the Geometry of the Reference Dataset

AdvFD: Boosting Visual Generation via Adversarial Fr'echet Distance Loss