Efficient Fréchet Distributional Decoder Alignment for Latent Generative Models

Xianghong Fang; Jiajun Zhu; Esma Aimeur; Dehan Kong; Tim G. J. Rudner

cs.CVOct 12, 2026

Efficient Fréchet Distributional Decoder Alignment for Latent Generative Models

Authors: Xianghong Fang, Jiajun Zhu, Esma Aimeur, Dehan Kong, Tim G. J. Rudner

Organizations: University of Toronto · Université de Montréal · Vijil

Abstract

Fréchet Distributional (FD) objectives provide an effective distribution-level signal for post-training generative models. For latent generative models, however, existing FD-based post-training methods optimize only the latent generator while keeping the decoder frozen: the decoder is trained on reconstruction latents obtained from real images but deployed on generator-produced latents at generation time. When these latent distributions differ, this decoder train-generation mismatch means that the decoder is deployed on inputs drawn from a distribution it was not directly trained on. We introduce Fréchet Distributional Decoder Alignment (FDDA), an efficient post-training method that freezes the latent generator and directly optimizes the decoder on generation-time latents using an FD objective. A single epoch of FDDA consistently improves gFID and gFDr $^6$ across diverse flow-based and autoregressive latent generative models. Across all 11 evaluated models, FDDA reduces gFDr $^6$ by 34.0-59.6% from the original pretrained models and provides a further 10.2-57.2% reduction after generator-side FD post-training. FDDA also reduces total wall-clock time by factors of $3.6\times$ - $328.7\times$ relative to generator-side FD post-training across these models. These results show that generator-side post-training alone does not exhaust the benefits of FD optimization, and that further gains can be obtained by adapting the decoder at substantially lower cost.

Figures & tables

Panel of Figure 1 — Figure 1: Decoder-side FD post-training with FDDA. (a) Existing methods optimize the latent generator with a frozen decoder, whereas FDDA freezes the generator and optimizes the decoder on generation-time latents. (b) FDDA provides complementary gains. Top row: FDDA applied to released checkpoints before generator-side FD post-training. Bottom row: FDDA applied after generator-side FD post-training. Gray and blue bars denote gFDr 6 before and after FDDA, and annotations report the relative reduction. (c) Decoder-side FD post-training substantially reduces training cost. Gray (left) and blue (right) bars denote total wall-clock hours on a single H100 GPU for generator-side and decoder-side FD post-training, and annotations show the reduction factor.

Models	Generator FD	Decoder FD	gFID ( $\boldsymbol{\downarrow}$ )	gIS ( $\boldsymbol{\uparrow}$ )	gFD ${}_{\text{conv}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{dino}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{mae}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{sig}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{clip}}$ ( $\boldsymbol{\downarrow}$ )	gFDr 6 ( $\boldsymbol{\downarrow}$ )
Discrete autoregressive generative models
LlamaGen-B	✗	✗	5.17	196.1	233.91	244.12	0.528	21.03	149.44	16.48
LlamaGen-B	✗	✔	2.74	209.3	237.39	190.90	0.250	11.50	115.52	10.88 $\downarrow_{34.0\%}$
LlamaGen-L	✗	✗	4.23	290.1	113.61	121.41	0.383	12.20	97.13	10.00
LlamaGen-L	✗	✔	1.87	303.5	103.27	87.98	0.109	5.19	73.07	5.60 $\downarrow_{44.0\%}$
GigaTok-S-S	✗	✗	4.34	268.5	135.95	182.66	0.432	15.25	112.26	12.29

Table 1: FDDA before generator-side FD post-training. We apply FDDA to the original pretrained latent generative models. All metrics are computed using the same evaluation protocol following [ Yang2026RepresentationFL ] . ✗and ✔indicate whether each component undergoes FD post-training.

Models	Generator FD	Decoder FD	gFID ( $\boldsymbol{\downarrow}$ )	gIS ( $\boldsymbol{\uparrow}$ )	gFD ${}_{\text{conv}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{dino}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{mae}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{sig}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{clip}}$ ( $\boldsymbol{\downarrow}$ )	gFDr 6 ( $\boldsymbol{\downarrow}$ )
Discrete autoregressive generative models
LlamaGen-B	✔	✗	2.45	277.9	118.37	105.41	0.174	6.07	70.55	6.33
LlamaGen-B	✔	✔	2.23	275.9	134.35	95.79	0.100	4.55	61.77	5.27 $\downarrow_{16.7\%}$
LlamaGen-L	✔	✗	1.32	315.9	74.54	57.09	0.104	3.18	54.27	3.96
LlamaGen-L	✔	✔	1.34	313.5	83.61	49.14	0.052	2.40	44.71	3.18 $\downarrow_{19.7\%}$
GigaTok-S-S	✔	✗	2.52	291.6	97.96	82.97	0.219	4.45	57.20	5.34

Table 2: FDDA after generator-side FD post-training. We apply FDDA to latent generative models whose generators have already undergone FD post-training. All metrics are computed using the same evaluation protocol following [ Yang2026RepresentationFL ] . ✗and ✔indicate whether each component undergoes FD post-training.

Models	NFE	#params	gFID ( $\boldsymbol{\downarrow}$ )	gIS ( $\boldsymbol{\uparrow}$ )	gFD ${}_{\text{conv}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{dino}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{mae}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{sig}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{clip}}$ ( $\boldsymbol{\downarrow}$ )	gFDr 6 ( $\boldsymbol{\downarrow}$ )
Without FD post-training
LlamaGen-L ‡	256 $\times$ 2	343M	4.23	290.1	113.61	121.41	0.383	12.20	97.13	10.00
GigaTok-S-S ‡	256 $\times$ 2	111M	4.34	268.5	135.95	182.66	0.432	15.25	112.26	12.29
TiTok-L-32 ‡	8 $\times$ 2	177M	2.75	194.1	190.86	192.05	0.567	18.86	157.93	15.32
VAR- $d24$ ‡	10 $\times$ 2	1.0B	2.11	314.7	94.78	76.34	0.286	7.37	73.93	6.79
VAR- $d30$ †	10 $\times$ 2	2B	1.97	304.6	96.57	75.35	0.29	7.19	74.61	6.70

Table 3: System-level comparison on class-conditional ImageNet

256\times 256

. NFE denotes the number of function evaluations during inference, and #Params denotes the number of parameters in the latent generator. All results use CFG where applicable;

\times 2

indicates the additional function evaluations for CFG. The best result for each metric is highlighted in bold. † : results reported in Tables 4 and F.7 of FD-Loss [ Yang2026RepresentationFL ] . ‡ : results measured by us following the evaluation protocol of [ Yang2026RepresentationFL ] . Decoder-side post-training is applied to the generator-side FD-post-trained models (FDDA+). ∗ : CFG is applied only over a time subinterval; NFE is reported as the full-CFG upper bound.

Models	Generator-side FD			Decoder-side FD (FDDA)				Cost Reduction
Models	Epochs	# Generator Params	Train (h)	# Decoder Params	Latent Pool (h)	Train (h)	Total (h)	Cost Reduction
Discrete autoregressive generative models
LlamaGen-B	1.25	111M	100.6	42.5M	1.50	2.77	4.27	23.6 $\times$
LlamaGen-L	1.25	343M	185.0	42.5M	4.02	2.78	6.80	27.2 $\times$
GigaTok-S-S	1.25	111M	72.2	83.8M	2.68	3.03	5.71	12.6 $\times$
TiTok-L-32	1.25	177M	16.7	337.6M	0.29	3.19	3.48	4.8 $\times$

Table 4: Computational efficiency of FDDA. We compare the wall-clock time of generator-side FD post-training and FDDA on a single H100 GPU. All FDDA models are trained for one epoch, with total time including both generation-time latent pool construction and decoder training. Cost reduction is computed as the ratio of generator-side FD training time to the total FDDA time.

Models	space	#params	Pretrained Decoder				FDDA-adapted Decoder
Models	space	#params	PSNR( $\boldsymbol{\uparrow}$ )	SSIM ( $\boldsymbol{\uparrow}$ )	rFID( $\boldsymbol{\downarrow}$ )	LPIPS( $\boldsymbol{\downarrow}$ )	PSNR( $\boldsymbol{\uparrow}$ )	SSIM( $\boldsymbol{\uparrow}$ )	rFID( $\boldsymbol{\downarrow}$ )	LPIPS( $\boldsymbol{\downarrow}$ )
LlamaGen-B	discrete	71.9M	22.53	55.3	2.20	0.138	21.73	52.9	2.82	0.154
LlamaGen-L	discrete	71.9M	22.53	55.3	2.20	0.138	21.93	54.0	2.55	0.150
GigaTok-S-S	discrete	136.1M	22.47	54.5	1.01	0.132	21.22	50.8	1.42	0.163
TiTok-L-32	discrete	641.1M	17.55	35.9	2.22	0.329	16.57	33.5	1.64	0.357
TiTok-B-64	discrete	204.8M	18.74	39.8	1.69	0.258	17.84	37.4	1.66	0.284

Table 5: Reconstruction performance before and after FDDA. We compare the pretrained and FDDA-adapted decoders using PSNR, SSIM, rFID, and LPIPS. FDDA is trained on generation-time latents, and we report its effect on reconstruction performance here.

Figure 2: Effect of FDDA training epochs.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Generator FD	Decoder FD	Decoder latents	gFID ( $\boldsymbol{\downarrow}$ )	gIS ( $\boldsymbol{\uparrow}$ )	gFD ${}_{\text{conv}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{dino}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{mae}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{sig}}$ ( $\boldsymbol{\downarrow}$ )	gFD ${}_{\text{clip}}$ ( $\boldsymbol{\downarrow}$ )	gFDr 6 ( $\boldsymbol{\downarrow}$ )
iMF-B	✗	✗	-	3.49	253.7	188.08	206.17	0.487	19.65	156.14	15.40
	✗	✔	$z_{r}$	3.27	265.1	229.78	172.91	0.361	15.13	141.03	12.91 $\downarrow_{16.2\%}$
	✗	✔	$z_{g}$	1.65	268.1	115.47	144.39	0.232	8.10	105.06	8.54 $\downarrow_{44.5\%}$
	✔	✗	-	0.86	307.1	65.97	89.05	0.154	5.46	70.35	5.57
	✔	✔	$z_{r}$	1.08	305.5	113.59	84.02	0.158	5.29	68.01	5.58 $\uparrow_{0.2\%\phantom{0}}$
	✔	✔	$z_{g}$	0.91	304.0	74.66	80.48	0.100	3.91	59.69	4.54 $\downarrow_{18.5\%}$

Appendix

Table 6: Effect of decoder post-training latents on iMF. We compare decoder-side FD post-training using reconstruction latents

z_{r}

or generation-time latents

z_{g}

, starting from either the original pretrained generator or its FD-post-trained counterpart. ✗and ✔indicate the absence and presence of FD post-training, respectively, for the component named in each column. All reported metrics evaluate generated images. The percentage beside gFDr 6 is the relative change from the row without decoder FD in the same generator setting; downward arrows indicate improvement.

Figure 3: Generated samples before generator-side FD post-training. Top: original pretrained model. Bottom: after FDDA.

Figure 4: Generated samples after generator-side FD post-training. Top: generator-side FD post-training. Bottom: after further applying FDDA.

Figure 5: Class 14: indigo bunting. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 6: Class 22: bald eagle. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 7: Class 42: agama. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 8: Class 81: ptarmigan. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 9: Class 88: macaw. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 10: Class 108: sea anemone. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 11: Class 130: flamingo. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 12: Class 207: golden retriever. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 13: Class 483: castle. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 14: Generated samples from LlamaGen-L. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 15: Generated samples from GigaTok-S-S. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 16: Generated samples from VAR- $d24$ . Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Figure 17: Generated samples from iMF-XL. Top: original pretrained model. Bottom: generator-side FD post-training + FDDA.

Flow-Based Generative Modeling

Flow-based generative modeling learns probability transports from simple noise to data for fast sampling, density estimation, and controllable generation.

See all 563 papers

Beyond Token-Level Cross-Entropy: Fréchet Distributional Post-Training for Autoregressive Image Generation
Aug 1, 2026Jinhua Zhang, Yisong Lin, Wei Long +1Generative Modeling Autoregressive Image Generation
University of Electronic Science and Technology of China
A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models
Sep 21, 2026Xianghong Fang, Wenjie Shu, Tongda Xu +3Distribution Shift Generative AI Evaluation
University of Toronto · Independent · Vijil
AdvFD: Boosting Visual Generation via Adversarial Fr'echet Distance Loss
Aug 11, 2026Mingju Gao, Jingkai Zhou, Kun Gai +2Diffusion Model Alignment Diffusion Model Fine-Tuning
Peking University · KlingAI Research

Aug 1, 2026J/K move · Enter open · S save

Beyond Token-Level Cross-Entropy: Fréchet Distributional Post-Training for Autoregressive Image Generation

Jinhua Zhang, Yisong Lin, Wei Long, Shuhang Gu

University of Electronic Science and Technology of China

Generative Modeling Autoregressive Image Generation

Open paper AskPDF arXiv

Autoregressive image generators are commonly pretrained with token-level cross-entropy under teacher forcing, yet evaluated by the distributional quality of decoded images. This creates an objective mismatch, because categorical errors have unequal image-level consequences, and a context mismatch, because inference conditions on model-generated histories. We introduce FD-loss post-training, which adapts a pretrained discrete generator using representation-space Fréchet distance as the sole objective. A dual-pass scheme first constructs detached rollout contexts through gradient-free generation under the model's native inference configuration, then performs differentiable replay with a probability-level straight-through estimator (STE) that preserves hard argmax decoding in the forward pass while propagating image-level gradients through temperature-scaled probabilities. Only the generator is updated, while the tokenizer and feature extractors remain frozen. Across eight completed configurations from four generator families on class-conditional ImageNet at

256\times256

, FD-loss post-training reduces FID and

\mathrm{FD}_{r6}

by 41.4% and 52.0% on average. The strongest FID result improves from 2.42 to 1.43 without adding parameters or inference steps.

Abstract

Flow-Based Generative Modeling

Explore similar work

Beyond Token-Level Cross-Entropy: Fréchet Distributional Post-Training for Autoregressive Image Generation

A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

AdvFD: Boosting Visual Generation via Adversarial Fr'echet Distance Loss