Latent generative models synthesize images by decoding generated latents, yet their decoder is trained only to reconstruct encoder latents and never sees the latents it decodes at test time, so the residual errors of the generator are rendered directly into the final image. Existing remedies retrain the generator, add cost at every sampling step, or make the decoder robust to perturbed real latents, assuming that generator errors resemble such perturbations. We find that they do not. Generated latents lie on the encoder latent manifold as closely as real ones, yet follow a different distribution. The decoder should therefore be adapted to the distribution of generated latents, which raises two challenges: (1) unpaired supervision, since a generated latent has no ground-truth image, and (2) latent-space drift, since a decoder freed from paired targets forgets the latent space it must decode. We propose Sample-Aligned Decoders (SAD), a post-training framework that fine-tunes only the decoder of a frozen generator at no inference cost. Generated-Sample Distribution Alignment matches decoded generated latents to unpaired real images in a frozen foundation-model feature space, and Encoder Anchor Preservation keeps the decoder close to the original one on encoder latents. Our analysis shows that the alignment is effective only on generated latents, where it corrects each sample in proportion to how much more often the generator produces such latents than the encoder; on encoder latents it can only work against the anchor. SAD improves four state-of-the-art ImageNet generators across three tokenizers, e.g., FID-50K from 1.15 to 0.98 for REPA-E, complements discriminator guidance, and extends to text-to-image generation. Our code and models are available at https://torpedo2003.github.io/SAD/.
Figures & tables
Figure 1: Motivation of SAD. (a) The decoder of a latent generative model is trained on encoder latents but decodes generated latents at test time, so the residual errors of the generator reach the final image. (b) SAD corrects them by fine-tuning only the decoder; FID-50K of released generators with D0 and with SAD.
Method
Generated latents
No paired targets
Keeps reconstr.
No inference cost
Diffusion dec. [ 38 , 9 ]
✗
✗
✓
✗
Noise-robust dec. [ 63 ]
✗
✗
✓
✓
RobusTok [ 42 ]
∘
✗
✓
✓
GAR [ 12 ]
∘
✗
✗
✓
FDDA [ 13 ] (contemp.)
✓
✓
✗
✓
Disc. guidance [ 21 ]
✓
✓
–
✗
Table 1: Comparison with related correction methods. Generated latents: trained on latents produced by the generator ( ∘ : only partially generated, i.e . teacher-forced or partially denoised). Reconstruction: the decoder stays valid on encoder latents.
Method
Tokenizer
FID ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
FD CLIP↓
Other generators
ADM-G [ 10 ]
–
4.59
5.25
186.7
0.82
0.52
46.9
VAR-d30 [ 54 ]
VQVAE
1.92
–
323.1
0.82
0.59
54.0
MAR-H [ 30 ]
KL-16
1.55
–
303.7
0.81
0.62
–
SiT-XL/2 [ 32 ]
SD-VAE
2.06
4.50
270.3
0.82
0.59
44.3
MDTv2-XL/2 [ 14 ]
SD-VAE
1.58
4.52
314.7
0.79
0.65
–
Table 2: Comparison with state-of-the-art generators on ImageNet 256×256 . With guidance; FID-50K, ADM evaluator. For each released generator, the first row uses its original decoder D0 and the second row SAD (GAN discrepancy) on identical latents. Bold marks a result no worse than D0 . FD CLIP uses a held-out reference and is computed by us, for other generators from their released samples or weights. † Class-balanced labels.
Method
Extra sampling
Disc. real images
FID ↓
sFID ↓
IS ↑
D0
–
–
2.30
4.51
277.8
DG [ 21 ]
+ 20%
1.28M
2.01
4.49
266.5
SAD (ours)
0
5k
2.13
4.25
280.5
DG + SAD (ours)
+ 20%
1.28M
1.86
4.02
269.3
Table 3: Comparison of generator-side and decoder-side correction. DiT-XL/2 in the official setting at guidance 1.5 (FID-50K, ADM evaluator). The best results are in bold and the second best are underlined.
Method
FID ↓
FD CLIP↓
rFID ↓
PSNR ↑
Original decoder D0
13.32
51.4
2.63
23.91
Replacing or re-training the decoder
sd-vae-ft-mse [ 43 ]
13.40
51.8
2.91
24.75
Consistency Decoder [ 38 ]
13.18
57.3
3.34
23.31
Fine-tuning, ℓ1 +LPIPS+GAN
13.40
48.6
7.02
24.51
Robustness to perturbed latents
Table 4: Comparison with decoder adaptation methods. DiT-XL/2 in the ablation setting. Noise-robust, paired and FDDA are our re-implementations. The best results are in bold and the second best are underlined. ∗ Trained with Inception, the FID network.
Decoder
FID ↓
FD CLIP↓
CLIP ↑
Rec. ↑
rFID ↓
Pick
D0
34.73
209.1
26.03
0.286
4.28
–
GAN, λ=0.01
23.55
141.6
26.91
0.416
4.29
48%
MMD, λ=0.1
28.35
150.4
26.19
0.343
4.26
67%
Real images
–
–
25.69
–
–
18%
Table 5: Text-to-image generation. PixArt- α on COCO (10k samples). Pick: share of samples preferred by PickScore over D0 (same latent; for real val2017 images, same caption). All tested weights are in Appendix D .
#
Variant
FID ↓
FD CLIP↓
rFID ↓
PSNR ↑
0
D0
13.32
51.4
2.63
23.91
Ablation on GSA
1
w/o GSA (EAP only)
13.31
49.9
2.65
23.92
2
GSA on E(x) instead of zgen
13.59
51.5
7.70
23.90
3
GSA with MMD
12.68
41.6
2.57
23.92
4
GSA with FD
12.79
41.8
2.59
23.91
Table 6: Ablation study on the modules of SAD. DiT-XL/2 in the ablation setting.
Figure 3: Sensitivity to the loss weights. DiT-XL/2 in the ablation setting. (a) Weight of LEAP with λ=0.05 . (b) Weight λ of LGSA with the weight of LEAP fixed to 1. Defaults: 1 and 0.05.
Figure 4: Qualitative comparison on DiT-XL/2. The same generated latents are decoded by D0 and by SAD. The bottom row shows their per-pixel absolute difference (brighter is larger).
Figure 5: Effect of where GSA is applied. DiT-XL/2 in the ablation setting. (a) Generation (FID on 10k samples) and reconstruction quality of all trained decoders. (b) Radially averaged power spectrum of decoded samples relative to real images.
Statistic
Encoder
Generated
Relative change ∥P(z)−z∥2/∥z∥2
0.017
0.016
AUC of this change
–
0.49
AUC of a small CNN on z
–
0.96
AUC of a small CNN on P(z)
–
0.95
Table 7: Distribution test of generated latents. DiT-XL/2, 10k latents each; P(z)=E(D0(z)) . AUC: encoder vs . generated.
Statistic
Real
D0
SAD
Saturation (HSV)
0.323
0.375
0.324
Brightness (HSV)
0.526
0.539
0.531
Colorfulness [ 16 ]
42.0
47.8
41.3
Clipped highlights (%)
4.8
9.4
5.9
Clipped shadows (%)
1.4
4.1
1.9
Table 8: Color and exposure statistics. DiT-XL/2, 50-step DPM-Solver++, 3k samples and 3k held-out real images. Clipped: pixels with a channel ≥250 or all channels ≤5 .
Figure 6: Per-sample correction against the density ratio. DiT-XL/2, 10k encoder and 10k generated latents. A classifier trained to separate generated from encoder latents scores every latent, and its logit ranks latents by r(z) . Each point is one quintile of a set by this score; y : mean pixel change ∣Dϕ(z)−D0(z)∣ made by SAD.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
trainable params
extra params at inference
extra GFLOPs per image
training (GPU-h)
SAD
49.5M
0
0
0.9
Pixel refiner
6.8M
6.8M
95 ( +15% )
0.6
Appendix
Table 9: Training and inference cost of SAD.
Method
FID ↓
sFID ↓
IS ↑
FD CLIP↓
Original decoder D0
2.30
4.51
277.8
42.8
Noise-robust decoder [ 63 ]
2.31
4.64
277.0
43.1
Paired post-training [ 42 , 12 ]
2.69
7.67
266.7
54.9
FDDA [ 13 ] ∗
2.02
4.84
274.9
48.9
FDDA [ 13 ] ∗ + EAP
2.27
4.14
279.4
51.2
SAD-GAN (ours)
2.13
4.25
280.5
36.8
Appendix
Table 10: Comparison with decoder adaptation methods in the official setting. DiT-XL/2 at guidance 1.5 (FID-50K, ADM evaluator; FD CLIP with a held-out reference). All decoders are trained on the same 40k generated latents. The best results are in bold and the second best are underlined. ∗ Inception, the FID network, is part of its training objective.
D0→ SAD
FID ↓
FD CLIP↓
rFID ↓
Pick
DiT-XL/2 on ImageNet (50k samples)
DPM-50, GAN
3.24 → 2.28
48.3 → 39.4
2.63 → 2.56
51%
DDPM-250 (official), GAN
2.30 → 2.13
42.8 → 36.8
2.63 → 2.69
50%
DDPM-250 (official), MMD
2.30 → 2.17
42.8 → 39.4
2.63 → 2.66
52%
LightningDiT with VA-VAE on ImageNet (50k samples)
LightningDiT-XL, GAN
1.38 → 1.23
29.4 → 28.2
1.18 → 1.17
70%
Appendix
Table 11: Results across generators and datasets. Each decoder is trained on latents of its own generator. ODE pool: training latents from the 50-step Heun ODE instead of the deployed SDE sampler. Pick: share of samples for which PickScore prefers SAD over D0 (ImageNet prompts: “a photo of a class ”).
Sampling setting
FID ↓
FD CLIP↓
D0
SAD
D0
SAD
cfg 1.5, 50 steps (train)
13.32
12.37
51.4
44.2
cfg 1.5, 20 steps
13.82
12.68
52.8
42.0
cfg 1.0, 50 steps
18.68
18.39
68.8
68.5
cfg 4.0, 50 steps
26.50
25.75
70.2
49.5
Appendix
Table 12: Transfer of one decoder to other sampler settings. SAD values are means over three training seeds.
Scale
Interval
FID ↓
FD CLIP↓
D0
SAD
D0
SAD
1.5
–
13.32
12.37
51.4
44.2
1.75
–
15.31
14.34
49.9
40.1
2.0
✓
13.53
12.65
49.6
40.7
2.5
✓
14.98
14.19
50.1
38.4
3.0
✓
16.51
15.82
51.9
37.9
Appendix
Table 13: SAD combined with guidance tuning. SAD values are means over three training seeds.
Figure 7: Training dynamics and data efficiency. DiT-XL/2 in the ablation setting. (a) GSA weight λ relative to the default. (b, c) FID and rFID during training. (d) Number of cached latents or real images.
Decoder
KID ↓
FD SigLIP↓
CMMD ↓
HPS
PickScore
HPS win
Pick win
DiT-XL/2, DPM-Solver++ (50 steps)
D0
1.17
10.0
0.227
0.235
20.42
–
–
LGSA on E(x) , GAN
1.25
9.6
0.267
0.231
20.39
25%
45%
SAD-GAN
0.59
8.0
0.267
0.231
20.42
31%
51%
SAD-MMD
0.72
7.9
0.187
0.235
20.42
48%
50%
DiT-XL/2, official code (DDPM-250)
Appendix
Table 14: Evaluation with metrics unused in training. KID ×103 , FD in SigLIP space, CMMD, HPS v2.1 and PickScore. Win: share of samples for which the preference model prefers the decoder (or the real image with the same caption) over D0 .
Statistic
Encoder
Generated (1.0)
Generated (1.5)
Relative change ∥P(z)−z∥2/∥z∥2
0.0168
0.0160
0.0162
AUC of this change
–
0.48
0.49
FID of D0(z) (encoder: rFID)
2.63
18.62
12.48
FID of D0(P(z)) (encoder: rFID)
3.91
20.67
13.40
AUC of a small CNN on z
–
0.94
0.96
AUC of a small CNN on P(z)
–
0.93
0.95
Appendix
Table 15: Full distribution test of generated latents. DiT-XL/2 in the ablation setting, 10k latents per set, generated latents at guidance 1.0 and 1.5. P(z)=E(D0(z)) with the posterior mean. AUC: encoder vs . generated latents.
Figure 8: PixArt- α samples decoded by D0 and by SAD-GAN at two values of λ . The last two examples show the failure modes at the larger λ .
Decoder
FID ↓
FD CLIP↓
CLIP Sc.
Rec.
rFID ↓
Pick win
D0
34.73
209.1
26.03
0.286
4.28
–
GAN, λ=0.005
24.67
155.9
27.21
0.400
4.28
39%
GAN, λ=0.01
23.55
141.6
26.91
0.416
4.29
48%
GAN, λ=0.02
25.51
135.6
26.40
0.365
4.40
48%
GAN, λ=0.05
20.45
116.8
26.82
0.408
4.91
24%
MMD, λ=0.1
28.35
150.4
26.19
0.343
4.26
67%
Appendix
Table 16: SAD on PixArt- α for all tested λ . COCO, 10k samples. Pick win: share of samples preferred by PickScore over D0 (same latent; for real val2017 images, same caption).
Figure 9: Qualitative comparison on other generators. (a) LightningDiT-XL, (b) REPA-E and (c) PixArt- α . The same generated latents are decoded by D0 and by SAD. The bottom row shows their per-pixel absolute difference (brighter is larger).
Figure 10: Additional samples decoded by D0 and SAD. DiT-XL/2 in the ablation setting (guidance 1.5); the last two samples use guidance 4.0, which is not seen in training. The third and sixth rows show the per-pixel absolute difference (brighter is larger).
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.
Fréchet Distributional (FD) objectives provide an effective distribution-level signal for post-training generative models. For latent generative models, however, existing FD-based post-training methods optimize only the latent generator while keeping the decoder frozen: the decoder is trained on reconstruction latents obtained from real images but deployed on generator-produced latents at generation time. When these latent distributions differ, this decoder train-generation mismatch means that the decoder is deployed on inputs drawn from a distribution it was not directly trained on. We introduce Fréchet Distributional Decoder Alignment (FDDA), an efficient post-training method that freezes the latent generator and directly optimizes the decoder on generation-time latents using an FD objective. A single epoch of FDDA consistently improves gFID and gFDr6 across diverse flow-based and autoregressive latent generative models. Across all 11 evaluated models, FDDA reduces gFDr6 by 34.0-59.6% from the original pretrained models and provides a further 10.2-57.2% reduction after generator-side FD post-training. FDDA also reduces total wall-clock time by factors of 3.6×-328.7× relative to generator-side FD post-training across these models. These results show that generator-side post-training alone does not exhaust the benefits of FD optimization, and that further gains can be obtained by adapting the decoder at substantially lower cost.
Xianghong Fang, Jiajun Zhu, Esma Aimeur +2
University of Toronto · Université de Montréal · Vijil
In latent generative models, reconstruction quality is often assumed to correlate with generative performance. However, reconstruction FID (rFID) can exhibit weak or even negative correlation with generation FID (gFID). We attribute this misalignment to a latent distribution mismatch: reconstruction evaluates the decoder on encoder-induced latents, whereas generation uses the same decoder on latents produced by the generative model. To characterize this shift, we introduce generation-aware reconstruction (GAR), which constructs a continuous trajectory from standard reconstruction toward generation by perturbing encoder latents with noise and denoising them through the generative model before decoding. GAR probes the decoder behavior along this trajectory, making the transition from encoder to generation-time latent distributions observable and diagnosable. The resulting trajectory-based diagnostic, GAR-FID, exhibits strong empirical correlation with gFID across diverse tokenizers and scales. Importantly, intermediate GAR latents become more generation-aware while preserving correspondence with their source images, thereby retaining paired supervision that is absent for fully generated latents. This correspondence enables decoder adaptation on intermediate GAR latents, consistently improving generative quality across model scales. Overall, latent distribution mismatch provides a useful perspective for evaluating and improving latent generative models.
Latent diffusion models (LDMs) enable high-fidelity synthesis by operating in learned latent spaces. However, training state-of-the-art LDMs requires complex staging: a tokenizer must be trained first, before the diffusion model can be trained in the frozen latent space. We propose UNITE - an autoencoder architecture for unified tokenization and latent diffusion. UNITE consists of a Generative Encoder that serves as both image tokenizer and latent generator via weight sharing. Our key insight is that tokenization and generation can be viewed as the same latent inference problem under different conditioning regimes: tokenization infers latents from fully observed images, whereas generation infers them from noise together with text or class conditioning. Motivated by this, we introduce a single-stage training procedure that jointly optimizes both tasks via two forward passes through the same Generative Encoder. The shared parameters enable gradients to jointly shape the latent space, encouraging a "common latent language". Across image and molecule modalities, UNITE achieves near state of the art performance without adversarial losses or pretrained encoders (e.g., DINO), reaching FID 2.12 and 1.73 for Base and Large models on ImageNet 256 x 256. We further analyze the Generative Encoder through the lenses of representation alignment and compression. These results show that single stage joint training of tokenization & generation from scratch is feasible.