Latent generative models synthesize images by decoding generated latents, yet their decoder is trained only to reconstruct encoder latents and never sees the latents it decodes at test time, so the residual errors of the generator are rendered directly into the final image. Existing remedies retrain the generator, add cost at every sampling step, or make the decoder robust to perturbed real latents, assuming that generator errors resemble such perturbations. We find that they do not. Generated latents lie on the encoder latent manifold as closely as real ones, yet follow a different distribution. The decoder should therefore be adapted to the distribution of generated latents, which raises two challenges: (1) unpaired supervision, since a generated latent has no ground-truth image, and (2) latent-space drift, since a decoder freed from paired targets forgets the latent space it must decode. We propose Sample-Aligned Decoders (SAD), a post-training framework that fine-tunes only the decoder of a frozen generator at no inference cost. Generated-Sample Distribution Alignment matches decoded generated latents to unpaired real images in a frozen foundation-model feature space, and Encoder Anchor Preservation keeps the decoder close to the original one on encoder latents. Our analysis shows that the alignment is effective only on generated latents, where it corrects each sample in proportion to how much more often the generator produces such latents than the encoder; on encoder latents it can only work against the anchor. SAD improves four state-of-the-art ImageNet generators across three tokenizers, e.g., FID-50K from 1.15 to 0.98 for REPA-E, complements discriminator guidance, and extends to text-to-image generation. Our code and models are available at https://torpedo2003.github.io/SAD/.
Figures & tables
Figure 1: Motivation of SAD. (a) The decoder of a latent generative model is trained on encoder latents but decodes generated latents at test time, so the residual errors of the generator reach the final image. (b) SAD corrects them by fine-tuning only the decoder; FID-50K of released generators with D0 and with SAD.
Method
Generated latents
No paired targets
Keeps reconstr.
No inference cost
Diffusion dec. [ 38 , 9 ]
✗
✗
✓
✗
Noise-robust dec. [ 63 ]
✗
✗
✓
✓
RobusTok [ 42 ]
∘
✗
✓
✓
GAR [ 12 ]
∘
✗
✗
✓
FDDA [ 13 ] (contemp.)
✓
✓
✗
✓
Disc. guidance [ 21 ]
✓
✓
–
✗
Table 1: Comparison with related correction methods. Generated latents: trained on latents produced by the generator ( ∘ : only partially generated, i.e . teacher-forced or partially denoised). Reconstruction: the decoder stays valid on encoder latents.
Method
Tokenizer
FID ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
FD CLIP↓
Other generators
ADM-G [ 10 ]
–
4.59
5.25
186.7
0.82
0.52
46.9
VAR-d30 [ 54 ]
VQVAE
1.92
–
323.1
0.82
0.59
54.0
MAR-H [ 30 ]
KL-16
1.55
–
303.7
0.81
0.62
–
SiT-XL/2 [ 32 ]
SD-VAE
2.06
4.50
270.3
0.82
0.59
44.3
MDTv2-XL/2 [ 14 ]
SD-VAE
1.58
4.52
314.7
0.79
0.65
–
Table 2: Comparison with state-of-the-art generators on ImageNet 256×256 . With guidance; FID-50K, ADM evaluator. For each released generator, the first row uses its original decoder D0 and the second row SAD (GAN discrepancy) on identical latents. Bold marks a result no worse than D0 . FD CLIP uses a held-out reference and is computed by us, for other generators from their released samples or weights. † Class-balanced labels.
Method
Extra sampling
Disc. real images
FID ↓
sFID ↓
IS ↑
D0
–
–
2.30
4.51
277.8
DG [ 21 ]
+ 20%
1.28M
2.01
4.49
266.5
SAD (ours)
0
5k
2.13
4.25
280.5
DG + SAD (ours)
+ 20%
1.28M
1.86
4.02
269.3
Table 3: Comparison of generator-side and decoder-side correction. DiT-XL/2 in the official setting at guidance 1.5 (FID-50K, ADM evaluator). The best results are in bold and the second best are underlined.
Method
FID ↓
FD CLIP↓
rFID ↓
PSNR ↑
Original decoder D0
13.32
51.4
2.63
23.91
Replacing or re-training the decoder
sd-vae-ft-mse [ 43 ]
13.40
51.8
2.91
24.75
Consistency Decoder [ 38 ]
13.18
57.3
3.34
23.31
Fine-tuning, ℓ1 +LPIPS+GAN
13.40
48.6
7.02
24.51
Robustness to perturbed latents
Table 4: Comparison with decoder adaptation methods. DiT-XL/2 in the ablation setting. Noise-robust, paired and FDDA are our re-implementations. The best results are in bold and the second best are underlined. ∗ Trained with Inception, the FID network.
Decoder
FID ↓
FD CLIP↓
CLIP ↑
Rec. ↑
rFID ↓
Pick
D0
34.73
209.1
26.03
0.286
4.28
–
GAN, λ=0.01
23.55
141.6
26.91
0.416
4.29
48%
MMD, λ=0.1
28.35
150.4
26.19
0.343
4.26
67%
Real images
–
–
25.69
–
–
18%
Table 5: Text-to-image generation. PixArt- α on COCO (10k samples). Pick: share of samples preferred by PickScore over D0 (same latent; for real val2017 images, same caption). All tested weights are in Appendix D .
#
Variant
FID ↓
FD CLIP↓
rFID ↓
PSNR ↑
0
D0
13.32
51.4
2.63
23.91
Ablation on GSA
1
w/o GSA (EAP only)
13.31
49.9
2.65
23.92
2
GSA on E(x) instead of zgen
13.59
51.5
7.70
23.90
3
GSA with MMD
12.68
41.6
2.57
23.92
4
GSA with FD
12.79
41.8
2.59
23.91
Table 6: Ablation study on the modules of SAD. DiT-XL/2 in the ablation setting.
Figure 3: Sensitivity to the loss weights. DiT-XL/2 in the ablation setting. (a) Weight of LEAP with λ=0.05 . (b) Weight λ of LGSA with the weight of LEAP fixed to 1. Defaults: 1 and 0.05.
Figure 4: Qualitative comparison on DiT-XL/2. The same generated latents are decoded by D0 and by SAD. The bottom row shows their per-pixel absolute difference (brighter is larger).
Figure 5: Effect of where GSA is applied. DiT-XL/2 in the ablation setting. (a) Generation (FID on 10k samples) and reconstruction quality of all trained decoders. (b) Radially averaged power spectrum of decoded samples relative to real images.
Statistic
Encoder
Generated
Relative change ∥P(z)−z∥2/∥z∥2
0.017
0.016
AUC of this change
–
0.49
AUC of a small CNN on z
–
0.96
AUC of a small CNN on P(z)
–
0.95
Table 7: Distribution test of generated latents. DiT-XL/2, 10k latents each; P(z)=E(D0(z)) . AUC: encoder vs . generated.
Statistic
Real
D0
SAD
Saturation (HSV)
0.323
0.375
0.324
Brightness (HSV)
0.526
0.539
0.531
Colorfulness [ 16 ]
42.0
47.8
41.3
Clipped highlights (%)
4.8
9.4
5.9
Clipped shadows (%)
1.4
4.1
1.9
Table 8: Color and exposure statistics. DiT-XL/2, 50-step DPM-Solver++, 3k samples and 3k held-out real images. Clipped: pixels with a channel ≥250 or all channels ≤5 .
Figure 6: Per-sample correction against the density ratio. DiT-XL/2, 10k encoder and 10k generated latents. A classifier trained to separate generated from encoder latents scores every latent, and its logit ranks latents by r(z) . Each point is one quintile of a set by this score; y : mean pixel change ∣Dϕ(z)−D0(z)∣ made by SAD.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
trainable params
extra params at inference
extra GFLOPs per image
training (GPU-h)
SAD
49.5M
0
0
0.9
Pixel refiner
6.8M
6.8M
95 ( +15% )
0.6
Appendix
Table 9: Training and inference cost of SAD.
Method
FID ↓
sFID ↓
IS ↑
FD CLIP↓
Original decoder D0
2.30
4.51
277.8
42.8
Noise-robust decoder [ 63 ]
2.31
4.64
277.0
43.1
Paired post-training [ 42 , 12 ]
2.69
7.67
266.7
54.9
FDDA [ 13 ] ∗
2.02
4.84
274.9
48.9
FDDA [ 13 ] ∗ + EAP
2.27
4.14
279.4
51.2
SAD-GAN (ours)
2.13
4.25
280.5
36.8
Appendix
Table 10: Comparison with decoder adaptation methods in the official setting. DiT-XL/2 at guidance 1.5 (FID-50K, ADM evaluator; FD CLIP with a held-out reference). All decoders are trained on the same 40k generated latents. The best results are in bold and the second best are underlined. ∗ Inception, the FID network, is part of its training objective.
D0→ SAD
FID ↓
FD CLIP↓
rFID ↓
Pick
DiT-XL/2 on ImageNet (50k samples)
DPM-50, GAN
3.24 → 2.28
48.3 → 39.4
2.63 → 2.56
51%
DDPM-250 (official), GAN
2.30 → 2.13
42.8 → 36.8
2.63 → 2.69
50%
DDPM-250 (official), MMD
2.30 → 2.17
42.8 → 39.4
2.63 → 2.66
52%
LightningDiT with VA-VAE on ImageNet (50k samples)
LightningDiT-XL, GAN
1.38 → 1.23
29.4 → 28.2
1.18 → 1.17
70%
Appendix
Table 11: Results across generators and datasets. Each decoder is trained on latents of its own generator. ODE pool: training latents from the 50-step Heun ODE instead of the deployed SDE sampler. Pick: share of samples for which PickScore prefers SAD over D0 (ImageNet prompts: “a photo of a class ”).
Sampling setting
FID ↓
FD CLIP↓
D0
SAD
D0
SAD
cfg 1.5, 50 steps (train)
13.32
12.37
51.4
44.2
cfg 1.5, 20 steps
13.82
12.68
52.8
42.0
cfg 1.0, 50 steps
18.68
18.39
68.8
68.5
cfg 4.0, 50 steps
26.50
25.75
70.2
49.5
Appendix
Table 12: Transfer of one decoder to other sampler settings. SAD values are means over three training seeds.
Scale
Interval
FID ↓
FD CLIP↓
D0
SAD
D0
SAD
1.5
–
13.32
12.37
51.4
44.2
1.75
–
15.31
14.34
49.9
40.1
2.0
✓
13.53
12.65
49.6
40.7
2.5
✓
14.98
14.19
50.1
38.4
3.0
✓
16.51
15.82
51.9
37.9
Appendix
Table 13: SAD combined with guidance tuning. SAD values are means over three training seeds.
Figure 7: Training dynamics and data efficiency. DiT-XL/2 in the ablation setting. (a) GSA weight λ relative to the default. (b, c) FID and rFID during training. (d) Number of cached latents or real images.
Decoder
KID ↓
FD SigLIP↓
CMMD ↓
HPS
PickScore
HPS win
Pick win
DiT-XL/2, DPM-Solver++ (50 steps)
D0
1.17
10.0
0.227
0.235
20.42
–
–
LGSA on E(x) , GAN
1.25
9.6
0.267
0.231
20.39
25%
45%
SAD-GAN
0.59
8.0
0.267
0.231
20.42
31%
51%
SAD-MMD
0.72
7.9
0.187
0.235
20.42
48%
50%
DiT-XL/2, official code (DDPM-250)
Appendix
Table 14: Evaluation with metrics unused in training. KID ×103 , FD in SigLIP space, CMMD, HPS v2.1 and PickScore. Win: share of samples for which the preference model prefers the decoder (or the real image with the same caption) over D0 .
Statistic
Encoder
Generated (1.0)
Generated (1.5)
Relative change ∥P(z)−z∥2/∥z∥2
0.0168
0.0160
0.0162
AUC of this change
–
0.48
0.49
FID of D0(z) (encoder: rFID)
2.63
18.62
12.48
FID of D0(P(z)) (encoder: rFID)
3.91
20.67
13.40
AUC of a small CNN on z
–
0.94
0.96
AUC of a small CNN on P(z)
–
0.93
0.95
Appendix
Table 15: Full distribution test of generated latents. DiT-XL/2 in the ablation setting, 10k latents per set, generated latents at guidance 1.0 and 1.5. P(z)=E(D0(z)) with the posterior mean. AUC: encoder vs . generated latents.
Figure 8: PixArt- α samples decoded by D0 and by SAD-GAN at two values of λ . The last two examples show the failure modes at the larger λ .
Decoder
FID ↓
FD CLIP↓
CLIP Sc.
Rec.
rFID ↓
Pick win
D0
34.73
209.1
26.03
0.286
4.28
–
GAN, λ=0.005
24.67
155.9
27.21
0.400
4.28
39%
GAN, λ=0.01
23.55
141.6
26.91
0.416
4.29
48%
GAN, λ=0.02
25.51
135.6
26.40
0.365
4.40
48%
GAN, λ=0.05
20.45
116.8
26.82
0.408
4.91
24%
MMD, λ=0.1
28.35
150.4
26.19
0.343
4.26
67%
Appendix
Table 16: SAD on PixArt- α for all tested λ . COCO, 10k samples. Pick win: share of samples preferred by PickScore over D0 (same latent; for real val2017 images, same caption).
Figure 9: Qualitative comparison on other generators. (a) LightningDiT-XL, (b) REPA-E and (c) PixArt- α . The same generated latents are decoded by D0 and by SAD. The bottom row shows their per-pixel absolute difference (brighter is larger).
Figure 10: Additional samples decoded by D0 and SAD. DiT-XL/2 in the ablation setting (guidance 1.5); the last two samples use guidance 4.0, which is not seen in training. The third and sixth rows show the per-pixel absolute difference (brighter is larger).
Latent space alignment maps learned representations into compatible coordinates to enable comparison, transfer, and interoperability across models or data sources.