Visual tokenizers increasingly inject semantic supervision into latent spaces to make downstream diffusion easier. Yet how these semantics should be organized to facilitate denoising remains underexplored. In this paper, we define the semantic recovery objective: the denoising process should recover the semantic content of the clean image from noisy latent, and a good tokenizer should make it easier. Existing approaches train a projector to predict the semantics directly from the noisy latent. We argue that this predicts the average of clean-image semantics, whereas what really needs to be aligned is the semantics of averaged clean latents. More importantly, we demonstrate that the semantic recovery error orthogonally decomposes into the error of the optimal semantic prediction directly from the noisy latent and the error between these two predictions. We therefore identify their consistency as the missing requirement and call it Semantic Affine Consistency (SAC). To examine whether this overlooked requirement is closely related to downstream generation, we introduce M_SAC, a tokenizer-side proxy for SAC. Across the evaluated tokenizers and diffusion model scales, M_SAC closely tracks generation quality, reaching a Pearson correlation of 0.960 with SiT-XL gFID, thereby motivating SAC-guided tokenizer training. We then introduce AffineTok, which promotes SAC through two complementary, training-only components. Global Semantic Coordination Token (GSCT) coordinates the semantic organization of clean latents, keeping semantic averaging meaningful, while Posterior-Mean Semantic Alignment (PMSA) predicts posterior-mean latents from noisy inputs and supervises their semantics. On ImageNet 256, compared with the baseline, AffineTok reduces gFID by 26% at 20 epochs and, with continued training, achieves a new state-of-the-art gFID of 1.21 without classifier-free guidance and 1.10 with guidance.
Figures & tables
Figure 1: Semantic Affine Consistency for diffusion-friendly visual tokenizers. (a) Existing methods predict semantics directly from a noisy latent, which are the average of clean-image semantics . Diffusion instead predicts the posterior-mean clean latent and then decodes its semantics, which are the semantics of the averaged clean latent . SAC requires these two predictions to agree, a condition absent from prior supervision. (b) Our tokenizer-side metric MSAC strongly correlates with SiT-XL gFID across 14 visual tokenizers, reaching a Pearson correlation of 0.960 . (c) SAC-guided training improves diffusability, reaching 1.21 gFID at 600 epochs without CFG.
Figure 2: Overall framework of AffineTok. The input image is processed in parallel by a frozen VFM and a trainable tokenizer. Its encoder jointly maps patch embeddings and a prepended learnable GSCT to a global output and clean patch latents z0 . After z0 is noised into zt , the resulting global and noisy patch representations receive semantic supervision; the global output is then discarded, while zt is decoded for reconstruction. Along the PMSA path, Gpost maps zt to a posterior-mean estimate, and Spost maps the estimate to semantics. Only clean patch latents are retained downstream.
Figure 3: SAC probing. We compute MSAC using neighboring latent interpolation to measure the agreement between the average of semantics and the semantics decoded from the averaged latent.
Figure 4: PMSA training. PMSA learns a posterior-mean predictor, calibrates a semantic projector on clean latents, and supervises the semantics recovered from predicted clean latents.
Figure 5: Targeted semantic probes for GSCT and PMSA (lower is better). Left: Cross-split semantic-axis inconsistency rates for RecTok and +GSCT across three angular thresholds. Right: Top-10 semantic retrieval miss rates for RecTok and +GSCT+PMSA under three severe noise levels.
Tokenizer-side metric
SiT-XL
SiT-B
Pearson
Spearman
Kendall
Pearson
Spearman
Kendall
iFID
0.896
0.881
0.758
0.856
0.830
0.692
MSACℓ2
0.933
0.943
0.846
0.896
0.874
0.744
MSAC
0.960
0.974
0.912
0.946
0.929
0.821
Table 1: Correlation between tokenizer-side metrics and downstream gFID. We report Pearson, Spearman, and Kendall correlations across 14 tokenizers for SiT-XL and 13 tokenizers for SiT-B.
Method
Epochs
Params
Without guidance
With guidance
gFID ↓
IS ↑
Prec. ↑
Rec. ↑
gFID ↓
IS ↑
Prec. ↑
Rec. ↑
Diffusion-centric systems
DiT Peebles and Xie (2023)
1400
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
MaskDiT Zheng et al. (2023)
1600
675M
5.69
177.9
0.74
0.60
2.28
276.6
0.80
0.61
SiT Ma et al. (2024)
1400
675M
8.61
131.7
0.68
0.67
2.06
270.3
0.82
0.59
MDTv2 Gao et al. (2023)
1080
675M
–
–
–
–
1.58
314.7
0.79
0.65
Table 2: Class-conditional generation on ImageNet 256 . Epochs and Params denote downstream generator training epochs and parameter count, respectively; a dash denotes an unreported result. Guided results use the guidance scheme reported by each source. Prior-method values are taken from RecTok ( Shi et al., 2026b ) . Ours achieves state-of-the-art gFID both without and with guidance.
Table 8
Tokenizer
rFID ↓
PSNR ↑
LPIPS ↓
L.P. Acc. ↑
MSAC↓
Pooled
Flat
RecTok
0.48
26.16
0.1016
62.32
58.22
52.120
Ours
0.49
25.83
0.1086
72.63
68.13
51.495
Table 5: Tokenizer reconstruction and representation quality on ImageNet-1K. Following RecTok ( Shi et al., 2025 ) , L.P. Acc. denotes top-1 linear-probing accuracy on frozen latents; Pooled averages the patch latents, while Flat vectorizes them. Best results are in bold.
An effective visual tokenizer must preserve image fidelity while producing latents that downstream diffusion models can learn efficiently. Prior work improves latent diffusability through semantic alignment or joint tokenizer--diffusion training. Revisiting REPA-E, a representative approach, we find that removing its diffusion objective and dedicated generative blocks while retaining semantic prediction from corrupted latents yields comparable downstream generation performance under a matched setting. This observation motivates a general principle of robust prediction: generation-relevant information should remain recoverable from corrupted latents. We instantiate this principle with two complementary objectives. RobustAlign predicts pretrained semantic features from corrupted latents to preserve high-level structure, while RobustRecon reconstructs the clean image to retain fine-grained spatial and appearance information. Together, they form Robust Alignment and Reconstruction VAE (RoAR-VAE). Our analysis suggests that robust prediction improves diffusability by making useful information more accessible in high-variance latent directions, which diffusion models tend to learn earlier. Empirically, RoAR-VAE improves average generation benchmark performance by 2.19 points over the strongest baseline tokenizer under the same training budget, while improving small-text reconstruction accuracy by 16.93 percentage points over FLUX.1-VAE. These results demonstrate that RoAR-VAE achieves a favorable balance between reconstruction fidelity and downstream generative learnability.
Latent diffusion models generate images as codes of an image tokenizer, whose decoder turns them into the pixel image. Recent tokenizers improve how well these codes reconstruct an image and how easily a generator learns them. Yet a generator does not reproduce the codes of training images, and when a generated code lands outside what the decoder expects, the decoded image contains imperfections. Current methods make the decoder more robust to generated codes by adding noise during training. However, generated codes lie between the training codes and resemble their mixture more than one of them plus noise. To improve generated images, decoders should therefore be trained on codes in the space between training samples. We propose InterpTok, which fine-tunes MAETok to decode mixtures of image codes into natural images. Since no target image exists between two training samples, a discriminator judges whether each decoded mixture looks as natural as its two sources. A mixing ratio loss requires the decoded mixture to keep the mixing ratio of the two codes. Together, they prevent collapse to a trivial solution and yield a sharp and natural composition of both images. InterpTok keeps MAETok's reconstruction quality and improves interpolation quality more than threefold, the best of all tokenizers in a recent benchmark. Beyond interpolation, InterpTok improves the generation FID of a matched diffusion transformer from 11.61 to 8.48 without guidance, and even reaches 8.11 at an eighth of the sampling steps, where the generator on MAETok's codes worsens. The generator trained on our tokenizer also reaches the final result of the one trained on MAETok at half the training steps. Our findings show that the space between image codes can and should be explicitly supervised rather than ignored.
Ole Beißwenger, Jan-Niklas Dihlmann, Hendrik Lensch
Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness.
Luyuan Zhang, Siyuan Li, Zedong Wang +7
1Tsinghua University · 2Westlake University · 3Zhejiang University +3