Image tokenizers convert pixels into compact latent sequences for efficient generative modeling. Most approaches arrange latents on a downsampled two-dimensional grid, tying sequence length to image resolution. We compare two variable-length strategies: computationally efficient patch-derived tokens, which inherit explicit spatial organization, and more demanding register tokens, which have no predefined spatial layout. Our reconstruction results reveal that patch-derived tokens perform substantially better under extreme compression, whereas register tokens achieve slightly better reconstruction quality at intermediate budgets. At high token budgets, both approaches achieve similar reconstruction quality. For downstream generative tasks, register tokens outperform patch-derived tokens. Adding representation alignment with external pretrained visual encoders substantially improves the generative performance of patch-derived tokens, allowing them to match or outperform register tokens depending on the token budget. We further find that both approaches learn spatially localized representations. Patch-derived tokens largely retain their explicitly imposed spatial organization, while register tokens develop an implicit two-dimensional spatial structure despite having no predefined layout. Representation alignment reduces this spatial organizational constraint, enabling individual tokens to exert a stronger non-local influence.
Figures & tables
Figure 1: Qualitative reconstructions from two trained image tokenizers: one using patch-derived tokens and the other using register tokens. Both models progressively recover finer visual details as more tokens become available, while organizing information differently across the latent sequence.
Figure 2: Overview of the investigated architectural designs, with patch-derived tokens on the left and register tokens on the right. Both architectures utilize nested dropout to obtain variable-length image representations. Patch-derived tokens are obtained by transforming image patches into the token representations, whereas register tokens are obtained by transforming dedicated learnable embeddings.
Token type
Encoder throughput
Decoder throughput
1
32
64
128
256
Patch-derived
1622.9
1756.8 (constant)
Register
901.3
1750.9
1573.9
1452.6
1217.8
932.7
Table 1: Inference-speed comparison on 256×256 images with a batch size of 128 on a single NVIDIA A100 GPU. The patch-derived token encoder and decoder each process 256 tokens in total, independent of the token budget, as image patches get transformed into tokens and vice versa. The register token encoder processes 256 image-patch embeddings and 256 latent tokens, causing a throughput gap to patch-derived tokens. The register token decoder processes between 257 and 512 tokens in total. Throughput is reported in images per second.
Token type
REPA
Metric
Token count
1
4
8
16
32
64
128
192
256
Register
No
rFID ↓
344.65
223.42
77.93
7.83
3.73
2.55
1.79
1.46
1.29
LPIPS ↓
0.600
0.560
0.518
0.455
0.395
0.341
0.293
0.271
0.258
MAE ↓
0.230
0.185
0.155
0.129
0.112
0.097
0.084
0.077
0.074
PSNR ↑
11.05
12.55
13.91
15.19
16.14
17.22
18.36
19.00
19.35
SSIM ↑
0.161
0.226
0.284
0.340
0.381
0.431
0.487
0.513
0.529
Table 2: Reconstruction metrics for the investigated variable-length tokenizers. REPA indicates whether representation alignment was applied during tokenizer training. All metrics are evaluated on the ImageNet 256×256 validation split.
Token type
Model
REPA
Token count
16
32
64
128
192
256
Register
LlamaGen-B
No
10.10
6.31
5.34
5.04
4.95
4.95
LlamaGen-B
Yes
9.65
8.90
9.94
13.70
15.20
15.30
LlamaGen-L
No
9.02
5.38
4.32
3.83
3.64
3.61
LlamaGen-L
Yes
7.22
5.34
4.08
3.22
3.11
3.11
Patch-derived
LlamaGen-B
No
9.90
7.85
7.03
6.46
6.39
6.38
Table 3: Class-conditional gFID ( ↓ ) on ImageNet-1K for LlamaGen models trained with variable-length tokenizers. REPA indicates whether representation alignment was applied during tokenizer training. Lower values are better.
Figure 3: Qualitative effect of corrupting a single token. Within each block, columns show the original reconstruction, the reconstruction after corrupting the indicated token, and their pixel-wise difference. For both patch-derived and register tokens, the corruption predominantly causes spatially localized changes. Although REPA reduces the strength of this locality, local effects remain visible.
Figure 4: Spatial influence of patch-derived tokens and register tokens, trained with and without REPA. Each heatmap shows the pixel-wise change in squared reconstruction error when one token is replaced by a randomly sampled codebook entry. Within each panel, heatmaps are arranged by token index in raster-scan order from top left to bottom right. Without REPA, both tokenization schemes exhibit predominantly localized influence, although some register tokens affect broader image regions. With REPA, patch-derived tokens remain largely localized, whereas many register tokens exhibit substantially more spatially diffuse influence. The color scale uses a power-law normalization to improve visibility. Results are averaged over ImageNet-1K validation images.
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens. The mix of information of different granularity also complicates the training of generators. This paper introduces SelfBootTok, a method that resolves this by cleanly decomposing information into global and local token groups. Through self-bootstrapped learning, the model predicts local details exclusively from global tokens, shifting the burden of visual details from the generator to the tokenizer. Consequently, our generator is far more efficient, requiring only global tokens and reducing computation by approximately 40%, while delivering superior reconstruction and generation. Moreover, this paradigm scales elegantly: by leveraging more data or parameters to self-supervise local representation learning, SelfBootTok achieves a new state-of-the-art gFID score of 1.56 using only 64 tokens.
Haozhe Chi, Jinghan Li, Hao Jiang +4
Peking University · Central Media Technology Institute, Huawei
Attention-based models often develop attention sinks, where a small number of tokens repeatedly attract attention and accumulate unusually large activations. In vision transformers, these outliers are closely related to registers, which have been diagnostically linked to global, low-frequency image structure. Existing work has largely studied registers through interpretability analyses and linear probes, leaving open whether they can be operationalized as plug-and-play signals for generation without retraining. We revisit this question in tokenized image generation. Using OpenCLIP and DINOv2 on ImageNet, we find that test-time register features exhibit stronger low-frequency concentration than both [CLS] readouts and patch-mean features, and show a consistent (albeit moderate) correlation with pixel-space DCT low-frequency energy. Motivated by these diagnostics, we introduce RegToken, a training-free procedure that converts register structure into a small set of global prior tokens by (i) NFN-based layer localization, (ii) TokenRank-guided subspace extraction, and (iii) a projection-and-conservation update on the register subspace. Inserted into a frozen compact 1D token generation pipeline, RegToken improves ImageNet generation and alignment metrics (e.g., FID-5k 20.5 to 20.1, SigLIP 3.6 to 3.9) without modifying pretrained weights, and accelerates test-time optimization (Steps@τ 74 to 52). Overall, our results suggest that structures often viewed as attention artifacts can be repurposed as lightweight global priors for tokenized generation.
Cheng-Yao Hong, Yifan Wang, Yuewei Lin +1
Stony Brook University · Brookhaven National Laboratory