Patch-Derived or Register Tokens? Comparing Variable-Length Image Tokenizers

Philipp Schenkel; Tobias Schürmann; Omer Sahin Tas

cs.CVOct 24, 2026

Patch-Derived or Register Tokens? Comparing Variable-Length Image Tokenizers

Authors: Philipp Schenkel, Tobias Schürmann, Omer Sahin Tas

Organizations: FZI Research Center for Information Technology · Karlsruhe Institute of Technology

Abstract

Image tokenizers convert pixels into compact latent sequences for efficient generative modeling. Most approaches arrange latents on a downsampled two-dimensional grid, tying sequence length to image resolution. We compare two variable-length strategies: computationally efficient patch-derived tokens, which inherit explicit spatial organization, and more demanding register tokens, which have no predefined spatial layout. Our reconstruction results reveal that patch-derived tokens perform substantially better under extreme compression, whereas register tokens achieve slightly better reconstruction quality at intermediate budgets. At high token budgets, both approaches achieve similar reconstruction quality. For downstream generative tasks, register tokens outperform patch-derived tokens. Adding representation alignment with external pretrained visual encoders substantially improves the generative performance of patch-derived tokens, allowing them to match or outperform register tokens depending on the token budget. We further find that both approaches learn spatially localized representations. Patch-derived tokens largely retain their explicitly imposed spatial organization, while register tokens develop an implicit two-dimensional spatial structure despite having no predefined layout. Representation alignment reduces this spatial organizational constraint, enabling individual tokens to exert a stronger non-local influence.

Figures & tables

Figure 1: Qualitative reconstructions from two trained image tokenizers: one using patch-derived tokens and the other using register tokens. Both models progressively recover finer visual details as more tokens become available, while organizing information differently across the latent sequence.

Figure 2: Overview of the investigated architectural designs, with patch-derived tokens on the left and register tokens on the right. Both architectures utilize nested dropout to obtain variable-length image representations. Patch-derived tokens are obtained by transforming image patches into the token representations, whereas register tokens are obtained by transforming dedicated learnable embeddings.

Token type	Encoder throughput	Decoder throughput
Token type	Encoder throughput	1	32	64	128	256
Patch-derived	$1622.9$	1756.8 (constant)
Register	$901.3$	$1750.9$	$1573.9$	$1452.6$	$1217.8$	$932.7$

Table 1: Inference-speed comparison on

256\times 256

images with a batch size of 128 on a single NVIDIA A100 GPU. The patch-derived token encoder and decoder each process 256 tokens in total, independent of the token budget, as image patches get transformed into tokens and vice versa. The register token encoder processes 256 image-patch embeddings and 256 latent tokens, causing a throughput gap to patch-derived tokens. The register token decoder processes between 257 and 512 tokens in total. Throughput is reported in images per second.

Token type	REPA	Metric	Token count
Token type	REPA	Metric	1	4	8	16	32	64	128	192	256
Register	No	rFID $\downarrow$	344.65	223.42	77.93	7.83	3.73	2.55	1.79	1.46	1.29
		LPIPS $\downarrow$	0.600	0.560	0.518	0.455	0.395	0.341	0.293	0.271	0.258
		MAE $\downarrow$	0.230	0.185	0.155	0.129	0.112	0.097	0.084	0.077	0.074
		PSNR $\uparrow$	11.05	12.55	13.91	15.19	16.14	17.22	18.36	19.00	19.35
		SSIM $\uparrow$	0.161	0.226	0.284	0.340	0.381	0.431	0.487	0.513	0.529

Table 2: Reconstruction metrics for the investigated variable-length tokenizers. REPA indicates whether representation alignment was applied during tokenizer training. All metrics are evaluated on the ImageNet

256\times 256

validation split.

Token type	Model	REPA	Token count
Token type	Model	REPA	16	32	64	128	192	256
Register	LlamaGen-B	No	$10.10$	$6.31$	$5.34$	$5.04$	$4.95$	$4.95$
	LlamaGen-B	Yes	$9.65$	$8.90$	$9.94$	$13.70$	$15.20$	$15.30$
	LlamaGen-L	No	$9.02$	$5.38$	$4.32$	$3.83$	$3.64$	$3.61$
	LlamaGen-L	Yes	$7.22$	$5.34$	$4.08$	$3.22$	$3.11$	$3.11$
Patch-derived	LlamaGen-B	No	$9.90$	$7.85$	$7.03$	$6.46$	$6.39$	$6.38$

Table 3: Class-conditional gFID (

\downarrow

) on ImageNet-1K for LlamaGen models trained with variable-length tokenizers. REPA indicates whether representation alignment was applied during tokenizer training. Lower values are better.

Panel of Figure 3 — Figure 3: Qualitative effect of corrupting a single token. Within each block, columns show the original reconstruction, the reconstruction after corrupting the indicated token, and their pixel-wise difference. For both patch-derived and register tokens, the corruption predominantly causes spatially localized changes. Although REPA reduces the strength of this locality, local effects remain visible.

Panel of Figure 4 — Figure 4: Spatial influence of patch-derived tokens and register tokens, trained with and without REPA. Each heatmap shows the pixel-wise change in squared reconstruction error when one token is replaced by a randomly sampled codebook entry. Within each panel, heatmaps are arranged by token index in raster-scan order from top left to bottom right. Without REPA, both tokenization schemes exhibit predominantly localized influence, although some register tokens affect broader image regions. With REPA, patch-derived tokens remain largely localized, whereas many register tokens exhibit substantially more spatially diffuse influence. The color scale uses a power-law normalization to improve visibility. Results are averaged over ImageNet-1K validation images.

Table 8

Image Tokenization

Image tokenization converts images into compact discrete or continuous visual units for efficient generation, compression, and multimodal reasoning.

See all 56 papers

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation
Mar 17, 2026Yunpeng Qu, Kaidong Zhang, Yukang Ding +2Image Tokenization Image Reconstruction
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
Balancing Image Compression and Generation with Bootstrapped Tokenization
Jun 4, 2026Haozhe Chi, Jinghan Li, Hao Jiang +4Learned Image Compression Image Tokenization
Peking University · Central Media Technology Institute, Huawei
Test-Time Registers as Global Priors for Tokenized Image Generation
Jul 18, 2026Cheng-Yao Hong, Yifan Wang, Yuewei Lin +1Image Tokenization Vision Transformer
Stony Brook University · Brookhaven National Laboratory

Mar 17, 2026J/K move · Enter open · S save

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

Yunpeng Qu, Kaidong Zhang, Yukang Ding, Ying Chen+1

Tsinghua University, Beijing, China · Alibaba Group, Beijing, China

Image Tokenization Image Reconstruction Autoregressive Image Generation

Open paper AskPDF arXiv

Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.

Abstract

Image Tokenization

Explore similar work

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

Balancing Image Compression and Generation with Bootstrapped Tokenization

Test-Time Registers as Global Priors for Tokenized Image Generation