MacTok: Robust Continuous Tokenization for Image Generation

Hengyu Zeng; Xin Gao; Guanghao Li; Yuxiang Yan; Jiaoyang Ruan; Junpeng Ma; Haoyu Albert Wang; Jian Pu

cs.CVMar 31, 2026

MacTok: Robust Continuous Tokenization for Image Generation

Authors: Hengyu Zeng, Xin Gao, Guanghao Li, Yuxiang Yan, Jiaoyang Ruan, Junpeng Ma, Haoyu Albert Wang, Jian Pu

Organizations: Fudan University

Abstract

Continuous image tokenizers enable efficient visual generation, and those based on variational frameworks can learn smooth, structured latent representations through KL regularization. Yet this often leads to posterior collapse when using fewer tokens, where the encoder fails to encode informative features into the compressed latent space. To address this, we introduce MacTok, a Masked Augmenting 1D Continuous Tokenizer that leverages image masking and representation alignment to prevent collapse while learning compact and robust representations. MacTok applies both random masking to regularize latent learning and DINO-guided semantic masking to emphasize informative regions in images, forcing the model to encode robust semantics from incomplete visual evidence. Combined with global and local representation alignment, MacTok preserves rich discriminative information in a highly compressed 1D latent space, requiring only 64 or 128 tokens. On ImageNet, MacTok achieves a competitive gFID of 1.44 at 256 $\times$ 256 and a state-of-the-art 1.52 at 512 $\times$ 512 with SiT-XL, while reducing token usage by up to 64 $\times$ . These results confirm that masking and semantic guidance together prevent posterior collapse and achieve efficient, high-fidelity tokenization.

Figures & tables

Figure 1: Effect of random masking in continuous tokenizers. Left: plain KL-VAE, latent token masking, and image token masking, with only the latter preventing posterior collapse. Right: collapsed latent space shows poor structure, while the uncollapsed one yields well-structured and diverse representations.

Figure 2: Generation results produced by generative models with MacTok using 64 and 128 tokens on ImageNet at 256 $\times$ 256 and 512 $\times$ 512.

Figure 3: Generation performance of MacTok with varying mask ratios sampled up to $M$ as detailed in Sec. 3.2 . The orange star corresponds to random and semantic mask with equal probability.

Figure 4: Overview of the MacTok framework. Top: Transformer-based encoder and decoder operating on image, latent, and mask tokens. Bottom left: DINO-guided image masking introduces semantic priors. Bottom center: Global and local representation alignment between latent and pretrained visual representations. Bottom right: Discriminator and perceptual networks provide auxiliary supervision.

							w/o CFG		w/ CFG
Method	# Params (G)	Tok. Model	Token Type	# Params (T)	# Tokens↓	Tok. rFID↓	gFID↓	IS↑	gFID↓	IS↑
Auto-regressive
ViT-VQGAN [ 58 ]	1.7B	VQ	2D	64M	1024	1.28	4.17	175.1	-	-
RQ-Trans. [ 28 ]	3.8B	RQ	2D	66M	256	3.20	-	-	3.80	323.7
MaskGIT [ 3 ]	227M	VQ	2D	66M	256	2.28	6.18	182.1	-	-
LlamaGen-3B [ 45 ]	3.1B	VQ	2D	72M	576	2.19	-	-	2.18	263.3

Table 1: System-level comparison on ImageNet 256

\times

256 conditional generation. “# Params (G)” denotes generator parameters; “Tok. Model” refers to the tokenizer model type; “Token Type” indicates 1D or 2D tokenization; “# Params (T)” denotes tokenizer parameters; and “# Tokens” represents the number of latent tokens. ‡ denotes methods that rely on pretrained vision models.

							w/o CFG		w/ CFG
Method	# Params (G)	Tok. Model	Token Type	# Params (T) †	# Tokens↓	Tok. rFID↓	gFID↓	IS↑	gFID↓	IS↑
GAN
BigGAN [ 3 ]	-	-	-	-	-	-	-	-	8.43	177.9
StyleGAN-XL [ 24 ]	168M	-	-	-	-	-	-	-	2.41	267.7
Auto-regressive
MaskGIT [ 3 ]	227M	VQ	2D	66M	1024	1.97	7.32	156.0	-	-

Table 2: System-level comparison on ImageNet 512

\times

512 conditional generation. SiT-XL trained with MacTok achieves state-of-the-art generation performance using only 64 and 128 tokens (

\dagger

: Large decoder for fair comparison;

\ddagger

: relies on pretrained vision models).

		Tok.			SiT-B
Tokenizer	#Tokens↓	rFID↓	PSNR↑	SSIM↑	gFID↓	IS↑
VA-VAE	256	0.28	26.30	0.846	4.33	222.1
MAETok	128	0.48	23.61	0.763	4.77	243.2
SoftVQ-VAE	64	0.88	22.13	0.706	4.09	256.9
SD-VAE	1024	0.61	26.04	0.834	7.66	187.5
MAR-VAE	256	0.53	-	-	6.26	177.5

Table 3: Comparison of continuous tokenizers. MacTok attains a better balance between compression and reconstruction quality, while delivering the best generation performance. All generation results are reported with optimal CFG scales.

Panel of Figure 5 — Figure 5: Visualization of latent space from (a) Collapsed; (b) MacTok-128 trained without representation alignment; (c) MacTok-128

Panel of Figure 6 — Figure 6: Linear probing accuracy (a) of ImageNet-1k val. and generation performance (b) of MacTok with training steps.

	Tok.			SiT-B
$M$	rFID↓	PSNR↑	SSIM↑	gFID↓	IS↑
0.4	0.49	24.95	0.809	15.49	78.6
0.5	0.54	25.02	0.812	14.87	81.1
0.6	0.57	24.95	0.808	14.79	81.8
0.7	0.56	24.93	0.808	14.59	82.5
0.8	0.59	24.89	0.805	14.92	80.8

Table 4: Ablation on maximum mask ratio

M

(w/o Decoder Fine-tuning). MacTok is evaluated over mask ratios from 0.4 to 0.8 and different DINO-guided semantic masking settings: “dino 100%” denotes full use of DINO-guided semantic masking, while “dino 50%” applies random and semantic masking with equal probability. Generation performance is reported without CFG.

Setup	Tok			SiT-B
MacTok	rFID↓	PSNR↑	SSIM↑	gFID↓	IS↑
+ random mask	0.58	24.30	0.779	6.01	234.8
+ local alignment	0.44	25.02	0.806	3.53	241.9
+ semantic mask	0.43	24.97	0.805	3.32	249.2
+ global alignment	0.43	25.03	0.806	3.15	258.3

Table 5: Ablation of different modules (w/ Decoder Fine-tuning). We report the impact of each module on MacTok’s reconstruction and generation performance with optimal CFG scales.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Comparison of different masking strategies of the KL loss curve.

Panel of Figure 8 — Figure 8: Visualization of laten space from (a) MacTok-64 trained without Representation alignment; (b) MacTok-64

Table 6: Ablation studies of decoder fine-tuning and model size, showing their effects on MacTok’s performance

							w/o CFG				w/ CFG
Method	# Params(G)	Tok. Model	Token Type	# Params(T)	#Tokens↓	Tok. rFID↓	gFID↓	IS↑	Prec↑	Recall↑	gFID↓	IS↑	Prec↑	Recall↑
Auto-regressive
ViT-VQGAN [ 58 ]	1.7B	VQ	2D	64M	1024	1.28	4.17	175.1	-	-	-	-	-	-
RQ-Trans. [ 28 ]	3.8B	RQ	2D	66M	256	3.20	-	-	-	-	3.80	323.7	-	-
MaskGIT [ 3 ]	227M	VQ	2D	66M	256	2.28	6.18	182.1	0.80	0.51	-	-	-	-
LlamaGen-3B [ 45 ]	3.1B	VQ	2D	72M	576	2.19	-	-	-	-	2.18	263.3	0.80	0.58

Appendix

Table 7: System-level comparison on ImageNet 256

\times

256 conditional generation. We report both Precision and Recall under classifier-free guidance (CFG) and non-CFG settings. “# Params (G)” denotes generator parameters; “Tok. Model” refers to the tokenizer model type; “Token Type” indicates 1D or 2D tokenization; “# Params (T)” denotes tokenizer parameters; and “# Tokens” represents the number of latent tokens.

							w/o CFG				w/ CFG
Method	# Params(G)	Tok. Model	Token Type	# Params(T)	#Tokens↓	Tok. rFID↓	gFID↓	IS↑	Prec↑	Recall↑	gFID↓	IS↑	Prec↑	Recall↑
GAN
BigGAN [ 3 ]	-	-	-	-	-	-	-	-	-	-	8.43	177.9	-	-
StyleGAN-XL [ 24 ]	168M	-	-	-	-	-	-	-	-	-	2.41	267.7	-	-
Auto-regressive
MaskGIT [ 3 ]	227M	VQ	2D	66M	1024	1.97	7.32	156.0	-	-	-	-	-	-

Appendix

Table 8: System-level comparison on ImageNet 512

\times

512 conditional generation. We report both Precision and Recall under classifier-free guidance (CFG) and non-CFG settings.

Method	Training Iter.	w/o CFG				w/ CFG
Method	Training Iter.	FID	IS	Prec.	Recall	FID	IS	Prec.	Recall
MacTok-64	400K	7.60	121.4	0.72	0.63	2.15	268.2	0.77	0.63
	1M	5.34	147.7	0.74	0.63	1.73	290.4	0.77	0.65
	2M	4.58	159.9	0.75	0.63	1.60	303.0	0.78	0.65
	3M	3.98	174.7	0.76	0.63	1.60	308.2	0.78	0.66
	4M	3.77	181.6	0.77	0.63	1.58	310.4	0.78	0.66

Appendix

Table 9: Generation performance over training of SiT-XL trained on MacTok with 64 and 128 tokens.

Figure 9: Reconstruction results of MacTok with 64 tokens.

Figure 10: Reconstruction results of MacTok with 128 tokens.

Figure 11: Reconstruction results of collapsed KL-VAE.

Figure 12: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“loggerhead turtle” (33).

Figure 13: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“macaw” (88).

Figure 14: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“Kakatoe galerita” (89).

Figure 15: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“golden retriever” (207).

Figure 16: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“Arctic wolf” (270).

Figure 17: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“Arctic fox” (279).

Figure 18: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“otter” (360).

Figure 19: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“panda” (388).

Figure 20: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“fire engine” (555).

Figure 21: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“space shuttle” (812).

Figure 22: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“ice cream” (928).

Figure 23: Uncurated 256 $\times$ 256 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“cheeseburger” (933).

Figure 24: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 128 tokens. We use CFG with 3.0. Class label =“white shark” (2).

Figure 25: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 128 tokens. We use CFG with 3.0. Class label =“Dungeness crab” (118).

Figure 26: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 128 tokens. We use CFG with 3.0. Class label =“Chesapeake Bay retriever” (209).

Figure 27: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 128 tokens. We use CFG with 3.0. Class label =“burrito” (965).

Figure 28: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 64 tokens. We use CFG with 3.0. Class label =“geyser” (974).

Figure 29: Uncurated 256 $\times$ 256 generation results of LightningDiT-XL with MacTok 64 tokens. We use CFG with 3.0. Class label =“valley” (979).

Figure 30: Uncurated 512 $\times$ 512 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“castle” (483).

Figure 31: Uncurated 512 $\times$ 512 generation results of SiT-XL with MacTok 128 tokens. We use CFG with 4.0. Class label =“cliff” (972).

Figure 32: Uncurated 512 $\times$ 512 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“coral reef” (973).

Figure 33: Uncurated 512 $\times$ 512 generation results of SiT-XL with MacTok 64 tokens. We use CFG with 4.0. Class label =“volcano” (980).

Explore similar work

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation
Mar 17, 2026Yunpeng Qu, Kaidong Zhang, Yukang Ding +2Image Tokenization Image Reconstruction
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
May 29, 2026Luyuan Zhang, Siyuan Li, Zedong Wang +7Variational Autoencoders Token Merging
1Tsinghua University · 2Westlake University · 3Zhejiang University +3
Vision Foundation Models as Generalist Tokenizers for Image Generation
May 18, 2026Anlin Zheng, Qi Han, Xin Wen +5Vision Foundation Models Image Tokenization
University of Hong Kong, Pokfulam, Hong Kong · StepFun

Mar 17, 2026J/K move · Enter open · S save

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

Yunpeng Qu, Kaidong Zhang, Yukang Ding, Ying Chen+1

Tsinghua University, Beijing, China · Alibaba Group, Beijing, China

Image Tokenization Image Reconstruction Autoregressive Image Generation

Open paper AskPDF arXiv

Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.

Abstract

Explore similar work

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging

Vision Foundation Models as Generalist Tokenizers for Image Generation