Semantic-Aware Prefix Learning for Token-Efficient Image Generation

Qingfeng Li; Haoxian Zhang; Xu He; Songlin Tang; Zhixue Fang; Xiaoqiang Liu; Pengfei Wan Guoqi Li

cs.CVMar 26, 2026

Semantic-Aware Prefix Learning for Token-Efficient Image Generation

Authors: Qingfeng Li, Haoxian Zhang, Xu He, Songlin Tang, Zhixue Fang, Xiaoqiang Liu, Pengfei Wan Guoqi Li

Organizations: Institute of Automation, Chinese Academy of Sciences · Kling Team, Kuaishou Technology · Tsinghua University

Abstract

Visual tokenizers play a central role in latent image generation by bridging high-dimensional images and tractable generative modeling. However, most existing tokenizers are still trained with reconstruction-dominated objectives, which often yield latent representations that are only weakly grounded in high-level semantics. Recent approaches improve semantic alignment, but typically treat semantic signals as auxiliary regularization rather than making them functionally necessary for representation learning. We propose SMAP, a SeMantic-Aware Prefix tokenizer that injects class-level semantic conditions into a query-based 1D tokenization framework. To make semantics indispensable during training, SMAP introduces a tail token dropping strategy, which forces semantic conditions and early latent prefixes to bear increasing responsibility under progressively reduced token budgets. To verify that the resulting latent space is useful for generation rather than reconstruction alone, we further introduce CARD, a hybrid Causal AutoRegressive--Diffusion generator. Extensive experiments on ImageNet show that SMAP consistently improves reconstruction quality across discrete and continuous tokenization settings, and that its semantically grounded latent space yields strong downstream generation performance under compact token budgets.

Figures & tables

Figure 1: Semantic-aware prefix learning in reconstruction and generation. Top: Using only the class condition, SMAP reconstructs images that already capture category-level semantics and coarse global structure. Middle: Adding latent tokens substantially improves reconstruction fidelity and restores instance-specific details, showing that semantic conditions and latent prefixes play complementary roles. Bottom: Based on the resulting semantically grounded token space, CARD generates high-quality class-conditional images.

Figure 2: Overview of our method. (a) proposes a novel mechanism for semantic injection. It extracts conditional embeddings from class labels and inserts them between visual patch tokens and learnable latent tokens. The condition embeddings act as intermediaries that interact jointly with image patches to guide the formation of latent tokens. It further strengthens semantic dependency through a tail token dropping strategy. (b) proposes a hybrid Causal AutoRegressive–Diffusion framework that fully leverages SMAP ’s capabilities. (c) shows the SMAP tokenization process for CARD generation.

Figure 3: ImageNet-1K reconstruction scaling and comparison. (a) Reconstruction FID ( $\mathrm{rFID}$ ) of SMAP under different token budgets and model scales. Across VQ, KL, and SoftVQ variants, increasing the number of latent tokens consistently improves reconstruction quality, and larger SMAP models achieve stronger performance under the same token budget. (b) Reconstruction comparison with prior 1D tokenizers. At matched token lengths, SMAP consistently outperforms TiTok and TA-TiTok , with the largest gains observed in continuous latent settings. Overall, the results show that SMAP scales favorably with both token budget and model capacity, while providing substantially better reconstruction quality than existing baselines.

Figure 4: Semantic identity is controlled by $C$ , while instance-level details are carried by $Z$ . We visualize reconstructions obtained by independently manipulating the semantic condition $C$ and latent tokens $Z$ . Using only $C$ with $Z=\emptyset$ yields coarse reconstructions that preserve category-level semantics. In contrast, cross-combining $C$ from one image with $Z$ from another transfers semantic identity and instance-specific appearance in a complementary manner.

Model	Depth	Width	Heads	$D_{\text{mlp}}$	$W_{\text{mlp}}$	#Params
CARD-B	24	768	12	6	1024	234M
CARD-L	32	1024	16	8	1280	568M
CARD-XL	40	1280	16	12	1536	1.1B

Table 1: Architecture Configuration of CARD. Following MAR , we scale up blocks across three configurations.

arch	tokens		TiTok		SMAP
arch	#	c	$\mathrm{rFID}\downarrow$	IS $\uparrow$	$\mathrm{rFID}\downarrow$	IS $\uparrow$
VQ	32	–	7.72	98.3	3.24 (-4.48)	230.4 (+132.1)
	64	–	4.25	138.0	2.29 (-1.96)	260.1 (+122.1)
	128	–	2.63	168.1	1.47 (-1.16)	280.5 (+112.4)
KL	32	16	2.56	171.7	1.07 (-1.49)	280.4 (+108.7)
	64	16	1.64	198.0	0.96 (-0.68)	299.5 (+101.5)

Table 2: Ablation on the improved one-stage training recipe. All models are trained and evaluated on ImageNet256 . We compare the baseline tokenizer ( TiTok ) against SMAP under matched token budgets. Numbers in parentheses indicate the change relative to the baseline. Lower

\mathrm{rFID}

and higher IS are better.

arch	tokens		SMAP w/o trunc.		SMAP w/ trunc.
arch	#	c	rFID $\downarrow$	IS $\uparrow$	rFID $\downarrow$	IS $\uparrow$
VQ	32	–	3.41	220.8	3.24 (-0.17)	230.4 (+9.6)
	64	–	2.50	253.4	2.29 (-0.21)	260.1 (+6.7)
	128	–	1.65	269.8	1.47 (-0.18)	280.5 (+10.7)
KL	32	16	1.20	269.1	1.07 (-0.13)	280.4 (+11.3)
	64	16	1.03	278.6	0.96 (-0.07)	299.5 (+20.9)

Table 3: Ablation on progressive token truncation. All models are trained and evaluated on ImageNet256 . We compare SMAP trained without and with progressive token truncation. Numbers in parentheses indicate the change relative to the corresponding model trained without truncation. Lower

\mathrm{rFID}

and higher IS are better.

Model (G)	# Params (G)	Model (T)	# Params (T)	# Tokens $\downarrow$	rFID $\downarrow$	w/o CFG		w/ CFG
Model (G)	# Params (G)	Model (T)	# Params (T)	# Tokens $\downarrow$	rFID $\downarrow$	gFID $\downarrow$	IS $\uparrow$	gFID $\downarrow$	IS $\uparrow$
Auto-regressive
VQGAN ( Esser et al., 2021 )	1.4B	VQ	23M	256	7.94	–	–	5.20	290.3
ViT-VQGAN ( Yu et al., 2021 )	1.7B	VQ	64M	1024	1.28	4.17	175.1	–	–
LlamaGen-3B ( Sun et al., 2024 )	3.1B	VQ	72M	576	2.19	–	–	2.18	263.3
TiTok-S-128 ( Yu et al., 2024b )	287M	VQ	72M	128	1.61	–	–	1.97	281.8

Table 4: System-level comparison on ImageNet 256

\times

256 conditional generation. SMAP + CARD achieves competitive performance under a compact 128-token budget across both KL and SoftVQ variants. “Model (G)” denotes the generator, “# Params (G)” its parameter count, “Model (T)” the tokenizer, “# Params (T)” its parameter count, and “# Tokens” the number of latent tokens used during generation. † indicates that the model was trained on data beyond ImageNet.

Figure 5: Effect of semantic-aware tokenization on downstream generation. We compare three settings: a reconstruction-only tokenizer with independent generator conditioning, a semantic-aware tokenizer with independent generator conditioning, and a shared-semantic setting in which the generator reuses the tokenizer’s learned semantic embedding space. Semantic-aware tokenizer pretraining consistently improves $\mathrm{gFID}$ across all token budgets, and semantic sharing yields a further gain in every setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Tokenizer	Encoder output	Regularization target	$\mathtt{Regu}(\cdot)$ instantiation	Output to decoder
VQ	$\mathbf{Z}_{\mathrm{1D}}\in\mathbb{R}^{K\times D}$	discrete codebook	nearest-neighbor vector quantization	quantized code embeddings
KL	$(\bm{\mu},\bm{\sigma})$	Gaussian posterior	reparameterized latent sampling	continuous sampled latents
SoftVQ	$\mathbf{Z}_{\mathrm{1D}}\in\mathbb{R}^{K\times D}$	soft code assignment	temperature-controlled soft quantization	weighted codebook mixtures

Appendix

Table 5: Instantiation of

\mathtt{Regu}(\cdot)

under different tokenizer formulations.

Table 6: Detailed results of preliminary experiments in the main paper.

tokenizer	rFID $\downarrow$	generator	w/o guidance		w/ guidance		P $\downarrow$	S $\downarrow$	T $\uparrow$
			gFID $\downarrow$	IS $\uparrow$	gFID $\downarrow$	IS $\uparrow$
diffusion-based generative models
VAE ( , ) ‡	0.19	UViT-L/4 ( Bao et al., 2023 )	18.03	76.9	4.67	213.3	287M	50	1.0
		UViT-H/4 ( Bao et al., 2023 )	15.71	101.3	4.05	263.8	501M	50	0.6
		DiT-XL/2 ( Peebles and Xie, 2023 )	12.03	105.3	3.04	240.8	675M	250	0.1
transformer-based generative models

Appendix

Table 7: ImageNet-1K

512\times 512

generation results evaluated with ADM ( Dhariwal and Nichol, 2021b ) . †: Trained on OpenImages ( Kuznetsova et al., 2020 ) ‡: Trained on OpenImages, LAION-Aesthetics/-Humans ( Schuhmann et al., 2022 ) . P: generator’s parameters. S: sampling steps. T: throughput as samples per seconds on A100 with float32 precision, measured with w/ guidance variants if available. “guidance” refers to classifier-free guidance.

Visual Tokenization

Visual tokenization turns images and video into compact, structured tokens for efficient perception, compression, and generation.

See all 66 papers

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation
Mar 17, 2026Yunpeng Qu, Kaidong Zhang, Yukang Ding +2Image Tokenization Image Reconstruction
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
MacTok: Robust Continuous Tokenization for Image Generation
Mar 31, 2026Hengyu Zeng, Xin Gao, Guanghao Li +5Image Tokenization Visual Token Compression
Fudan University
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
May 29, 2026Luyuan Zhang, Siyuan Li, Zedong Wang +7Variational Autoencoders Token Merging
1Tsinghua University · 2Westlake University · 3Zhejiang University +3

Mar 17, 2026J/K move · Enter open · S save

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

Yunpeng Qu, Kaidong Zhang, Yukang Ding, Ying Chen+1

Tsinghua University, Beijing, China · Alibaba Group, Beijing, China

Image Tokenization Image Reconstruction Autoregressive Image Generation

Open paper AskPDF arXiv

Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.

Abstract

Visual Tokenization

Explore similar work

Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

MacTok: Robust Continuous Tokenization for Image Generation

MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging