Modern visual tokenizers aim to preserve fine-grained visual details while learning high-level semantic representations for downstream generation and understanding. Query-based 1D visual tokenizers encode images into flexible latent sequences, yet how individual learnable queries gather and encode image information remains poorly understood. We reveal that these queries spontaneously develop complementary roles under semantic supervision: a small set of Content queries captures high-level semantic information, while a larger set of Detail queries preserves fine-grained visual details for reconstruction. To adapt token processing to these heterogeneous roles, we introduce MoETok, a 1D visual tokenizer that routes tokens through sparsely activated experts. Jointly optimized with reconstruction and semantic objectives, the router learns role-specialized processing paths adapted to each token's representation. Across model scales and token budgets, MoETok consistently improves reconstruction, semantic representation, and autoregressive generation over dense tokenizers under matched supervision and active computation. On ImageNet, it combines strong reconstruction and semantic representation with state-of-the-art class-conditional generation within the 1D tokenizer family. We further validate MoETok at scale in a unified autoregressive model jointly trained for understanding and generation with a next-token cross-entropy objective. The resulting model combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks including DPG-Bench (87.61), GenEval2 (40.04 GM), and WISE (0.49).
Figures & tables
Figure 1: Semantic supervision promotes query-role differentiation. Each query is plotted by attention variance Vq and per-query linear probe score. Blue and orange indicate Detail and Content queries; model-name colors distinguish tokenizers without and with semantic supervision.
Figure 2
Figure 3: MoETok adapts token processing through sparse expert routing. (a) A 1D tokenizer learns with reconstruction and feature regularization. (b) A standard ViT block. (c) A sparse MoE layer replaces the shared FFN with token-dependent expert selection and weighted aggregation.
Figure 4: Conditional routing learns role-specialized visual processing. Expert use broadens mid-encoder (a) ; role separation peaks mid-encoder and remains high at the output (b) . Experts show distinct role preferences (c) ; MoETok retains higher attention variance at the bottleneck (d) .
Method
Routed tokens
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Dense
–
1.566
41.83
61.52
12.18
MoETok
Patch
1.494
44.26
63.16
11.59
Dense
–
1.522
48.72
59.17
10.35
MoETok
Patch + Query
1.368
60.22
65.03
9.59
Table 2: 1D tokenization amplifies the semantic benefits of expert routing. Each pair matches active FFN capacity; gFID is evaluated without guidance.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Small
Base
Large
Encoder depth
8
12
24
Encoder heads
8
12
16
Encoder width
512
768
1024
Decoder depth
12
24
24
Decoder heads
12
16
16
Decoder width
768
1024
1024
Appendix
Table A.1: Tokenizer model scales. Model scaling jointly varies encoder and decoder capacity.
Configuration
Setting
Experts / Top- k
8 / 2
Latent dimension / codebook size
256 / 8K
Training epochs
100
Global batch size
512
Learning-rate schedule
Cosine decay
Base learning rate
2×10−4
Appendix
Table A.2: Tokenizer optimization settings for the main results.
Configuration
Pretraining
Language-model initialization
Qwen3.5-4B
Visual tokenizer
MoETok E8/T2; dimension 256; codebook 8K
Visual token count
256
Generation / understanding loss weights
1.5:1.0
Generation / understanding token ratio
60%:40%
Peak learning rate
10−4
Appendix
Table A.3: Unified-model pretraining. Understanding and generation share one autoregressive model and visual vocabulary.
Configuration
Supervised fine-tuning
Language-model initialization
Pretraining checkpoint at 100,000 iterations
Visual tokenizer
MoETok E8/T2; dimension 256; codebook 8K
Visual token count
256
Generation / understanding loss weights
1.0:1.5
Generation / understanding token ratio
40%:60%
Peak learning rate
5×10−5
Appendix
Table A.4: Unified-model supervised fine-tuning. SFT increases the relative weight and token share of understanding tasks.
Figure B.1: Query-level attention variance is consistent across datasets. Each point compares the same query on COCO and ImageNet; r and ρ denote Pearson and Spearman correlations.
Figure B.2: Content and Detail attention patterns across encoder depth. MoETok preserves clearer role differentiation near the bottleneck than Dense.
Figure B.3: Encoder attention aligns with reconstruction contributions. Detail queries exhibit highly consistent spatial patterns in encoder attention (left) and reconstruction changes under zero masking (right). Content queries with image-adaptive attention have negligible reconstruction effects.
Figure B.4: Expert utilization across depth in 1D tokenizers. Both expert-pool sizes exhibit concentrated early-layer routing and broader utilization in deeper layers. Loss-free balancing makes utilization more uniform across most layers.
Loss-free balancing
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Disabled
1.368
60.22
65.03
9.59
Enabled
1.491
55.39
63.29
12.65
Appendix
Table B.1: Loss-free balancing in a 1D tokenizer. For E4/T2, task-driven routing achieves better reconstruction, semantic representation, and generation than the balancing variant.
Encoder
Decoder
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Dense
Dense
1.522
48.72
59.17
10.35
MoE
Dense
1.368
60.22
65.03
9.59
Dense
MoE
1.813
49.41
60.39
9.95
MoE
MoE
1.458
62.75
66.61
11.23
Appendix
Table C.1: Encoder and decoder routing in 1D tokenizers. Routing in the encoder gives the best reconstruction and generation; extending routing to the decoder further improves semantic representation.
Experts / Top- k
Width
Total / Active
Linear probe ↑
Zero-shot ↑
gFID ↓
2 / 1
8D
16D/8D
55.08
61.69
10.80
4 / 2
4D
16D/8D
60.22
65.03
9.59
8 / 4
2D
16D/8D
57.21
64.02
9.85
Appendix
Table C.2: Routing granularity in 1D tokenizers. Total and active expert capacities are fixed at 16D and 8D . E4/T2 performs best across all three metrics.
Dim.
Codebook
Linear probe ↑
Zero-shot ↑
rFID ↓
gFID ↓
256
8K
60.63
66.37
1.53
8.99
256
16K
62.77
67.48
1.40
8.74
256
64K
62.01
67.28
1.39
10.32
64
8K
58.84
64.61
1.56
9.70
256
8K
60.63
66.37
1.53
8.99
1024
8K
59.38
66.69
1.49
9.42
Appendix
Table C.3: Latent dimension and codebook size. We vary codebook size at dimension 256 (top) and dimension with an 8K codebook (bottom). gFID uses CFG scale 1.
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China
Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content. Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details. We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement. A content-adaptive router ranks candidate locations by refinement priority and determines the number of 2D tokens for each image under a target average token budget. At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features. On ImageNet 256 x 256, MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.
Chenge Li, Yefei He, Xinyu Liu +3
Zhejiang University · The Hong Kong University of Science and Technology · University of Adelaide
Building a unified visual tokenizer is essential for bridging the gap between visual understanding and generation. Yet existing approaches struggle with the inherent conflict between these tasks, as a single token space is forced to support both high-level semantic abstraction and low-level pixel reconstruction. We propose WinTok, a concise hybrid tokenizer that achieves a win-win performance by explicitly decoupling the two objectives. WinTok supplements pixel tokens with a set of learnable semantic tokens, effectively mitigating cross-task interference without incurring the computational overhead of dual tokenizers. To further enhance understanding capability, we introduce an asymmetric token distillation mechanism: the semantic tokens are guided by pretrained semantic embeddings from any visual foundation model, enabling them to inherit strong discriminative power while maintaining flexibility. Across 10 challenging benchmarks, WinTok delivers consistent improvements in reconstruction, understanding, and generation. Trained on only 50M open-source data, WinTok surpasses the strong baseline UniTok by 11.2% in classification accuracy and achieves a competitive reconstruction rFID of 0.41, despite using substantially less training data. Code is released at https://github.com/markywg/WinTok.
Yiwei Guo, Shaobin Zhuang, Zhipeng Huang +4
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · WeChat Vision, Tencent Inc. · Shanghai Jiao Tong University