Modern visual tokenizers aim to preserve fine-grained visual details while learning high-level semantic representations for downstream generation and understanding. Query-based 1D visual tokenizers encode images into flexible latent sequences, yet how individual learnable queries gather and encode image information remains poorly understood. We reveal that these queries spontaneously develop complementary roles under semantic supervision: a small set of Content queries captures high-level semantic information, while a larger set of Detail queries preserves fine-grained visual details for reconstruction. To adapt token processing to these heterogeneous roles, we introduce MoETok, a 1D visual tokenizer that routes tokens through sparsely activated experts. Jointly optimized with reconstruction and semantic objectives, the router learns role-specialized processing paths adapted to each token's representation. Across model scales and token budgets, MoETok consistently improves reconstruction, semantic representation, and autoregressive generation over dense tokenizers under matched supervision and active computation. On ImageNet, it combines strong reconstruction and semantic representation with state-of-the-art class-conditional generation within the 1D tokenizer family. We further validate MoETok at scale in a unified autoregressive model jointly trained for understanding and generation with a next-token cross-entropy objective. The resulting model combines competitive visual understanding with state-of-the-art generation among unified discrete models, matching or surpassing diffusion-based models such as BAGEL and Qwen-Image on benchmarks including DPG-Bench (87.61), GenEval2 (40.04 GM), and WISE (0.49).
Figures & tables
Figure 1: Semantic supervision promotes query-role differentiation. Each query is plotted by attention variance Vq and per-query linear probe score. Blue and orange indicate Detail and Content queries; model-name colors distinguish tokenizers without and with semantic supervision.
Figure 2
Figure 3: MoETok adapts token processing through sparse expert routing. (a) A 1D tokenizer learns with reconstruction and feature regularization. (b) A standard ViT block. (c) A sparse MoE layer replaces the shared FFN with token-dependent expert selection and weighted aggregation.
Figure 4: Conditional routing learns role-specialized visual processing. Expert use broadens mid-encoder (a) ; role separation peaks mid-encoder and remains high at the output (b) . Experts show distinct role preferences (c) ; MoETok retains higher attention variance at the bottleneck (d) .
Method
Routed tokens
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Dense
–
1.566
41.83
61.52
12.18
MoETok
Patch
1.494
44.26
63.16
11.59
Dense
–
1.522
48.72
59.17
10.35
MoETok
Patch + Query
1.368
60.22
65.03
9.59
Table 2: 1D tokenization amplifies the semantic benefits of expert routing. Each pair matches active FFN capacity; gFID is evaluated without guidance.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Small
Base
Large
Encoder depth
8
12
24
Encoder heads
8
12
16
Encoder width
512
768
1024
Decoder depth
12
24
24
Decoder heads
12
16
16
Decoder width
768
1024
1024
Appendix
Table A.1: Tokenizer model scales. Model scaling jointly varies encoder and decoder capacity.
Configuration
Setting
Experts / Top- k
8 / 2
Latent dimension / codebook size
256 / 8K
Training epochs
100
Global batch size
512
Learning-rate schedule
Cosine decay
Base learning rate
2×10−4
Appendix
Table A.2: Tokenizer optimization settings for the main results.
Configuration
Pretraining
Language-model initialization
Qwen3.5-4B
Visual tokenizer
MoETok E8/T2; dimension 256; codebook 8K
Visual token count
256
Generation / understanding loss weights
1.5:1.0
Generation / understanding token ratio
60%:40%
Peak learning rate
10−4
Appendix
Table A.3: Unified-model pretraining. Understanding and generation share one autoregressive model and visual vocabulary.
Configuration
Supervised fine-tuning
Language-model initialization
Pretraining checkpoint at 100,000 iterations
Visual tokenizer
MoETok E8/T2; dimension 256; codebook 8K
Visual token count
256
Generation / understanding loss weights
1.0:1.5
Generation / understanding token ratio
40%:60%
Peak learning rate
5×10−5
Appendix
Table A.4: Unified-model supervised fine-tuning. SFT increases the relative weight and token share of understanding tasks.
Figure B.1: Query-level attention variance is consistent across datasets. Each point compares the same query on COCO and ImageNet; r and ρ denote Pearson and Spearman correlations.
Figure B.2: Content and Detail attention patterns across encoder depth. MoETok preserves clearer role differentiation near the bottleneck than Dense.
Figure B.3: Encoder attention aligns with reconstruction contributions. Detail queries exhibit highly consistent spatial patterns in encoder attention (left) and reconstruction changes under zero masking (right). Content queries with image-adaptive attention have negligible reconstruction effects.
Figure B.4: Expert utilization across depth in 1D tokenizers. Both expert-pool sizes exhibit concentrated early-layer routing and broader utilization in deeper layers. Loss-free balancing makes utilization more uniform across most layers.
Loss-free balancing
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Disabled
1.368
60.22
65.03
9.59
Enabled
1.491
55.39
63.29
12.65
Appendix
Table B.1: Loss-free balancing in a 1D tokenizer. For E4/T2, task-driven routing achieves better reconstruction, semantic representation, and generation than the balancing variant.
Encoder
Decoder
rFID ↓
Linear probe ↑
Zero-shot ↑
gFID ↓
Dense
Dense
1.522
48.72
59.17
10.35
MoE
Dense
1.368
60.22
65.03
9.59
Dense
MoE
1.813
49.41
60.39
9.95
MoE
MoE
1.458
62.75
66.61
11.23
Appendix
Table C.1: Encoder and decoder routing in 1D tokenizers. Routing in the encoder gives the best reconstruction and generation; extending routing to the decoder further improves semantic representation.
Experts / Top- k
Width
Total / Active
Linear probe ↑
Zero-shot ↑
gFID ↓
2 / 1
8D
16D/8D
55.08
61.69
10.80
4 / 2
4D
16D/8D
60.22
65.03
9.59
8 / 4
2D
16D/8D
57.21
64.02
9.85
Appendix
Table C.2: Routing granularity in 1D tokenizers. Total and active expert capacities are fixed at 16D and 8D . E4/T2 performs best across all three metrics.
Dim.
Codebook
Linear probe ↑
Zero-shot ↑
rFID ↓
gFID ↓
256
8K
60.63
66.37
1.53
8.99
256
16K
62.77
67.48
1.40
8.74
256
64K
62.01
67.28
1.39
10.32
64
8K
58.84
64.61
1.56
9.70
256
8K
60.63
66.37
1.53
8.99
1024
8K
59.38
66.69
1.49
9.42
Appendix
Table C.3: Latent dimension and codebook size. We vary codebook size at dimension 256 (top) and dimension with an 8K codebook (bottom). gFID uses CFG scale 1.