Organizations: Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Shanghai Key Laboratory of Multimodal Embodied AI
Autoregressive (AR) language models rely on causal tokenization, but extending this paradigm to vision remains non-trivial. Current visual tokenizers either flatten 2D patches into non-causal sequences or enforce heuristic orderings that misalign with the "next-token prediction" pattern. Recent diffusion autoencoders similarly fall short: conditioning the decoder on all tokens lacks causality, while applying nested dropout mechanism introduces imbalance. To address these challenges, we present CaTok, a 1D causal image tokenizer with a MeanFlow decoder. By selecting tokens over time intervals and binding them to the MeanFlow objective, as illustrated in Fig. 1, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling, while naturally capturing diverse visual concepts across token intervals. To further stabilize and accelerate training, we propose a straightforward regularization REPA-A, which aligns encoder features with Vision Foundation Models (VFMs). Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM with fewer training epochs, and the AR model attains performance comparable to leading approaches.
Figures & tables
Figure 2: Comparison among different decoders. a) Naïve flow decoders [ 47 ] condition on all 1D tokens from the encoder without dropout, leading the 1D tokens to lack causality; b) Consistency decoders obtain k by random sampling [ 2 , 63 ] or timestep binding [ 39 , 56 ] , and condition on the first k 1D tokens, which biases toward early tokens, introducing imbalance, leading to degraded performance of AR generation; c) Our MeanFlow decoder conditions on 1D tokens within the time interval [r,t] to model the average velocity field along the subpath, which inherently maintains causality and balance of the 1D visual tokens, and supporting one-step sampling during image reconstruction or generation.
Figure 3: Architecture of our CaTok . CaTok is a diffusion autoencoder with a causal Vision Transformer (ViT) [ 12 ] encoder and a MeanFlow Diffusion Transformer (DiT) [ 40 ] decoder. The encoder leverages registers [ 9 ] to extract rich visual information into 1D tokens, which are then conditioned to the decoder through time interval selecting. With two flow objectives and two representation alignment objectives, CaTok learns causal 1D representations that support both one-step and multi-step sampling, while naturally capturing diverse visual concepts across different token intervals.
Figure 4: We visualize the causal mask mechanism in ViT in a). After training CaTok , we freeze the encoder to extract 1D tokens. During AR training stage, these tokens are optimized with a class token prefix using teacher forcing under a diffusion loss [ 29 ] . At sampling time, we input a learned class token, the AR model predicts the corresponding visual 1D tokens, and these tokens are then conditioned to the decoder for generation.
Method
Token
#Dim.
#Param.
Epochs
VQ
rFID ↓
PSNR ↑
SSIM ↑
One-step 2D tokenizers
VQGAN [ 15 ]
16x16
16
307M
-
✓
7.94
-
-
LlamaGen [ 51 ]
16x16
8
72M
40
✓
2.19
20.67
0.589
MaskBit [ 62 ]
16x16
12
54M
270
✓
1.37
21.50
0.560
MAR-VAE [ 29 ]
16x16
16
-
-
×
1.22
-
-
OpenMagViT-V2 [ 34 ]
16x16
-
116M
270
✓
1.17
21.63
0.640
Table 1: Reconstruction results on ImageNet 256 × 256 benchmark. “Token” denotes the number of tokens used for reconstruction, and “Dim.” denotes the dimension of these tokens. “Param.” indicates the model size, and “VQ” specifies whether the tokens are vector-quantized. “ ↓ ” or “ ↑ ” denote lower or higher values are better. † : enabling one-step sampling. ⋆ : trained by official codebase [ 63 ] .
Method
Generator
#Param.
Token
Step
gFID ↓
IS ↑
2D autoregressive models
VQGAN [ 15 ]
Tam. Trans. [ 15 ]
1.4B
256
256
15.78
74.3
RQ-VAE [ 25 ]
RQ-Trans. [ 25 ]
3.8B
256
68
7.55
134.0
Causal MAR [ 29 ]
MAR-L [ 29 ]
481M
256
256
4.07
232.4
LlamaGen [ 51 ]
LlamaGen-L [ 51 ]
343M
256
256
3.80
248.3
VAR [ 52 ]
VAR-d16 [ 52 ]
310M
680
10
3.30
274.4
Table 2: Class-conditional generation results on ImageNet-1K 256 × 256 benchmark. “#Param.” denotes the parameters of generator, “Token” and “Step” indicates the number of tokens (learned tokens in tokenization training / used tokens in AR modeling) and steps used for generation, respectively. “ ↓ ” or “ ↑ ” denote lower or higher values are better. ⋆ : trained by the official codebase [ 63 ] .
Table 3: Ablation on technique designs.
Figure 5: Qualitative Results. 256 × 256 generated images on ImageNet-1K with CaTok -L-32.
Figure 6: Effectiveness of our REPA-A. a) We apply principal component analysis (PCA) to visualize image features from the CaTok encoder. b) Training curves of the smoothed MSE between prediction and target, with the MeanFlow loss ( LMF ) added at 25K steps.
Training Config
CaTok -B
CaTok -L
AR modeling
Optimizer
AdamW
Peak learning rate
1×10−4
5×10−5
Minimum learning rate
0
Learning rate schedule
cosine decay
constant
Batch size
1024
2048
Weight decay
0.05
Table 4: Detailed configuration of CaTok -B and CaTok -L for tokenization and AR modeling.
Method
#Param.
Token
Step
rFID
gFID
LlamaGen
343M
256
256
2.19
3.80
CaTok -LlamaGen
343M
256
256
3.81
3.35
Table 5: Reconstruction and generation results of CaTok -VQ
Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.
Wenda Chu, Bingliang Zhang, Jiaqi Han +4
ByteDance Seed · California Institute of Technology · Stanford University
We introduce QuadTok, a novel framework for visual tokenization and autoregressive image generation. Compared to traditional approaches using 2D grids or 1D token sequences, we propose a hierarchical quadtree structure, bridging the gap between 2D spatial binding and 1D sequence-level flexibility. The QuadTok tokenizer dynamically allocates representational capacity to visually intricate areas while leaving homogeneous regions at a coarse resolution. Compared with a fixed 256-token grid, our ImageNet-trained tokenizer saves approximately 10% of tokens on ImageNet and 9% when transferred zero-shot to the COCO dataset, while maintaining comparable reconstruction fidelity. Furthermore, the natural causality introduced by the tree structure seamlessly enables autoregressive image generation. Conditioned on a quadtree topology supplied before generation, our 947M GPT-style generative model achieves a 2.08 gFID on the ImageNet 256×256 benchmark. Additionally, leveraging the strong spatial correlation preserved by the quadtree structure, the QuadTok generator enables zero-shot spatially controlled image generation capabilities. Code: https://github.com/myc634/QuadTok.
Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose SemTok, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.
Yunpeng Qu, Kaidong Zhang, Yukang Ding +2
Tsinghua University, Beijing, China · Alibaba Group, Beijing, China