Organizations: Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Shanghai Key Laboratory of Multimodal Embodied AI
Autoregressive (AR) language models rely on causal tokenization, but extending this paradigm to vision remains non-trivial. Current visual tokenizers either flatten 2D patches into non-causal sequences or enforce heuristic orderings that misalign with the "next-token prediction" pattern. Recent diffusion autoencoders similarly fall short: conditioning the decoder on all tokens lacks causality, while applying nested dropout mechanism introduces imbalance. To address these challenges, we present CaTok, a 1D causal image tokenizer with a MeanFlow decoder. By selecting tokens over time intervals and binding them to the MeanFlow objective, as illustrated in Fig. 1, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling, while naturally capturing diverse visual concepts across token intervals. To further stabilize and accelerate training, we propose a straightforward regularization REPA-A, which aligns encoder features with Vision Foundation Models (VFMs). Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM with fewer training epochs, and the AR model attains performance comparable to leading approaches.
Figures & tables
Figure 2: Comparison among different decoders. a) Naïve flow decoders [ 47 ] condition on all 1D tokens from the encoder without dropout, leading the 1D tokens to lack causality; b) Consistency decoders obtain k by random sampling [ 2 , 63 ] or timestep binding [ 39 , 56 ] , and condition on the first k 1D tokens, which biases toward early tokens, introducing imbalance, leading to degraded performance of AR generation; c) Our MeanFlow decoder conditions on 1D tokens within the time interval [r,t] to model the average velocity field along the subpath, which inherently maintains causality and balance of the 1D visual tokens, and supporting one-step sampling during image reconstruction or generation.
Figure 3: Architecture of our CaTok . CaTok is a diffusion autoencoder with a causal Vision Transformer (ViT) [ 12 ] encoder and a MeanFlow Diffusion Transformer (DiT) [ 40 ] decoder. The encoder leverages registers [ 9 ] to extract rich visual information into 1D tokens, which are then conditioned to the decoder through time interval selecting. With two flow objectives and two representation alignment objectives, CaTok learns causal 1D representations that support both one-step and multi-step sampling, while naturally capturing diverse visual concepts across different token intervals.
Figure 4: We visualize the causal mask mechanism in ViT in a). After training CaTok , we freeze the encoder to extract 1D tokens. During AR training stage, these tokens are optimized with a class token prefix using teacher forcing under a diffusion loss [ 29 ] . At sampling time, we input a learned class token, the AR model predicts the corresponding visual 1D tokens, and these tokens are then conditioned to the decoder for generation.
Method
Token
#Dim.
#Param.
Epochs
VQ
rFID ↓
PSNR ↑
SSIM ↑
One-step 2D tokenizers
VQGAN [ 15 ]
16x16
16
307M
-
✓
7.94
-
-
LlamaGen [ 51 ]
16x16
8
72M
40
✓
2.19
20.67
0.589
MaskBit [ 62 ]
16x16
12
54M
270
✓
1.37
21.50
0.560
MAR-VAE [ 29 ]
16x16
16
-
-
×
1.22
-
-
OpenMagViT-V2 [ 34 ]
16x16
-
116M
270
✓
1.17
21.63
0.640
Table 1: Reconstruction results on ImageNet 256 × 256 benchmark. “Token” denotes the number of tokens used for reconstruction, and “Dim.” denotes the dimension of these tokens. “Param.” indicates the model size, and “VQ” specifies whether the tokens are vector-quantized. “ ↓ ” or “ ↑ ” denote lower or higher values are better. † : enabling one-step sampling. ⋆ : trained by official codebase [ 63 ] .
Method
Generator
#Param.
Token
Step
gFID ↓
IS ↑
2D autoregressive models
VQGAN [ 15 ]
Tam. Trans. [ 15 ]
1.4B
256
256
15.78
74.3
RQ-VAE [ 25 ]
RQ-Trans. [ 25 ]
3.8B
256
68
7.55
134.0
Causal MAR [ 29 ]
MAR-L [ 29 ]
481M
256
256
4.07
232.4
LlamaGen [ 51 ]
LlamaGen-L [ 51 ]
343M
256
256
3.80
248.3
VAR [ 52 ]
VAR-d16 [ 52 ]
310M
680
10
3.30
274.4
Table 2: Class-conditional generation results on ImageNet-1K 256 × 256 benchmark. “#Param.” denotes the parameters of generator, “Token” and “Step” indicates the number of tokens (learned tokens in tokenization training / used tokens in AR modeling) and steps used for generation, respectively. “ ↓ ” or “ ↑ ” denote lower or higher values are better. ⋆ : trained by the official codebase [ 63 ] .
Table 3: Ablation on technique designs.
Figure 5: Qualitative Results. 256 × 256 generated images on ImageNet-1K with CaTok -L-32.
Figure 6: Effectiveness of our REPA-A. a) We apply principal component analysis (PCA) to visualize image features from the CaTok encoder. b) Training curves of the smoothed MSE between prediction and target, with the MeanFlow loss ( LMF ) added at 25K steps.
Training Config
CaTok -B
CaTok -L
AR modeling
Optimizer
AdamW
Peak learning rate
1×10−4
5×10−5
Minimum learning rate
0
Learning rate schedule
cosine decay
constant
Batch size
1024
2048
Weight decay
0.05
Table 4: Detailed configuration of CaTok -B and CaTok -L for tokenization and AR modeling.
Method
#Param.
Token
Step
rFID
gFID
LlamaGen
343M
256
256
2.19
3.80
CaTok -LlamaGen
343M
256
256
3.81
3.35
Table 5: Reconstruction and generation results of CaTok -VQ