Autoregressive (AR) image generation has shown strong potential for scalable high-fidelity synthesis by modeling images as discrete token sequences. However, traditional next token prediction (NTP) continues to suffer from sparse and myopic supervision, insufficiently discriminative representations, and high training cost caused by dense computation over the full token sequence. To address these issues, we propose multi-token autoregressive (MTAR), a unified training framework that improves autoregressive image generation from three aspects: prediction objectives, representation regularization, and training efficiency. Specifically, MTAR introduces multi-token prediction (MTP) to alleviate the sparsity and myopia of traditional NTP by imposing joint supervision on multiple future tokens; employs token-level contrastive regularization (TCR) to explicitly enhance the separability of sampled token representations and thereby improve representation discriminability; and incorporates semantic dropping (SD) as a semantics-aware training acceleration strategy to reduce redundant computation on low-information tokens while preserving informative learning signals. All three components are applied only during training and introduce no additional overhead during autoregressive inference. On ImageNet, MTAR achieves a better balance between generation quality and training efficiency. Compared with LlamaGen, MTAR achieves up to 0.95 lower FID and 39% faster training. Moreover, even with only 1/3 of the training iterations, it still attains performance comparable to or better than the baseline, substantially reducing training time.
Figures & tables
Figure 1. (a) Traditional NTP provides only short-range supervision over the full token sequence, leading to sparse and myopic training signals, while requiring autoregressive modeling over the entire sequence with high computational cost. (b) Our framework improves training with MTP and SD. MTP introduces more forward-looking and denser auxiliary supervision signals, thus enhancing contextual dependency modeling and alleviating the myopic nature of conventional single-step prediction, as indicated by the orange-red dashed arrows. SD further improves training efficiency by discarding less informative patches; the white patches denote the dropped regions. A two-panel schematic comparing conventional next-token prediction with the proposed training framework. Panel (a) shows all image tokens retained and connected primarily through short-range, single-step prediction signals. Panel (b) shows retained tokens receiving additional orange-red dashed connections to more distant future targets, representing multi-token prediction. Several image patches are colored white to indicate tokens removed by semantic dropping, resulting in a shorter training sequence.
Figure 2. Comparison of total training time (hours) and total GPU memory usage across all training GPUs between LlamaGen ( Sun et al., 2024 ) and MTAR for the Base and Large models, along with the corresponding FID values. LlamaGen is trained for 300 epochs, whereas MTAR is trained for only 100 epochs. For the efficiency measurements, the Base models are trained on a single NVIDIA A800 GPU, while the Large models are trained on two NVIDIA A800 GPUs. MTAR consistently achieves better generation quality and efficiency than the corresponding LlamaGen baselines: for the Base model, FID improves from 5.46 to 5.37, while training time and memory are reduced from 115 to 30 and from 45 GiB to 31 GiB; for the Large model, FID improves from 3.80 to 3.36, while training time and memory are reduced from 175 to 42 and from 97 GiB to 73 GiB. Both methods use torch.compile . A grouped comparison of LlamaGen and MTAR across the Base and Large model settings, showing training time, GPU memory usage, and FID. MTAR has lower values than LlamaGen for all three metrics in both settings.
Figure 3. Overview of the proposed framework. (a) Overall framework under the MTP (R 1 , B) setting: besides the original NTP (R 1 ) head, an additional MTP head is introduced to provide auxiliary supervision for the token directly below the current token. (b) TCR: Sampled token features are passed through two Dropout-perturbed branches and a shared MLP to construct a cross-view similarity matrix. The diagonal entries bring matched token pairs closer, while the off-diagonal entries push different sampled tokens apart. A two-panel overview of MTAR. Panel (a) shows the NTP head predicting the next token and an MTP head predicting the token below. Panel (b) shows TCR constructing a similarity matrix from two Dropout-perturbed views of sampled token features.
Figure 4. Semantic Drop. (a) DINOv3 ( Siméoni et al., 2025 ) extracts a token importance map for each image (darker: higher semantic importance). Low-importance tokens are dropped, retaining only semantically salient tokens for training. (b) Retained tokens preserve their original spatial indices, and both NTP and MTP losses are computed accordingly. DINOv3 identifies important image tokens, which retain their original positions for NTP and MTP training.
Type
Model
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
GAN
BigGAN ( Brock, 2018 )
112M
6.95
224.5
0.89
0.38
GigaGAN ( Kang et al., 2023 )
569M
3.45
225.5
0.84
0.61
Diffusion
ADM ( Dhariwal and Nichol, 2021 )
554M
10.94
101.0
0.69
0.63
CDM ( Ho et al., 2022 )
–
4.88
158.7
–
–
LDM-4-G ( Rombach et al., 2022 )
400M
3.60
247.7
0.87
0.48
DiT-L/2 ( Peebles and Xie, 2023 )
458M
5.02
167.2
0.75
0.57
Table 1. Model comparisons on the class-conditional ImageNet 256×256 benchmark ( Deng et al., 2009 ) . Metrics include Fréchet inception distance (FID), inception score (IS), precision, and recall. “ ↓ ” and “ ↑ ” indicate that lower and higher values are better, respectively. † denotes results reported by NAR ( He et al., 2025 ) . “-re” denotes the use of rejection sampling.
Model
Epochs
FID ↓
IS ↑
Precision ↑
Recall ↑
Speedup
LlamaGen-B ( Sun et al., 2024 )
300
5.46
193.61
0.84
0.46
1.00 ×
MTAR-B
50
5.76
198.70
0.86
0.42
7.62 ×
100
5.37
209.07
0.86
0.44
3.81 ×
250
4.66
211.54
0.84
0.47
1.73 ×
300
4.50
213.96
0.84
0.48
1.27 ×
LlamaGen-L ( Sun et al., 2024 )
300
3.80
248.28
0.83
0.52
1.00 ×
Table 2. Comparison between LlamaGen ( Sun et al., 2024 ) and MTAR in terms of generation quality and training efficiency. Speedup is measured with respect to the total training time of the baseline LlamaGen ( Sun et al., 2024 ) .
Method
Right (R 2 )
Bottom (B)
Bottom-Right (B.R 1 )
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
MTP (R 1 , R 2 )
✓
138M
20.13
59.32
0.75
0.45
MTP (R 1 , B)
✓
138M
19.75
62.12
0.75
0.45
MTP (R 1 , B.R 1 )
✓
138M
19.96
61.05
0.75
0.44
Table 3. Ablation of MTP head directions based on the NTP head (R 1 ). We compare auxiliary prediction along three directions: right, bottom, and bottom-right.
Method
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
Speedup
LlamaGen-B
111M
21.96
54.04
0.71
0.46
1.00 ×
MTP
138M
19.75
62.12
0.75
0.45
0.79 ×
TCR
111M
20.55
57.78
0.72
0.47
0.98 ×
SD
111M
22.02
54.99
0.70
0.47
1.62 ×
MTP+TCR
138M
18.78
64.33
0.73
0.48
0.79 ×
MTP+SD
138M
20.79
66.17
0.71
0.47
1.27 ×
Table 4. Ablation study of MTP, TCR, and SD on LlamaGen-B ( Sun et al., 2024 ) .
Method
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
MTP (R 1 , R 2 , R 3 )
164M
19.79
41.77
0.74
0.47
MTP (R 1 , R 2 , B)
164M
20.34
59.85
0.72
0.46
MTP (R 1 , B, B.R 1 )
164M
19.94
62.33
0.74
0.46
MTP (R 1 , R 2 , R 3 , R 4 )
191M
20.95
57.15
0.73
0.45
MTP (R 1 , R 2 , B, B.R 1 )
191M
20.95
57.18
0.73
0.45
MTP (R 1 , R 2 , R 3 , R 4 , R 5 )
218M
20.18
59.46
0.74
0.45
Table 5. Ablation on the number of MTP heads. Performance does not consistently improve as more auxiliary heads are added.
Sampled Tokens
FID ↓
IS ↑
Precision ↑
Recall ↑
16384
20.70
58.60
0.73
0.46
8192
19.83
58.62
0.73
0.47
4096
19.71
57.82
0.72
0.48
2048
18.78
64.33
0.73
0.48
1024
19.18
63.54
0.73
0.48
512
18.94
64.45
0.74
0.48
Table 6. Ablation study on the number of sampled tokens for TCR. We observe that using 2,048 sampled tokens yields the best overall performance.
Drop Strategy
FID ↓
IS ↑
Precision ↑
Recall ↑
Random
19.97
62.69
0.72
0.45
SigLIP2 ( Tschannen et al., 2025 )
19.70
63.84
0.73
0.46
DINOv3 ( Siméoni et al., 2025 )
18.65
66.32
0.75
0.46
Table 7. Ablation on drop strategies. DINOv3 ( Siméoni et al., 2025 ) -guided SD achieves the best performance.
Drop Rate
FID ↓
IS ↑
Precision ↑
Recall ↑
Speedup
60%
19.31
62.93
0.74
0.45
1.81 ×
50%
18.65
66.32
0.75
0.46
1.60 ×
40%
18.91
65.32
0.74
0.46
1.38 ×
Table 8. Ablation on the patch drop rate in SD. A 50% drop rate achieves the best overall trade-off. Speedup is measured relative to the MTP+TCR configuration without SD.
SD Schedule
FID ↓
IS ↑
Precision ↑
Recall ↑
Speedup
40%:60%
18.65
65.54
0.75
0.46
1.23 ×
50%:50%
18.60
65.01
0.74
0.48
1.31 ×
60%:40%
18.41
64.85
0.74
0.48
1.39 ×
70%:30%
18.68
64.38
0.74
0.48
1.49 ×
80%:20%
18.65
66.32
0.75
0.46
1.60 ×
90%:10%
19.40
62.89
0.75
0.46
1.73 ×
Table 9. Ablation on the two-stage SD training schedule. The 80%:20% schedule provides a favorable quality–efficiency trade-off. Speedup is measured relative to the MTP+TCR configuration without SD.
Figure 5. We retain the top-50% patches according to the importance scores produced by SigLIP2 ( Tschannen et al., 2025 ) and DINOv3 ( Siméoni et al., 2025 ) , and visualize the corresponding VQ-decoded reconstructions. Compared with SigLIP2 ( Tschannen et al., 2025 ) , DINOv3 ( Siméoni et al., 2025 ) better preserves the semantic content and structural integrity of the foreground object. VQ-decoded reconstructions using the top-50% patches selected by SigLIP2 and DINOv3, with DINOv3 preserving foreground objects more clearly.
Figure 6. Qualitative comparison between LlamaGen-L ( Sun et al., 2024 ) and MTAR-L on ImageNet 256×256 ( Deng et al., 2009 ) . Under the same class conditions, MTAR-L produces images with clearer semantics, more coherent structures, and fewer artifacts. Red boxes highlight representative artifacts in LlamaGen-L ( Sun et al., 2024 ) samples. ImageNet samples generated by LlamaGen-L and MTAR-L, with red boxes marking artifacts in the LlamaGen-L results.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
config
value
training hyper-parameters
optimizer
AdamW
learning rate
1e-4
weight decay
5e-2
optimizer momentum
(0.9,0.95)
batch size
256
Appendix
Table 10. Detailed training and sampling hyper-parameters of MTAR.
SD Schedule
FID ↓
IS ↑
Precision ↑
Recall ↑
Speedup ↑
40%:5%:55%
18.62
66.47
0.74
0.47
1.25 ×
40%:10%:50%
18.77
63.01
0.75
0.46
1.27 ×
50%:5%:45%
18.56
67.11
0.75
0.46
1.33 ×
50%:10%:40%
18.99
65.04
0.75
0.46
1.35 ×
60%:5%:35%
18.20
68.25
0.75
0.47
1.42 ×
60%:10%:30%
18.56
65.77
0.74
0.46
1.44 ×
Appendix
Table 11. Ablation study on three-stage SD schedules. The three entries denote the relative durations of stages with patch-drop rates of 50%, 25%, and 0%, respectively. Different schedules lead to different quality–efficiency trade-offs, and 60%:5%:35% achieves the best FID among the evaluated three-stage variants.
Method
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
RAR-B ( Yu et al., 2025 )
285M
18.70
72.28
0.71
0.50
RAR-B+MTP(B)
320M
17.82
76.49
0.73
0.51
Appendix
Table 12. Generalization of MTP(B) to RAR-B ( Yu et al., 2025 ) . Adding MTP(B) consistently improves FID, IS, precision, and recall, demonstrating that multi-token auxiliary supervision transfers beyond the LlamaGen backbone.
Figure 7. Optimization behavior under different MTP-head configurations. (a) Primary autoregressive loss L(R1) across training epochs for single-head and multi-head prediction sets. (b) Primary-head loss L(R1) and auxiliary-head loss L(B) for the (R1,B) configuration. The comparison illustrates how additional prediction targets alter the optimization of the main autoregressive objective and motivates limiting the number of auxiliary heads to avoid excessive interference.
Figure 8. Token-level linear-probing accuracy on the ImageNet validation set. MTAR-L consistently outperforms LlamaGen-L across all Top- N settings, indicating better linear separability of visual-token representations.
Similarity
Ratio
Sim > 0.9
2.01%
Sim > 0.8
8.73%
Appendix
Table 13. Proportions of highly similar token pairs under different cosine similarity thresholds.
Figure 9. Randomly selected examples before and after SD for the ImageNet classes picket fence and spider web , using 50% token retention.
Figure 10. Generated samples from MTAR-L for ImageNet class ID 269 (wolf).
Figure 11. Generated samples from MTAR-L for ImageNet class ID 323 (butterfly).
Figure 12. Generated samples from MTAR-L for ImageNet class ID 90 (lorikeet).
Figure 13. Generated samples from MTAR-L for ImageNet class ID 979 (valley).
Figure 14. Uncurated random samples from LlamaGen and MTAR.
Figure 15. Comparison of class-conditional samples produced by LlamaGen-L and MTAR-L on ImageNet 256×256 .
Figure 16. Additional comparison of class-conditional samples produced by LlamaGen-L and MTAR-L on ImageNet 256×256 .
Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.
Wenda Chu, Bingliang Zhang, Jiaqi Han +4
ByteDance Seed · California Institute of Technology · Stanford University
Test-time strategies have shown remarkable success in improving large language models, but their application to next-token prediction (NTP) autoregressive (AR) image generation remains largely underexplored. Existing test-time scaling (TTS) methods for visual autoregressive models (VAR) rely on frequent partial decoding and external reward models, which are inefficient and often ineffective for NTP-based image generation due to the inherent instability of intermediate decoding results. To address these limitations, we propose ScalingAR, a novel test-time scaling framework tailored for NTP-based AR image generation. ScalingAR introduces token entropy as a confidence signal and operates at two complementary levels: (i) Profile Level, integrates intrinsic uncertainty and conditional utilization into a unified confidence state, and (ii) Policy Level, leverages this state for adaptive trajectory pruning and dynamic guidance scheduling. Without requiring early decoding or auxiliary rewards, ScalingAR achieves significant improvements across diverse benchmarks. Experiments show that ScalingAR (I) improves base models by 12.5% on GenEval and 15.2% on TIIF-Bench, (II) reduces visual token consumption by 62.0% while outperforming baselines, and (III) enhances robustness, mitigating performance degradation by 26.0% in challenging scenarios. These results establish ScalingAR as a robust and efficient test-time scaling solution for autoregressive image generation.
Autoregressive (AR) image models have recently demonstrated remarkable generative capability, but their sequential nature results in significant inference latency. Existing training-free acceleration methods typically verify tokens independently, overlooking the strong co-occurrence patterns between adjacent visual tokens. This independence assumption often leads to contextual inconsistency and limits decoding efficiency. In this work, we introduce a novel training-free acceleration framework that performs phrase-level speculative verification, enabling the model to jointly validate multiple correlated tokens within each decoding window. To construct such phrase units, we analyze token co-occurrence statistics from the training corpus and group frequently co-occurring tokens into semantically coherent visual phrases. During inference, the proposed phrase-level verification evaluates aggregated likelihood ratios over each phrase, allowing simultaneous acceptance of multiple tokens while preserving generation quality. Extensive experiments on autoregressive text-to-image generation show that our method significantly reduces the number of function evaluations (NFE) and achieves up to 30% faster decoding without compromising visual fidelity. Our findings reveal that modeling short-range token co-occurrence provides an effective and general principle for accelerating autoregressive inference.