Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation

Guo Niu; Xiongfei Yao; Teng Wang; Nannan Zhu

cs.CVAug 26, 2026

Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation

Authors: Guo Niu, Xiongfei Yao, Teng Wang, Nannan Zhu

Organizations: Foshan University Foshan, China · The University of Hong Kong Hong Kong, China · Sun Yat-sen University Guangzhou, China

Abstract

Autoregressive (AR) image generation has shown strong potential for scalable high-fidelity synthesis by modeling images as discrete token sequences. However, traditional next token prediction (NTP) continues to suffer from sparse and myopic supervision, insufficiently discriminative representations, and high training cost caused by dense computation over the full token sequence. To address these issues, we propose multi-token autoregressive (MTAR), a unified training framework that improves autoregressive image generation from three aspects: prediction objectives, representation regularization, and training efficiency. Specifically, MTAR introduces multi-token prediction (MTP) to alleviate the sparsity and myopia of traditional NTP by imposing joint supervision on multiple future tokens; employs token-level contrastive regularization (TCR) to explicitly enhance the separability of sampled token representations and thereby improve representation discriminability; and incorporates semantic dropping (SD) as a semantics-aware training acceleration strategy to reduce redundant computation on low-information tokens while preserving informative learning signals. All three components are applied only during training and introduce no additional overhead during autoregressive inference. On ImageNet, MTAR achieves a better balance between generation quality and training efficiency. Compared with LlamaGen, MTAR achieves up to 0.95 lower FID and 39% faster training. Moreover, even with only 1/3 of the training iterations, it still attains performance comparable to or better than the baseline, substantially reducing training time.

Figures & tables

Figure 1. (a) Traditional NTP provides only short-range supervision over the full token sequence, leading to sparse and myopic training signals, while requiring autoregressive modeling over the entire sequence with high computational cost. (b) Our framework improves training with MTP and SD. MTP introduces more forward-looking and denser auxiliary supervision signals, thus enhancing contextual dependency modeling and alleviating the myopic nature of conventional single-step prediction, as indicated by the orange-red dashed arrows. SD further improves training efficiency by discarding less informative patches; the white patches denote the dropped regions. A two-panel schematic comparing conventional next-token prediction with the proposed training framework. Panel (a) shows all image tokens retained and connected primarily through short-range, single-step prediction signals. Panel (b) shows retained tokens receiving additional orange-red dashed connections to more distant future targets, representing multi-token prediction. Several image patches are colored white to indicate tokens removed by semantic dropping, resulting in a shorter training sequence.

Figure 2. Comparison of total training time (hours) and total GPU memory usage across all training GPUs between LlamaGen ( Sun et al., 2024 ) and MTAR for the Base and Large models, along with the corresponding FID values. LlamaGen is trained for 300 epochs, whereas MTAR is trained for only 100 epochs. For the efficiency measurements, the Base models are trained on a single NVIDIA A800 GPU, while the Large models are trained on two NVIDIA A800 GPUs. MTAR consistently achieves better generation quality and efficiency than the corresponding LlamaGen baselines: for the Base model, FID improves from 5.46 to 5.37, while training time and memory are reduced from 115 to 30 and from 45 GiB to 31 GiB; for the Large model, FID improves from 3.80 to 3.36, while training time and memory are reduced from 175 to 42 and from 97 GiB to 73 GiB. Both methods use torch.compile . A grouped comparison of LlamaGen and MTAR across the Base and Large model settings, showing training time, GPU memory usage, and FID. MTAR has lower values than LlamaGen for all three metrics in both settings.

Figure 3. Overview of the proposed framework. (a) Overall framework under the MTP (R 1 , B) setting: besides the original NTP (R 1 ) head, an additional MTP head is introduced to provide auxiliary supervision for the token directly below the current token. (b) TCR: Sampled token features are passed through two Dropout-perturbed branches and a shared MLP to construct a cross-view similarity matrix. The diagonal entries bring matched token pairs closer, while the off-diagonal entries push different sampled tokens apart. A two-panel overview of MTAR. Panel (a) shows the NTP head predicting the next token and an MTP head predicting the token below. Panel (b) shows TCR constructing a similarity matrix from two Dropout-perturbed views of sampled token features.

Figure 4. Semantic Drop. (a) DINOv3 ( Siméoni et al., 2025 ) extracts a token importance map for each image (darker: higher semantic importance). Low-importance tokens are dropped, retaining only semantically salient tokens for training. (b) Retained tokens preserve their original spatial indices, and both NTP and MTP losses are computed accordingly. DINOv3 identifies important image tokens, which retain their original positions for NTP and MTP training.

Type	Model	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
GAN	BigGAN ( Brock, 2018 )	112M	6.95	224.5	0.89	0.38
GAN	GigaGAN ( Kang et al., 2023 )	569M	3.45	225.5	0.84	0.61
Diffusion	ADM ( Dhariwal and Nichol, 2021 )	554M	10.94	101.0	0.69	0.63
	CDM ( Ho et al., 2022 )	–	4.88	158.7	–	–
	LDM-4-G ( Rombach et al., 2022 )	400M	3.60	247.7	0.87	0.48
	DiT-L/2 ( Peebles and Xie, 2023 )	458M	5.02	167.2	0.75	0.57

Table 1. Model comparisons on the class-conditional ImageNet

256\times 256

benchmark ( Deng et al., 2009 ) . Metrics include Fréchet inception distance (FID), inception score (IS), precision, and recall. “

\downarrow

” and “

\uparrow

” indicate that lower and higher values are better, respectively. † denotes results reported by NAR ( He et al., 2025 ) . “-re” denotes the use of rejection sampling.

Model	Epochs	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Speedup
LlamaGen-B ( Sun et al., 2024 )	300	5.46	193.61	0.84	0.46	1.00 $\times$
MTAR-B	50	5.76	198.70	0.86	0.42	7.62 $\times$
	100	5.37	209.07	0.86	0.44	3.81 $\times$
	250	4.66	211.54	0.84	0.47	1.73 $\times$
	300	4.50	213.96	0.84	0.48	1.27 $\times$
LlamaGen-L ( Sun et al., 2024 )	300	3.80	248.28	0.83	0.52	1.00 $\times$

Table 2. Comparison between LlamaGen ( Sun et al., 2024 ) and MTAR in terms of generation quality and training efficiency. Speedup is measured with respect to the total training time of the baseline LlamaGen ( Sun et al., 2024 ) .

Method	Right (R 2 )	Bottom (B)	Bottom-Right (B.R 1 )	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
MTP (R 1 , R 2 )	✓			138M	20.13	59.32	0.75	0.45
MTP (R 1 , B)		✓		138M	19.75	62.12	0.75	0.45
MTP (R 1 , B.R 1 )			✓	138M	19.96	61.05	0.75	0.44

Table 3. Ablation of MTP head directions based on the NTP head (R 1 ). We compare auxiliary prediction along three directions: right, bottom, and bottom-right.

Method	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Speedup
LlamaGen-B	111M	21.96	54.04	0.71	0.46	1.00 $\times$
MTP	138M	19.75	62.12	0.75	0.45	0.79 $\times$
TCR	111M	20.55	57.78	0.72	0.47	0.98 $\times$
SD	111M	22.02	54.99	0.70	0.47	1.62 $\times$
MTP+TCR	138M	18.78	64.33	0.73	0.48	0.79 $\times$
MTP+SD	138M	20.79	66.17	0.71	0.47	1.27 $\times$

Table 4. Ablation study of MTP, TCR, and SD on LlamaGen-B ( Sun et al., 2024 ) .

Method	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
MTP (R 1 , R 2 , R 3 )	164M	19.79	41.77	0.74	0.47
MTP (R 1 , R 2 , B)	164M	20.34	59.85	0.72	0.46
MTP (R 1 , B, B.R 1 )	164M	19.94	62.33	0.74	0.46
MTP (R 1 , R 2 , R 3 , R 4 )	191M	20.95	57.15	0.73	0.45
MTP (R 1 , R 2 , B, B.R 1 )	191M	20.95	57.18	0.73	0.45
MTP (R 1 , R 2 , R 3 , R 4 , R 5 )	218M	20.18	59.46	0.74	0.45

Table 5. Ablation on the number of MTP heads. Performance does not consistently improve as more auxiliary heads are added.

Sampled Tokens	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
16384	20.70	58.60	0.73	0.46
8192	19.83	58.62	0.73	0.47
4096	19.71	57.82	0.72	0.48
2048	18.78	64.33	0.73	0.48
1024	19.18	63.54	0.73	0.48
512	18.94	64.45	0.74	0.48

Table 6. Ablation study on the number of sampled tokens for TCR. We observe that using 2,048 sampled tokens yields the best overall performance.

Drop Strategy	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
Random	19.97	62.69	0.72	0.45
SigLIP2 ( Tschannen et al., 2025 )	19.70	63.84	0.73	0.46
DINOv3 ( Siméoni et al., 2025 )	18.65	66.32	0.75	0.46

Table 7. Ablation on drop strategies. DINOv3 ( Siméoni et al., 2025 ) -guided SD achieves the best performance.

Drop Rate	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Speedup
60%	19.31	62.93	0.74	0.45	1.81 $\times$
50%	18.65	66.32	0.75	0.46	1.60 $\times$
40%	18.91	65.32	0.74	0.46	1.38 $\times$

Table 8. Ablation on the patch drop rate in SD. A 50% drop rate achieves the best overall trade-off. Speedup is measured relative to the MTP+TCR configuration without SD.

SD Schedule	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Speedup
40%:60%	18.65	65.54	0.75	0.46	1.23 $\times$
50%:50%	18.60	65.01	0.74	0.48	1.31 $\times$
60%:40%	18.41	64.85	0.74	0.48	1.39 $\times$
70%:30%	18.68	64.38	0.74	0.48	1.49 $\times$
80%:20%	18.65	66.32	0.75	0.46	1.60 $\times$
90%:10%	19.40	62.89	0.75	0.46	1.73 $\times$

Table 9. Ablation on the two-stage SD training schedule. The 80%:20% schedule provides a favorable quality–efficiency trade-off. Speedup is measured relative to the MTP+TCR configuration without SD.

Figure 5. We retain the top-50% patches according to the importance scores produced by SigLIP2 ( Tschannen et al., 2025 ) and DINOv3 ( Siméoni et al., 2025 ) , and visualize the corresponding VQ-decoded reconstructions. Compared with SigLIP2 ( Tschannen et al., 2025 ) , DINOv3 ( Siméoni et al., 2025 ) better preserves the semantic content and structural integrity of the foreground object. VQ-decoded reconstructions using the top-50\% patches selected by SigLIP2 and DINOv3, with DINOv3 preserving foreground objects more clearly. — Figure 5. We retain the top-50% patches according to the importance scores produced by SigLIP2 ( Tschannen et al., 2025 ) and DINOv3 ( Siméoni et al., 2025 ) , and visualize the corresponding VQ-decoded reconstructions. Compared with SigLIP2 ( Tschannen et al., 2025 ) , DINOv3 ( Siméoni et al., 2025 ) better preserves the semantic content and structural integrity of the foreground object. VQ-decoded reconstructions using the top-50% patches selected by SigLIP2 and DINOv3, with DINOv3 preserving foreground objects more clearly.

Figure 6. Qualitative comparison between LlamaGen-L ( Sun et al., 2024 ) and MTAR-L on ImageNet $256\times 256$ ( Deng et al., 2009 ) . Under the same class conditions, MTAR-L produces images with clearer semantics, more coherent structures, and fewer artifacts. Red boxes highlight representative artifacts in LlamaGen-L ( Sun et al., 2024 ) samples. ImageNet samples generated by LlamaGen-L and MTAR-L, with red boxes marking artifacts in the LlamaGen-L results.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

config	value
training hyper-parameters
optimizer	AdamW
learning rate	1e-4
weight decay	5e-2
optimizer momentum	$(0.9,0.95)$
batch size	256

Appendix

Table 10. Detailed training and sampling hyper-parameters of MTAR.

SD Schedule	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Speedup $\uparrow$
40%:5%:55%	18.62	66.47	0.74	0.47	1.25 $\times$
40%:10%:50%	18.77	63.01	0.75	0.46	1.27 $\times$
50%:5%:45%	18.56	67.11	0.75	0.46	1.33 $\times$
50%:10%:40%	18.99	65.04	0.75	0.46	1.35 $\times$
60%:5%:35%	18.20	68.25	0.75	0.47	1.42 $\times$
60%:10%:30%	18.56	65.77	0.74	0.46	1.44 $\times$

Appendix

Table 11. Ablation study on three-stage SD schedules. The three entries denote the relative durations of stages with patch-drop rates of 50%, 25%, and 0%, respectively. Different schedules lead to different quality–efficiency trade-offs, and 60%:5%:35% achieves the best FID among the evaluated three-stage variants.

Method	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
RAR-B ( Yu et al., 2025 )	285M	18.70	72.28	0.71	0.50
RAR-B+MTP(B)	320M	17.82	76.49	0.73	0.51

Appendix

Table 12. Generalization of MTP(B) to RAR-B ( Yu et al., 2025 ) . Adding MTP(B) consistently improves FID, IS, precision, and recall, demonstrating that multi-token auxiliary supervision transfers beyond the LlamaGen backbone.

Figure 7. Optimization behavior under different MTP-head configurations. (a) Primary autoregressive loss

L(R_{1})

across training epochs for single-head and multi-head prediction sets. (b) Primary-head loss

L(R_{1})

and auxiliary-head loss

L(B)

for the

(R_{1},B)

configuration. The comparison illustrates how additional prediction targets alter the optimization of the main autoregressive objective and motivates limiting the number of auxiliary heads to avoid excessive interference.

Figure 8. Token-level linear-probing accuracy on the ImageNet validation set. MTAR-L consistently outperforms LlamaGen-L across all Top-

N

settings, indicating better linear separability of visual-token representations.

Similarity	Ratio
Sim $>$ 0.9	2.01%
Sim $>$ 0.8	8.73%

Appendix

Table 13. Proportions of highly similar token pairs under different cosine similarity thresholds.

Figure 9. Randomly selected examples before and after SD for the ImageNet classes picket fence and spider web , using 50% token retention.

Figure 10. Generated samples from MTAR-L for ImageNet class ID 269 (wolf).

Figure 11. Generated samples from MTAR-L for ImageNet class ID 323 (butterfly).

Figure 12. Generated samples from MTAR-L for ImageNet class ID 90 (lorikeet).

Figure 13. Generated samples from MTAR-L for ImageNet class ID 979 (valley).

Figure 14. Uncurated random samples from LlamaGen and MTAR.

Figure 15. Comparison of class-conditional samples produced by LlamaGen-L and MTAR-L on ImageNet $256\times 256$ .

Figure 16. Additional comparison of class-conditional samples produced by LlamaGen-L and MTAR-L on ImageNet $256\times 256$ .

Explore similar work

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer
May 1, 2026Wenda Chu, Bingliang Zhang, Jiaqi Han +4Image Tokenization Visual Tokenization
ByteDance Seed · California Institute of Technology · Stanford University
ScalingAR: Scaling Confidence for Autoregressive Image Generation
Sep 30, 2025Harold Haodong Chen, Xianfeng Wu, Wen-Jie Shu +4Language Model Decoding Visual Autoregressive Models
HKUST(GZ) · UNC-Chapel Hill · ZODA +3
SJD-PV: Speculative Jacobi Decoding with Phrase Verification for Autoregressive Image Generation
Mar 2, 2026Zhehao Yu, Baoquan Zhang, Bingqi Shan +5Speculative Decoding Autoregressive Image Generation
Harbin Institute of Technology, Shenzhen · SIFAR

May 1, 2026J/K move · Enter open · S save

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li+3

ByteDance Seed · California Institute of Technology · Stanford University

Image Tokenization Visual Tokenization Autoregressive Image Generation Visual Token Compression

Open paper AskPDF arXiv

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.

Abstract

Explore similar work

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

ScalingAR: Scaling Confidence for Autoregressive Image Generation

SJD-PV: Speculative Jacobi Decoding with Phrase Verification for Autoregressive Image Generation