Unlocking Any-Order Generation in Pretrained Autoregressive Image Models

Rishav Pramanik; Marco Pedersoli; Zhaozheng Yin

cs.CVOct 29, 2026

Unlocking Any-Order Generation in Pretrained Autoregressive Image Models

Authors: Rishav Pramanik, Marco Pedersoli, Zhaozheng Yin

Organizations: Stony Brook University, New York, USA · International Laboratory on Learning Systems (ILLS) · LIVIA, ÉTS Montréal, QC, Canada

Abstract

Autoregressive (AR) image generators are powerful in principle but inflexible in practice: training with a fixed raster-scan order hardcodes a single factorization of the image distribution into the model, leaving it unable to inpaint, outpaint, or edit at inference time. Existing solutions, including the modern discrete-diffusion family, train from scratch on a mixture of token orderings, paying roughly 3x the compute of standard raster-scan training; discrete diffusion further sacrifices compatibility with the AR inference ecosystem (KV caching, vLLM) by construction. We argue that this cost is unnecessary. A pretrained raster-scan AR model has already learned the joint distribution over image tokens; what it lacks is not knowledge but the ability to express that knowledge through non-sequential conditional pathways. We propose a recipe for adapting pretrained raster-scan models into any-order generators using two components: future-aware positional embeddings that resolve a target-position identifiability problem at negligible overhead, and an ordering curriculum that traces a stable optimization path from the pretrained solution to a good any-order basin. Applied to four base models across two architectures and two scales (LlamaGen-L/XL, RAR-L/XL), the recipe substantially outperforms MaskGIT on ImageNet inpainting (FID 6.21 vs. 10.44) and outpainting, is competitive with MAR-L, trades a 1.2-1.9 increase in class-conditional FID for this flexibility, and costs 13-33% additional compute beyond pretraining, while leaving the standard AR inference stack untouched. Inpainting, outpainting, and editing, our results suggest, need not be added to autoregressive image models. They can be unlocked from the ones that exist. Code is available at https://github.com/Ordered-AR/unlockedadaptation.

Figures & tables

Figure 1: A single autoregressive image model, adapted from a pretrained raster-scan generator, performs inpainting (left), region-conditioned editing (center), and outpainting (right) without task-specific training, sampler modification, or auxiliary networks. The “Pretrained (raster)” column shows the same LlamaGen-L [ 1 ] model from which our model is adapted, prompted to perform the same tasks under the natural raster-scan generation procedure. Failure modes in this column reflect the structural incapacity of fixed-order AR models to condition on non-contiguous context, not weak performance at a shared task. After adaptation (“Ours”), the same weights produce coherent results across all three capabilities, varying only in which tokens are observed at inference time. Quantitative comparisons against MaskGIT [ 2 ] on inpainting and outpainting are reported in Section 4 .

Type	Model	#Para.	FID $\downarrow$	IS $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Steps
Diffusion	ADM [ 40 ]	554M	4.59	186.70	0.82	0.523	250
	LDM-4 [ 43 ]	400M	3.60	247.7	–	–	250
	DiT-XL [ 37 ]	675M	2.27	278.2	0.83	0.57	250
Bi-directional AR	MaskGIT-re [ 2 ]	227M	6.80	214.0	0.82	0.51	8
	TiTok-S-128 [ 44 ]	287M	1.97	281.8	–	–	64
Causal AR	VAR [ 4 ]	600M	2.57	302.6	0.83	0.56	10

Table 1: Class-conditional generation on ImageNet

256\times 256

. Our adapted models (rows marked Ours ) trade a

1.17

–

1.93

FID increase relative to the pretrained raster-scan baselines they are adapted from for full any-order capability (Section 4.3 ). ∗ Reported by the original paper.

Figure 2: Class-conditional generation after adaptation, further accelerated by parallel decoding. Top: pretrained LlamaGen-L (raster). Middle: our adapted model with sequential decoding. Bottom: our adapted model with parallel decoding (64 steps, $4\times$ fewer forward passes).

Model	Total epochs	Pretrained + adapted $\downarrow$	Random init $\downarrow$
LlamaGen-L	$400$ ( $300{+}100$ )	$\mathbf{4.24}$	$5.60$
LlamaGen-XL	$125$ ( $100{+}25$ )	$\mathbf{3.91}$	$4.81$

Table 2: Value of pretraining: FID at matched total epochs with the same architecture and curriculum.

	Inpainting				Outpainting
Method	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
MaskGIT (cosine)	$10.44$	$116.2$	$0.680$	$0.575$	$12.17$	$107.0$	$0.664$	$0.631$
MaskGIT (Halton)	$7.82$	$123.5$	$0.581$	$0.575$	$7.91$	$119.1$	$0.691$	$0.612$
MAR-L	$6.63$	$134.2$	$0.696$	$\mathbf{0.598}$	$6.86$	$127.2$	$\mathbf{0.736}$	$\mathbf{0.649}$
Ours	$\mathbf{6.21}$	$\mathbf{157.0}$	$\mathbf{0.710}$	$\mathbf{0.598}$	$\mathbf{5.84}$	$\mathbf{159.4}$	$0.720$	$0.637$

Table 3: Inpainting and outpainting on ImageNet

256\times 256

, evaluated on

50{,}000

samples with TF-Inception. Inpainting uses center masks; outpainting extends a centered

128\times 128

crop to full resolution. Lower is better for FID; higher for IS, Precision, Recall.

Method	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
MaskGIT (cosine)	$0.234$	$19.20\pm 3.60$	$0.484\pm 0.21$
MaskGIT (Halton)	$0.196$	$18.20\pm 3.50$	$0.452\pm 0.23$
MAR-L	$0.174$	$20.15\pm 4.60$	$0.550\pm 0.20$
Ours	$\mathbf{0.125}$	$\mathbf{24.65\pm 3.34}$	$\mathbf{0.682\pm 0.17}$

Table 4: Reference-based completion metrics. LPIPS: random region covering

30

–

50\%

5{,}000

held-out images. PSNR/SSIM (mean

\pm

std): generated region of the

50{,}000

center-mask inpainting examples.

Variant	FID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
No curriculum ( $\alpha=1$ from step $0$ )	$5.86$	$191.27$	$0.78$	$0.51$
Ours (linear curriculum)	$\mathbf{4.24}$	$\mathbf{231.62}$	$\mathbf{0.83}$	$\mathbf{0.52}$

Table 5: Effect of the ordering curriculum on LlamaGen-L. Adapting with

\alpha=1

from step zero (no curriculum) leaves the model in a partially adapted state across all metrics; the linear curriculum used in our main results recovers the gap.

Method	Train cost	Any-order	KV cache	vLLM
LlamaGen pretraining [ 1 ] (raster only)	$1\times$	✗	✓	✓
RandAR [ 18 ] (from scratch)	$\text{3}\times$	✓	✓	✗
RAR [ 5 ] (from scratch)	$\text{4}\times$	✗	✓	✓
Discrete diffusion [ 2 ]	$\text{3}\times$	✓	✗	✗
Ours (LlamaGen-XL + adaptation)	$\mathbf{1.25\times}$	✓	✓	✓

Table 6: Training and inference compatibility relative to standard AR pretraining. Our adaptation recipe achieves any-order generation at

\approx 1.25\times

the cost of standard pretraining (LlamaGen-XL), while preserving compatibility with the AR inference ecosystem.

Generation steps	$576$	$192$	$128$	$64$
FID $\downarrow$	$3.91$	$4.12$	$4.81$	$5.03$

Table 7: Parallel decoding on LlamaGen-XL: generation steps vs. class-conditional FID.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Optimization
	LlamaGen-L	LlamaGen-XL	RAR-L	RAR-XL
Optimizer	AdamW	AdamW	AdamW	AdamW
$(\beta_{1},\beta_{2})$	$(0.9,0.95)$	$(0.9,0.95)$	$(0.9,0.95)$	$(0.9,0.95)$
Weight decay	$0.05$	$0.05$	$0.05$	$0.05$
Base learning rate	$1\!\times\!10^{-4}$	$1\!\times\!10^{-4}$	$5\!\times\!10^{-5}$	$5\!\times\!10^{-5}$
Embedding-table LR multiplier	$10\times$	$10\times$	$10\times$	$10\times$

Appendix

Table 8: Adaptation hyperparameters by base model. Notation matches Section 3 .

Autoregressive Image Generation

Autoregressive image generation predicts visual tokens or scales sequentially to synthesize images, offering controllable alternatives to diffusion.

See all 115 papers

Revisiting Autoregressive Models for Generative Image Classification
Mar 19, 2026Ilia Sudakov, Artem Babenko, Dmitry BaranchukVisual Autoregressive Models Image Classification
1HSE University · 2Yandex Research
FlashAR: Efficient Post-Training Acceleration for Autoregressive Image Generation
May 10, 2026Junkang Zhou, Yefei He, Feng Chen +2Parallel Decoding Autoregressive Image Generation
Zhejiang University, China · University of Adelaide, Australia
End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer
May 1, 2026Wenda Chu, Bingliang Zhang, Jiaqi Han +4Image Tokenization Visual Tokenization
ByteDance Seed · California Institute of Technology · Stanford University

Mar 19, 2026J/K move · Enter open · S save

Revisiting Autoregressive Models for Generative Image Classification

Ilia Sudakov, Artem Babenko, Dmitry Baranchuk

1HSE University · 2Yandex Research

Visual Autoregressive Models Image Classification Autoregressive Modeling

Open paper AskPDF arXiv

Class-conditional generative models have emerged as accurate and robust classifiers, with diffusion models demonstrating clear advantages over other visual generative paradigms, including autoregressive (AR) models. In this work, we revisit visual AR-based generative classifiers and identify an important limitation of prior approaches: their reliance on a fixed token order, which imposes a restrictive inductive bias for image understanding. We observe that single-order predictions rely more on partial discriminative cues, while averaging over multiple token orders provides a more comprehensive signal. Based on this insight, we leverage recent any-order AR models to estimate order-marginalized predictions, unlocking the high classification potential of AR models. Our approach consistently outperforms diffusion-based classifiers across diverse image classification benchmarks, while being up to 25x more efficient. Compared to state-of-the-art self-supervised discriminative models, our method delivers competitive classification performance - a notable achievement for generative classifiers.

Abstract

Autoregressive Image Generation

Explore similar work

Revisiting Autoregressive Models for Generative Image Classification

FlashAR: Efficient Post-Training Acceleration for Autoregressive Image Generation

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer