Know, then Draw: A New Learning Paradigm for Visual Generation

Yucheng Mao; Chongjian Ge; Zhuowen Tu; Hao Tan; Hanwen Jiang

cs.CVOct 27, 2026

Know, then Draw: A New Learning Paradigm for Visual Generation

Authors: Yucheng Mao, Chongjian Ge, Zhuowen Tu, Hao Tan, Hanwen Jiang

Organizations: University of California, San Diego · Adobe Research

Abstract

Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.

Figures & tables

Figure 2: Overview of our learning paradigm. Autoregressive pretraining acquires visual knowledge through discrete token prediction. The pretrained backbone is then frozen and augmented with diffusion modules that reuse its attention and representations to render high-fidelity images. This division of labor specializes autoregression for knowledge acquisition and diffusion for rendering.

Figure 3: Training the three visual tokenizers. Recon and Semantic-Recon use image reconstruction supervision, whereas Semantic-LM learns its codebook through language modeling.

Method	GenEval $\uparrow$							DPG $\uparrow$
	Overall	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.
PDM	61.16	97.50	47.47	33.75	87.23	50.00	51.00	62.56
LDM	57.34	97.50	55.56	27.50	74.47	38.00	51.00	64.33
LDM $\xrightarrow{}$ PDM	56.61	90.00	60.61	32.50	75.53	39.00	42.00	60.01
AR $\xrightarrow{}$ PDM (ours)	65.51	97.50	66.67	40.00	81.91	51.00	56.00	71.11

Tab. 1: Comparison between different learning paradigms under a matched data budget. Final-generator scores as percentages (

\uparrow

); GenEval Overall is the mean of the six category scores.

Tokenizer Properties			AR-only Generation		Post-train Diffusion Generation
Tokenizer	Supervision	rFID $\downarrow$	GenEval $\uparrow$	DPG $\uparrow$	GenEval $\uparrow$	$\Delta$ GenEval	DPG $\uparrow$	$\Delta$ DPG
Recon	Recon	1.04	57.49	65.94	54.67	$-2.82$	62.98	$-3.04$
Sem-Recon	Recon	40.5	50.44	67.03	65.51	$+15.07$	71.11	$+4.08$
Sem-LM	LM	$\infty$	0.00	0.00	64.84	$+64.84$	67.72	$+67.72$

Tab. 2: Semantic quality, rather than reconstruction or AR-only generation quality, determines final generation performance. Recon performs best in reconstruction and AR-only generation, yet produces the weakest final generator; Sem-LM remains strong downstream despite having no reconstruction supervision or image decoder.

\Delta

denotes the change after diffusion post-training.

Figure 4: AR-only generation quality does not predict final generation quality. Top: AR-only decodes; bottom: final images after diffusion post-training. The semantic tokenizers produce stronger final images despite weaker or unavailable AR-only decodes. Semantic-LM has no decoder.

Figure 5: Scaling laws for Pretraining and Post-training. (a) AR pretraining; (b) pixel-diffusion post-training. Left: training loss versus full-sequence compute on logarithmic axes, with the lower envelope highlighted in gray; color bars indicate model size. Middle and right: envelope-optimal model and token allocations. Fitting protocols are given in Appendix A.4 and Appendix A.5 .

		GenEval							DPG
Model	Size (B)	Single	Two	Count	Colors	Pos.	C-attr.	Overall $\uparrow$	Overall $\uparrow$
PixArt- $\alpha$ ( Chen et al., 2023 )	0.6	0.98	0.50	0.44	0.80	0.08	0.07	0.48	71.11
SDXL ( Podell et al., 2023 )	2.6	0.98	0.74	0.39	0.85	0.15	0.23	0.55	74.65
DALL-E 3 ( Betker et al., 2023 )	–	0.96	0.87	0.47	0.83	0.43	0.45	0.67	83.50
PixNerd-XXL/16 ( Wang et al., 2026 )	1.2	0.97	0.86	0.44	0.83	0.71	0.53	0.73	80.90
SD3-Medium ( Esser et al., 2024 )	2.0	0.99	0.94	0.72	0.89	0.33	0.60	0.74	84.08

Tab. 3: Text-to-image generation on GenEval and DPG-Bench. Model size is in billions of parameters (B), as reported for each system; component accounting may differ. Dash denotes unavailable results. We use LLM to rewrite prompt in order to match the training distribution.

Figure 6: Compute-optimal post-training share

r

against model size.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	PDM	LDM	LDM $\xrightarrow{}$ PDM	AR $\xrightarrow{}$ PDM
Architecture and representation
Backbone width	1408	1280	1280	1408
Layers / heads	16 / 11	15 / 10	8 / 10	8 / 11
FFN width	3840	3584	3584	3840
Representation	Pixels	FLUX.2	SVG Autoencoder-P	Three tokenizers
Latent channels	—	128 (packed)	384	Discrete codes

Appendix

Tab. 4: Training recipes for the controlled generation comparisons. All routes match cumulative image exposure and use approximately 400M final parameters. Pretraining–post-training budgets include both stages; all AR tokenizer variants share the AR

\xrightarrow{}

PDM recipe.

Sweep	Width	Layers	Heads	FFN	Parameters
Depth	512	4	4	1536	48.9M
	512	8	4	1536	62.5M
	512	12	4	1536	76.2M
	512	24	4	1536	117.1M
	512	32	4	1536	144.4M
Shared	512	16	4	1536	89.8M

Appendix

Tab. 5: Architectures used for AR hyperparameter transfer. Each row is trained at all 12 base learning rates. The reference architecture appears in both panels of Fig. 7 . Parameter counts include all trainable AR parameters, count the tied code table once, and exclude the frozen visual tokenizer and text encoder. FFN denotes feed-forward hidden dimension; each attention head has dimension 128. Parameter counts are rounded to one decimal place in millions (M).

Panel of Figure 7 — Figure 7: Pretraining learning-rate transfer across width and depth. Each run trains for 50,000 updates. Each point averages 21 raw training cross-entropy measurements; crosses mark the minimum among the 12 tested base learning rates.

$N$ (M)	Layers (depth)	Hidden dim. $W$	Heads	Head dim.	FFN dim. $F$
55.71	6	512	4	128	1,536
71.30	7	640	5	128	1,792
99.96	9	768	6	128	2,048
137.73	10	896	7	128	2,560
191.53	12	1,024	8	128	2,816
260.95	14	1,152	9	128	3,072

Appendix

Tab. 6: AR architectures used for the scaling fits. Models are sorted by trainable parameters

N

;

W

and

F

denote hidden and FFN dimensions. Shared vocabulary weights are counted once; frozen encoders and tokenizers are excluded.

Method	$N_{20}$ (M)	$D_{20}$ (B)	$a$	$b$
Envelope	296.96	56.12	0.5268	0.4732
IsoFLOP	270.58	61.60	0.4705	0.5295
Parametric	199.05	83.73	0.4823	0.5177

Appendix

Tab. 7: Full-sequence AR allocation laws.

N=N_{20}(C/10^{20})^{a}

and

D=D_{20}(C/10^{20})^{b}

Figure 8: Post-training learning-rate search. Mean training loss at approximately 20,000 updates for four width/depth pairs. Each point averages 41 measurements; the largest model uses its final available windows. Crosses mark the minimum for each model.

Parameters (M)	Layers	Width	Heads	FFN width
25.13	6	512	4	1536
43.46	7	640	5	1792
77.23	9	768	6	2048
120.73	10	896	7	2560
184.00	12	1024	8	2816
265.85	14	1152	9	3072

Appendix

Tab. 8: Post-training model family, ordered by trainable diffusion parameter count. Attention heads have dimension 128. The AR backbone and external encoders remain frozen and are excluded from the parameter counts.

Method	$N_{20}$ (M)	$D_{20}$ (B)	$a$	$b$
Parametric	513.48	32.46	0.7021	0.2979
Envelope	395.28	42.16	0.5492	0.4508
IsoFLOP	377.99	44.09	0.4538	0.5462

Appendix

Tab. 9: Post-training allocation fits. With

x=C/10^{20}

N_{\rm opt}=N_{20}x^{a}

and

D_{\rm opt}=D_{20}x^{b}

, using full-sequence token counts.

Panel of Figure 9 — Figure 9: Pixel-diffusion learning-rate transfer. Original width and depth search results after 50,000 training updates. Points average the final 10,000 updates; crosses mark the best tested learning rate.

Sweep	Width	Layers	Best base LR	Training loss
Width	256	16	$8\times 10^{-4}$	0.1405
Width	384	16	$8\times 10^{-4}$	0.1309
Width	512	16	$10^{-3}$	0.1258
Width	640	16	$10^{-3}$	0.1231
Width	768	16	$10^{-3}$	0.1214
Width	1024	16	$8\times 10^{-4}$	0.1195

Appendix

Tab. 10: Grid minima in the original PDM hyperparameter search. The shared width-512, depth-16 architecture is evaluated in both sweeps.

Parameters (M)	Layers	Width	Heads	FFN width
23.83	6	512	4	1536
69.97	9	768	6	2048
164.08	12	1024	8	2816
319.16	15	1280	10	3584
529.29	18	1536	12	4096
844.62	21	1792	14	4864

Appendix

Tab. 11: Pure PDM scaling architectures, ordered by trainable parameter count. All attention heads have dimension 128.

Method	$N_{20}$ (M)	$D_{20}$ (B)	$a$	$b$
Envelope	394.37	42.26	0.5744	0.4256
IsoFLOP	409.73	40.68	0.5161	0.4839
Parametric	535.04	31.15	0.7143	0.2857

Appendix

Tab. 12: Full-sequence PDM allocation laws. For

x=C/10^{20}

N_{\rm opt}=N_{20}x^{a}

and

D_{\rm opt}=D_{20}x^{b}

Figure 10: Pixel-diffusion scaling from scratch. Full-sequence compute accounting for eight model sizes. Top: training losses and envelope allocations. Bottom: six representative IsoFLOP profiles and allocation fits using all nine accepted minima. Stars mark the quadratic minima.

$N_{\mathrm{AR}}$ (M)	$N_{\mathrm{diff}}$ (M)	$C_{\mathrm{pre}}$	$C_{\mathrm{post}}$	$r$
260.95	265.85	$7.82\times 10^{19}$	$4.86\times 10^{19}$	0.383
343.39	360.77	$1.32\times 10^{20}$	$8.47\times 10^{19}$	0.391
425.72	457.95	$1.98\times 10^{20}$	$1.31\times 10^{20}$	0.398
549.65	604.00	$3.22\times 10^{20}$	$2.16\times 10^{20}$	0.402
688.28	763.60	$4.93\times 10^{20}$	$3.32\times 10^{20}$	0.402
860.44	966.18	$7.53\times 10^{20}$	$5.09\times 10^{20}$	0.403

Appendix

Tab. 13: Induced pretraining–post-training budget allocation at the envelope compute-optimal. For each jointly scaled model, each stage is placed at its own envelope optimum and

r=C_{\mathrm{post}}/(C_{\mathrm{pre}}+C_{\mathrm{post}})

N

excludes frozen modules;

C

uses the full-sequence

6ND

proxy. Models below

261

M are omitted as noisy.

Figure 11: Selected generation samples from our final text-to-image model.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Generative Refinement Networks for Visual Synthesis
Apr 14, 2026Jian Han, Jinlai Liu, Jiahuan Wang +2Binary Quantization T2V Generation
ByteDance
Visual Autoregressive Generation with Next Token Imagination
Oct 27, 2026Ruihang Jiang, Zongyao Zhao, Zhaolin Wang +2Visual Autoregressive Models Autoregressive Image Generation
University of Hong Kong · University of Surrey
Autoregressive Visual Generation Needs a Prologue
May 7, 2026Bowen Zheng, Weijian Luo, Guang Yang +2Visual Autoregressive Models Autoregressive Image Generation
The Chinese University of Hong Kong, Shenzhen · hi-Lab, Xiaohongshu Inc

Apr 14, 2026J/K move · Enter open · S save

Generative Refinement Networks for Visual Synthesis

Jian Han, Jinlai Liu, Jiahuan Wang, Bingyue Peng+1

ByteDance

Binary Quantization T2V Generation T2I Generation Image Reconstruction Autoregressive Image Generation

Open paper AskPDF arXiv

While diffusion models dominate the field of visual generation, they are computationally inefficient, applying a uniform computational effort regardless of different complexity. In contrast, autoregressive (AR) models are inherently complexity-aware, as evidenced by their variable likelihoods, but are often hindered by lossy discrete tokenization and error accumulation. In this work, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm that addresses these issues. At its core, GRN addresses the discrete tokenization bottleneck through a theoretically near-lossless Hierarchical Binary Quantization (HBQ), achieving a reconstruction quality comparable to continuous counterparts. Built upon HBQ's latent space, GRN fundamentally upgrades AR generation with a global refinement mechanism that progressively perfects and corrects artworks -- like a human artist painting. Besides, GRN integrates an entropy-guided sampling strategy, enabling complexity-aware, adaptive-step generation without compromising visual quality. On the ImageNet benchmark, GRN establishes new records in image reconstruction (0.56 rFID) and class-conditional image generation (1.81 gFID). We also scale GRN to more challenging text-to-image and text-to-video generation, delivering superior performance on an equivalent scale. We release all models and code to foster further research on GRN.

Abstract

T2I Generation

Explore similar work

Generative Refinement Networks for Visual Synthesis

Visual Autoregressive Generation with Next Token Imagination

Autoregressive Visual Generation Needs a Prologue