Visual generation requires both high-level semantic knowledge and fine-grained pixel details. We propose a learning paradigm that assigns these roles to autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Within this framework, we study the objective, representation, and scaling of knowledge acquisition and rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality. We further find that AR-only generation quality does not necessarily correlate with final performance; semantic quality matters more. This motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss. We establish compute-optimal scaling laws for knowledge acquisition and rendering, and find that autoregressive pretraining lowers the fitted asymptotic rendering loss compared with pixel diffusion models trained from scratch. Guided by the laws, we train a 3B-parameter text-to-image model with only 6,600 H100 GPU-hours, achieving 86 on GenEval and 87 on DPG-Bench and validating the proposed paradigm at scale.
Figures & tables
Figure 2: Overview of our learning paradigm. Autoregressive pretraining acquires visual knowledge through discrete token prediction. The pretrained backbone is then frozen and augmented with diffusion modules that reuse its attention and representations to render high-fidelity images. This division of labor specializes autoregression for knowledge acquisition and diffusion for rendering.
Figure 3: Training the three visual tokenizers. Recon and Semantic-Recon use image reconstruction supervision, whereas Semantic-LM learns its codebook through language modeling.
Method
GenEval ↑
DPG ↑
Overall
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
PDM
61.16
97.50
47.47
33.75
87.23
50.00
51.00
62.56
LDM
57.34
97.50
55.56
27.50
74.47
38.00
51.00
64.33
LDM PDM
56.61
90.00
60.61
32.50
75.53
39.00
42.00
60.01
AR PDM (ours)
65.51
97.50
66.67
40.00
81.91
51.00
56.00
71.11
Tab. 1: Comparison between different learning paradigms under a matched data budget. Final-generator scores as percentages ( ↑ ); GenEval Overall is the mean of the six category scores.
Tokenizer Properties
AR-only Generation
Post-train Diffusion Generation
Tokenizer
Supervision
rFID ↓
GenEval ↑
DPG ↑
GenEval ↑
Δ GenEval
DPG ↑
Δ DPG
Recon
Recon
1.04
57.49
65.94
54.67
−2.82
62.98
−3.04
Sem-Recon
Recon
40.5
50.44
67.03
65.51
+15.07
71.11
+4.08
Sem-LM
LM
∞
0.00
0.00
64.84
+64.84
67.72
+67.72
Tab. 2: Semantic quality, rather than reconstruction or AR-only generation quality, determines final generation performance. Recon performs best in reconstruction and AR-only generation, yet produces the weakest final generator; Sem-LM remains strong downstream despite having no reconstruction supervision or image decoder. Δ denotes the change after diffusion post-training.
Figure 4: AR-only generation quality does not predict final generation quality. Top: AR-only decodes; bottom: final images after diffusion post-training. The semantic tokenizers produce stronger final images despite weaker or unavailable AR-only decodes. Semantic-LM has no decoder.
Figure 5: Scaling laws for Pretraining and Post-training. (a) AR pretraining; (b) pixel-diffusion post-training. Left: training loss versus full-sequence compute on logarithmic axes, with the lower envelope highlighted in gray; color bars indicate model size. Middle and right: envelope-optimal model and token allocations. Fitting protocols are given in Appendix A.4 and Appendix A.5 .
GenEval
DPG
Model
Size (B)
Single
Two
Count
Colors
Pos.
C-attr.
Overall ↑
Overall ↑
PixArt- α ( Chen et al., 2023 )
0.6
0.98
0.50
0.44
0.80
0.08
0.07
0.48
71.11
SDXL ( Podell et al., 2023 )
2.6
0.98
0.74
0.39
0.85
0.15
0.23
0.55
74.65
DALL-E 3 ( Betker et al., 2023 )
–
0.96
0.87
0.47
0.83
0.43
0.45
0.67
83.50
PixNerd-XXL/16 ( Wang et al., 2026 )
1.2
0.97
0.86
0.44
0.83
0.71
0.53
0.73
80.90
SD3-Medium ( Esser et al., 2024 )
2.0
0.99
0.94
0.72
0.89
0.33
0.60
0.74
84.08
Tab. 3: Text-to-image generation on GenEval and DPG-Bench. Model size is in billions of parameters (B), as reported for each system; component accounting may differ. Dash denotes unavailable results. We use LLM to rewrite prompt in order to match the training distribution.
Figure 6: Compute-optimal post-training share r against model size.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
PDM
LDM
LDM PDM
AR PDM
Architecture and representation
Backbone width
1408
1280
1280
1408
Layers / heads
16 / 11
15 / 10
8 / 10
8 / 11
FFN width
3840
3584
3584
3840
Representation
Pixels
FLUX.2
SVG Autoencoder-P
Three tokenizers
Latent channels
—
128 (packed)
384
Discrete codes
Appendix
Tab. 4: Training recipes for the controlled generation comparisons. All routes match cumulative image exposure and use approximately 400M final parameters. Pretraining–post-training budgets include both stages; all AR tokenizer variants share the AR PDM recipe.
Sweep
Width
Layers
Heads
FFN
Parameters
Depth
512
4
4
1536
48.9M
512
8
4
1536
62.5M
512
12
4
1536
76.2M
512
24
4
1536
117.1M
512
32
4
1536
144.4M
Shared
512
16
4
1536
89.8M
Appendix
Tab. 5: Architectures used for AR hyperparameter transfer. Each row is trained at all 12 base learning rates. The reference architecture appears in both panels of Fig. 7 . Parameter counts include all trainable AR parameters, count the tied code table once, and exclude the frozen visual tokenizer and text encoder. FFN denotes feed-forward hidden dimension; each attention head has dimension 128. Parameter counts are rounded to one decimal place in millions (M).
Figure 7: Pretraining learning-rate transfer across width and depth. Each run trains for 50,000 updates. Each point averages 21 raw training cross-entropy measurements; crosses mark the minimum among the 12 tested base learning rates.
N (M)
Layers (depth)
Hidden dim. W
Heads
Head dim.
FFN dim. F
55.71
6
512
4
128
1,536
71.30
7
640
5
128
1,792
99.96
9
768
6
128
2,048
137.73
10
896
7
128
2,560
191.53
12
1,024
8
128
2,816
260.95
14
1,152
9
128
3,072
Appendix
Tab. 6: AR architectures used for the scaling fits. Models are sorted by trainable parameters N ; W and F denote hidden and FFN dimensions. Shared vocabulary weights are counted once; frozen encoders and tokenizers are excluded.
Method
N20 (M)
D20 (B)
a
b
Envelope
296.96
56.12
0.5268
0.4732
IsoFLOP
270.58
61.60
0.4705
0.5295
Parametric
199.05
83.73
0.4823
0.5177
Appendix
Tab. 7: Full-sequence AR allocation laws. N=N20(C/1020)a and D=D20(C/1020)b .
Figure 8: Post-training learning-rate search. Mean training loss at approximately 20,000 updates for four width/depth pairs. Each point averages 41 measurements; the largest model uses its final available windows. Crosses mark the minimum for each model.
Parameters (M)
Layers
Width
Heads
FFN width
25.13
6
512
4
1536
43.46
7
640
5
1792
77.23
9
768
6
2048
120.73
10
896
7
2560
184.00
12
1024
8
2816
265.85
14
1152
9
3072
Appendix
Tab. 8: Post-training model family, ordered by trainable diffusion parameter count. Attention heads have dimension 128. The AR backbone and external encoders remain frozen and are excluded from the parameter counts.
Method
N20 (M)
D20 (B)
a
b
Parametric
513.48
32.46
0.7021
0.2979
Envelope
395.28
42.16
0.5492
0.4508
IsoFLOP
377.99
44.09
0.4538
0.5462
Appendix
Tab. 9: Post-training allocation fits. With x=C/1020 , Nopt=N20xa and Dopt=D20xb , using full-sequence token counts.
Figure 9: Pixel-diffusion learning-rate transfer. Original width and depth search results after 50,000 training updates. Points average the final 10,000 updates; crosses mark the best tested learning rate.
Sweep
Width
Layers
Best base LR
Training loss
Width
256
16
8×10−4
0.1405
Width
384
16
8×10−4
0.1309
Width
512
16
10−3
0.1258
Width
640
16
10−3
0.1231
Width
768
16
10−3
0.1214
Width
1024
16
8×10−4
0.1195
Appendix
Tab. 10: Grid minima in the original PDM hyperparameter search. The shared width-512, depth-16 architecture is evaluated in both sweeps.
Parameters (M)
Layers
Width
Heads
FFN width
23.83
6
512
4
1536
69.97
9
768
6
2048
164.08
12
1024
8
2816
319.16
15
1280
10
3584
529.29
18
1536
12
4096
844.62
21
1792
14
4864
Appendix
Tab. 11: Pure PDM scaling architectures, ordered by trainable parameter count. All attention heads have dimension 128.
Method
N20 (M)
D20 (B)
a
b
Envelope
394.37
42.26
0.5744
0.4256
IsoFLOP
409.73
40.68
0.5161
0.4839
Parametric
535.04
31.15
0.7143
0.2857
Appendix
Tab. 12: Full-sequence PDM allocation laws. For x=C/1020 , Nopt=N20xa and Dopt=D20xb .
Figure 10: Pixel-diffusion scaling from scratch. Full-sequence compute accounting for eight model sizes. Top: training losses and envelope allocations. Bottom: six representative IsoFLOP profiles and allocation fits using all nine accepted minima. Stars mark the quadratic minima.
NAR (M)
Ndiff (M)
Cpre
Cpost
r
260.95
265.85
7.82×1019
4.86×1019
0.383
343.39
360.77
1.32×1020
8.47×1019
0.391
425.72
457.95
1.98×1020
1.31×1020
0.398
549.65
604.00
3.22×1020
2.16×1020
0.402
688.28
763.60
4.93×1020
3.32×1020
0.402
860.44
966.18
7.53×1020
5.09×1020
0.403
Appendix
Tab. 13: Induced pretraining–post-training budget allocation at the envelope compute-optimal. For each jointly scaled model, each stage is placed at its own envelope optimum and r=Cpost/(Cpre+Cpost) . N excludes frozen modules; C uses the full-sequence 6ND proxy. Models below 261 M are omitted as noisy.
Figure 11: Selected generation samples from our final text-to-image model.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
While diffusion models dominate the field of visual generation, they are computationally inefficient, applying a uniform computational effort regardless of different complexity. In contrast, autoregressive (AR) models are inherently complexity-aware, as evidenced by their variable likelihoods, but are often hindered by lossy discrete tokenization and error accumulation. In this work, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm that addresses these issues. At its core, GRN addresses the discrete tokenization bottleneck through a theoretically near-lossless Hierarchical Binary Quantization (HBQ), achieving a reconstruction quality comparable to continuous counterparts. Built upon HBQ's latent space, GRN fundamentally upgrades AR generation with a global refinement mechanism that progressively perfects and corrects artworks -- like a human artist painting. Besides, GRN integrates an entropy-guided sampling strategy, enabling complexity-aware, adaptive-step generation without compromising visual quality. On the ImageNet benchmark, GRN establishes new records in image reconstruction (0.56 rFID) and class-conditional image generation (1.81 gFID). We also scale GRN to more challenging text-to-image and text-to-video generation, delivering superior performance on an equivalent scale. We release all models and code to foster further research on GRN.
The space of possible visual content is enormous, and arguably unbounded. Any training set, however large, therefore covers only a tiny fraction of it. Yet generative models are typically trained to fit only the images they are given. In visual autoregressive (AR) generation, for example, each token is predicted from the preceding ground-truth (GT) tokens and supervised solely by the GT next token. Confining learning to observed data in this way limits how much of the visual distribution a model can capture. In this work, we hypothesize that letting a model imagine beyond its training data, by exploring plausible visual realizations it has never observed, can ease this limitation. We use visual AR generation as a proof of concept, since its fixed, discrete token space makes imagination easier to steer and constrain. We propose Next Token Imagination (NTI), a method that allows the model to explore beyond the GT next token, subject to a carefully designed criterion that keeps this imagination bounded by the underlying visual distribution. Extensive experiments on ImageNet show that NTI substantially and consistently improves the fidelity of generated images, approaching state-of-the-art diffusion and AR models while using a smaller model and lower training cost.
In this work, we propose Prologue, an approach to bridging the reconstruction-generation gap in autoregressive (AR) image generation. Instead of modifying visual tokens to satisfy both reconstruction and generation, Prologue generates a small set of prologue tokens prepended to the visual token sequence. These prologue tokens are trained exclusively with the AR cross-entropy (CE) loss, while visual tokens remain dedicated to reconstruction. This decoupled design lets us optimize generation through the AR model's true distribution without affecting reconstruction quality, which we further formalize from an ELBO perspective. On ImageNet 256x256, Prologue-Base reduces gFID from 21.01 to 10.75 without classifier-free guidance while keeping reconstruction almost unchanged; Prologue-Large reaches a competitive rFID of 0.99 and gFID of 1.46 using a standard AR model without auxiliary semantic supervision. Interestingly, driven only by AR gradients, prologue tokens exhibit emergent semantic structure: linear probing on 16 prologue tokens reaches 35.88% Top-1, far above the 23.71% of the first 16 tokens from a standard tokenizer; resampling with fixed prologue tokens preserves a similar high-level semantic layout. Our results suggest a new direction: generation quality can be improved by introducing a separate learned generative representation while leaving the original representation intact.
Bowen Zheng, Weijian Luo, Guang Yang +2
The Chinese University of Hong Kong, Shenzhen · hi-Lab, Xiaohongshu Inc