An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

Dengyang Jiang; Ruoyi Du; Zhennan Chen; Dongyang Liu; Zanyi Wang; Mingzhe Zheng; Xiangpeng Yang; Huanqia Cai

cs.CVAug 17, 2026

An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

Authors: Dengyang Jiang, Ruoyi Du, Zhennan Chen, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Xiangpeng Yang, Huanqia Cai,

Organizations: The Hong Kong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Nanjing University · University of California, San Diego

Abstract

This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.

Figures & tables

Figure 1: Images generated by Z-Image-Turbo (Pixel) trained through our method. Offering only $\sim$ 0.2 second inference latency on an enterprise-grade H800 GPU.

Figure 2: Evaluation curves over training steps. The model trained in the latent space consistently achieve superior results (measured by GenEval [ 27 ] and DPG [ 34 ] ) and faster convergence compared to those trained in the pixel space.

Figure 3: Qualitative samples over training iterations. The latent space model consistently demonstrates faster image structure learning and superior final image quality in pre-training progress.

Figure 4: Pixel-space training that initialized from a pre-trained latent-space model yields remarkable faster convergence than training from scratch.

Figure 5: Pixel-Space trained with latent-weight initialization enables earlier formation of coherent image structures and improves final image quality compared with training from scratch.

Figure 6: Effect of training data source. Source-latent-model-generated data enables substantially faster convergence than real data alone, while mixing real and self-generated data achieves the best overall trade-off.

Figure 7: Qualitative comparison of training data choices. Training only on source latent-model generations can retain artifacts inherited from the source model. Mixing high-quality real images provides direct pixel-level supervision and improves these details.

Figure 8: Training with $x$ -prediction achieves overall better performance and faster convergence than $v$ -prediction.

Panel of Table 1 — Table 1: Decoder design comparison. Inference latency and GFLOPs are measured with decoder-only Inference for obtaining a $1024\times 1024$ image on a single H800 GPU. The best and second-best results in each row are bolded and underlined , respectively.

Figure 9: The JiT head exhibits visible grid-like artifacts at patch boundaries, whereas the DiP head produces smoother spatial transitions.

Panel of Table 2 — Table 2: Ablation on noise scale $\gamma$ .

Figure 10: Qualitative comparison of different noise scales. Noise scales of $1$ , $4$ , and $8$ exhibit noticeable color shifts, whereas that of $2$ produces more balanced color rendition.

Figure 11: Progressive patch-size adaptation improves convergence and final performance.

Model	NFE	Latency (s) $\downarrow$	GenEval $\uparrow$	DPG $\uparrow$	OneIG $\uparrow$	LongText $\uparrow$
Z-Image (latent-space)	100	20.12	0.7510	86.91	0.566	0.9332
L2P (pixel-space)	100	18.26	0.7612	86.00	0.499	0.7798
Ours (pixel-space)	100	4.56 (4.41 $\times$ )	0.7644	87.60	0.564	0.9252
Z-Image-Turbo (latent-space)	4	0.95	0.7625	85.31	0.520	0.8639
Ours-Turbo (pixel-space)	4	0.20 (4.75 $\times$ )	0.7698	86.85	0.512	0.8732
FLUX2-klein-Base (latent-space)	100	20.25	0.8027	84.89	0.541	0.5456

Table 3: Performance comparison of pixel and latent models across benchmarks [ 27 , 34 , 6 , 26 ] . Images are generated using original benchmark prompts rather than Prompt-Enhanced (PE) [ 71 ] variants. Inference latency is measured for a

1024\times 1024

image on a single H800 GPU without any system-level optimizations.

Figure 12: Qualitative comparison of patch-size adaptation. Direct ps32 training introduces local artifacts. Adapting from ps16 to ps32 preserves these details, whereas further adaptation to ps64 reintroduces artifacts.

Figure 13: End-to-end inference latency (S) per $1024\times 1024$ image on a single NVIDIA H100 GPU without any system-level optimizations such as FlashAttention [ 18 ] .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Section 4.1	Section 4.2	Section 4.3
Weight initialization	From scratch / Latent-pretrained	Latent-pretrained	Latent-pretrained
Training data	Self-generated + Real	Real images / Self-generated samples / Other-model samples / Self-generated + Real	Self-generated + Real
Prediction target	$x$ -prediction	$x$ -prediction	$v$ -prediction / $x$ -prediction
Decoder architecture	DiP	DiP	DiP
Noise scale	$\gamma=2$	$\gamma=2$	$\gamma=2$
Patch size	ps16	ps16	ps16

Appendix

Table 4: Hyperparameter settings for component-wise analysis (I). Bold entries denote the configurations varied in the corresponding section.

	Section 4.4	Section 4.5	Section 5.1
Weight initialization	Latent-pretrained	Latent-pretrained	Latent-pretrained
Training data	Self-generated + Real	Self-generated + Real	Self-generated + Real
Prediction target	$x$ -prediction	$x$ -prediction	$x$ -prediction
Decoder architecture	JiT / DiP / PiT / Deco / VAE	DiP	DiP
Noise scale	$\gamma=2$	$\gamma\in\{1,2,4,8\}$	$\gamma=2$
Patch size	ps16	ps16	ps16 / ps32 / ps32-adapt16 / ps64-adapt32

Appendix

Table 5: Hyperparameter settings for component-wise analysis (II). Bold entries denote the configurations varied in the corresponding section.

Pixel-Space Diffusion Models

Pixel-space diffusion models generate or restore images by learning to denoise raw pixels rather than compressed latent codes.

See all 56 papers

L2P: Unlocking Latent Potential for Pixel Generation
May 12, 2026Zhennan Chen, Junwei Zhu, Xu Chen +7Latent Diffusion Models Pixel-Space Diffusion Models
Nanjing University · Tencent Youtu Lab · Hainan-biuh +1
Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning
Oct 7, 2026Hanqiu Li Cai, Chema GarabitoT2I Diffusion Models Pixel-Space Diffusion Models
SperidLabs
PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
Oct 13, 2026Ruixiao Dong, Jianmin Bao, Bozhi Luan +6Diffusion-Based Image Generation Pixel-Space Diffusion Models
University of Science and Technology of China · KlingAI Research · Peking University

May 12, 2026J/K move · Enter open · S save

L2P: Unlocking Latent Potential for Pixel Generation

Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang+6

Nanjing University · Tencent Youtu Lab · Hainan-biuh+1

Latent Diffusion Models Pixel-Space Diffusion Models Diffusion-Based Image Generation Diffusion Models

Open paper AskPDF arXiv

Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.

Abstract

Pixel-Space Diffusion Models

Explore similar work

L2P: Unlocking Latent Potential for Pixel Generation

Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts