PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

Ruixiao Dong; Jianmin Bao; Bozhi Luan; Zijie Meng; Xin Tao; Pengfei Wan; Kun Gai; Li Li

cs.CVOct 13, 2026

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

Authors: Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao, Pengfei Wan, Kun Gai, Li Li,

Organizations: University of Science and Technology of China · KlingAI Research · Peking University

Abstract

Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE consistently outperforms its dense counterparts across model scales, demonstrating effective sparse capacity scaling in pixel space. On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256x256 and 1.53 at 512x512 without representation alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models. Code and models will be released soon.

Figures & tables

Figure 1: Progress in pixel-space image generation on ImageNet-1K at

256\times 256

. PixelMoE sparsely scales model capacity with MoE and compares favorably with leading approaches.

Figure 2: ImageNet-1K generation with PixelMoE. Left: Dense vs. PixelMoE after 100 training epochs, where PixelMoE consistently achieves lower FID at comparable active parameter counts. PixelMoE-B/L/H abbreviate PixelMoE-B/L/H-E32A4, respectively, with 32 routed experts and top-4 activation. Right: Comparison with representative pixel-space methods. Marker size denotes active parameter counts.

Figure 3: Overview of PixelMoE. Left: PixelMoE performs Pixel-Space Hierarchical Flow Matching using a shared PixelMoE backbone and DDT prediction head; the initial dense blocks are omitted for clarity. Right: Each PixelMoE block consists of self-attention followed by an LLM-style sparse MoE layer with Token-Choice routed experts and a shared expert. A balancing bias is employed to maintain balanced expert utilization. RMSNorm and timestep/class conditioning are omitted.

Figure 4: Pixel-Space Hierarchical Flow Matching in PixelMoE. PixelMoE is jointly trained on a low-resolution coarse stage and a full-resolution refinement stage with stage-wise endpoint prediction. During sampling, generation first follows the low-resolution Stage 0 flow to obtain a coarse endpoint, which is then transitioned to the Stage 1 starting state for full-resolution refinement.

Table 2: Sparse scaling analysis of PixelMoE. #PS denotes the patch size.

Figure 6: Expert routing across semantic classes and along the denoising trajectory. The left panel shows expert routing across ImageNet-1K classes, while the right panel shows how routing evolves along the denoising trajectory.

Figure 7: Global expert utilization across layers. The routed expert loads of PixelMoE-L-E32A4 remain close to a uniform distribution across the analyzed MoE layers. Expert loads are normalized by their expected values under uniform routing, with the dashed line indicating 1.

Figure 8: Qualitative results of PixelMoE. Samples at $256\times 256$ and $512\times 512$ resolutions.

Table 5: Ablation studies of the sparse MoE design in PixelMoE-L-E32A4.

Table 6: Ablation studies of routing and generation components in PixelMoE. Explicit routing guidance is evaluated on PixelMoE-L-E32A4, while the remaining ablations are conducted on PixelMoE-B-E32A4.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Depth	Hidden Dim.	Heads	DDT Depth	DDT Dim.	Routed Dim.	Shared Dim.	#Dense
PixelMoE-B-E32A4	12	768	12	2	1536	384	632	4
PixelMoE-L-E32A4	24	1024	16	2	2048	512	836	4
PixelMoE-H-E32A4	32	1280	16	4	2560	640	1048	4

Appendix

Table 7: PixelMoE model configurations. DDT Dim. represents the hidden state dimension in the DDT head. Routed Dim. and Shared Dim. denote the intermediate dimensions of the routed-expert and shared-expert feed-forward networks (FFNs), respectively. #Dense denotes the number of dense blocks in the PixelMoE backbone.

Input and training
	$256\times 256$			$512\times 512$
Configuration	Base	Large	Huge	Base	Large	Huge
Training resolution	$256^{2}$	$256^{2}$	$256^{2}$	$512^{2}$	$512^{2}$	$512^{2}$
Patch size	$8$	$8$	$8$	$16$	$16$	$16$
Noise scale	$1.0$	$1.0$	$1.0$	$2.0$	$2.0$	$2.0$
Crop range	–	–	$1.1^{\dagger}$	–	–	$1.1^{\dagger}$

Appendix

Table 8: Training configurations of PixelMoE. For

512\times 512

generation, we increase the patch size from

8

16

and the Gaussian noise scale from

1.0

2.0

. † For the final PixelMoE-H-E32A4 models, LlamaGen-style crop augmentation with a crop range of

1.1

and dropout with a rate of

0.2

are applied after epoch

200

; the

100

-epoch controlled experiments use the standard preprocessing without these additional regularization measures.

Configuration	Setting
Sampling solver	Heun
Global timestep grid	$50$
Global grid spacing	Uniform
Timestep shifting	None
CFG scale	Swept per model
CFG interval	Swept per model

Appendix

Table 9: Evaluation configurations. CFG scale, CFG interval, and EMA decay are jointly selected for each model according to FID-10K, and the selected configuration is fixed for the final FID-50K evaluation.

Panel of Figure 9 — Figure 9: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at $256\times 256$ . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.

Panel of Figure 10 — Figure 10: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at $256\times 256$ . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.

Panel of Figure 11 — Figure 11: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at $512\times 512$ . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 328 papers

An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models
Aug 17, 2026Dengyang Jiang, Ruoyi Du, Zhennan Chen +10Pixel-Space Diffusion Models T2I Diffusion Models
The Hong Kong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Nanjing University +1
MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
Aug 5, 2026Mohammadreza Hami, Mohammadreza Samadi, Chao Gao +1Pixel-Space Diffusion Models Diffusion Model Inference Acceleration
Huawei Technologies Canada
L2P: Unlocking Latent Potential for Pixel Generation
May 12, 2026Zhennan Chen, Junwei Zhu, Xu Chen +7Latent Diffusion Models Pixel-Space Diffusion Models
Nanjing University · Tencent Youtu Lab · Hainan-biuh +1

Aug 17, 2026J/K move · Enter open · S save

An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

Dengyang Jiang, Ruoyi Du, Zhennan Chen, Dongyang Liu+9

The Hong Kong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Nanjing University+1

Pixel-Space Diffusion Models T2I Diffusion Models Diffusion Models Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.

Abstract

Diffusion-Based Image Generation

Explore similar work

An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation

L2P: Unlocking Latent Potential for Pixel Generation