Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE consistently outperforms its dense counterparts across model scales, demonstrating effective sparse capacity scaling in pixel space. On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256x256 and 1.53 at 512x512 without representation alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models. Code and models will be released soon.
Figures & tables
Figure 1: Progress in pixel-space image generation on ImageNet-1K at 256×256 . PixelMoE sparsely scales model capacity with MoE and compares favorably with leading approaches.
Figure 2: ImageNet-1K generation with PixelMoE. Left: Dense vs. PixelMoE after 100 training epochs, where PixelMoE consistently achieves lower FID at comparable active parameter counts. PixelMoE-B/L/H abbreviate PixelMoE-B/L/H-E32A4, respectively, with 32 routed experts and top-4 activation. Right: Comparison with representative pixel-space methods. Marker size denotes active parameter counts.
Figure 3: Overview of PixelMoE. Left: PixelMoE performs Pixel-Space Hierarchical Flow Matching using a shared PixelMoE backbone and DDT prediction head; the initial dense blocks are omitted for clarity. Right: Each PixelMoE block consists of self-attention followed by an LLM-style sparse MoE layer with Token-Choice routed experts and a shared expert. A balancing bias is employed to maintain balanced expert utilization. RMSNorm and timestep/class conditioning are omitted.
Figure 4: Pixel-Space Hierarchical Flow Matching in PixelMoE. PixelMoE is jointly trained on a low-resolution coarse stage and a full-resolution refinement stage with stage-wise endpoint prediction. During sampling, generation first follows the low-resolution Stage 0 flow to obtain a coarse endpoint, which is then transitioned to the Stage 1 starting state for full-resolution refinement.
Figure 5
Table 2: Sparse scaling analysis of PixelMoE. #PS denotes the patch size.
Figure 6: Expert routing across semantic classes and along the denoising trajectory. The left panel shows expert routing across ImageNet-1K classes, while the right panel shows how routing evolves along the denoising trajectory.
Figure 7: Global expert utilization across layers. The routed expert loads of PixelMoE-L-E32A4 remain close to a uniform distribution across the analyzed MoE layers. Expert loads are normalized by their expected values under uniform routing, with the dashed line indicating 1.
Figure 8: Qualitative results of PixelMoE. Samples at 256×256 and 512×512 resolutions.
Table 5: Ablation studies of the sparse MoE design in PixelMoE-L-E32A4.
Table 6: Ablation studies of routing and generation components in PixelMoE. Explicit routing guidance is evaluated on PixelMoE-L-E32A4, while the remaining ablations are conducted on PixelMoE-B-E32A4.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Depth
Hidden Dim.
Heads
DDT Depth
DDT Dim.
Routed Dim.
Shared Dim.
#Dense
PixelMoE-B-E32A4
12
768
12
2
1536
384
632
4
PixelMoE-L-E32A4
24
1024
16
2
2048
512
836
4
PixelMoE-H-E32A4
32
1280
16
4
2560
640
1048
4
Appendix
Table 7: PixelMoE model configurations. DDT Dim. represents the hidden state dimension in the DDT head. Routed Dim. and Shared Dim. denote the intermediate dimensions of the routed-expert and shared-expert feed-forward networks (FFNs), respectively. #Dense denotes the number of dense blocks in the PixelMoE backbone.
256×256
512×512
Configuration
Base
Large
Huge
Base
Large
Huge
Input and training
Training resolution
2562
2562
2562
5122
5122
5122
Patch size
8
8
8
16
16
16
Noise scale
1.0
1.0
1.0
2.0
2.0
2.0
Crop range
–
–
1.1†
–
–
1.1†
Appendix
Table 8: Training configurations of PixelMoE. For 512×512 generation, we increase the patch size from 8 to 16 and the Gaussian noise scale from 1.0 to 2.0 . † For the final PixelMoE-H-E32A4 models, LlamaGen-style crop augmentation with a crop range of 1.1 and dropout with a rate of 0.2 are applied after epoch 200 ; the 100 -epoch controlled experiments use the standard preprocessing without these additional regularization measures.
Configuration
Setting
Sampling solver
Heun
Global timestep grid
50
Global grid spacing
Uniform
Timestep shifting
None
CFG scale
Swept per model
CFG interval
Swept per model
Appendix
Table 9: Evaluation configurations. CFG scale, CFG interval, and EMA decay are jointly selected for each model according to FID-10K, and the selected configuration is fixed for the final FID-50K evaluation.
Figure 9: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at 256×256 . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.
Figure 10: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at 256×256 . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.
Figure 11: Uncurated samples of PixelMoE-H-E32A4 on ImageNet-1K at 512×512 . Samples are generated conditioned on the specified classes using the same sampling configuration as the reported quantitative results.
This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.
Dengyang Jiang, Ruoyi Du, Zhennan Chen +10
The Hong Kong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Nanjing University +1
Pixel-space diffusion models avoid the reconstruction ceiling of latent diffusion models by generating directly in image space. However, their substantially higher token count makes generation expensive due to the quadratic complexity of self-attention. Several existing efficiency methods reduce this cost by using larger patches at selected denoising steps, thereby representing the image with fewer tokens. Yet, each step still uses a single patch size uniformly across the entire image, overlooking that different regions suffer different fidelity losses when coarsened. We introduce MOSAIK, a damage-guided framework that varies patch size across regions and denoising steps. MOSAIK adapts the PixelDiT backbone to generate arbitrary heterogeneous patch layouts, and a lightweight predictor uses intermediate denoising features to estimate the fidelity loss caused by coarsening each region. Given a token budget, our damage-guided layout predictor assigns fine patches to sensitive regions and coarse patches elsewhere. Remarkably, while reducing FLOPs by 70% and token count by 83%, MOSAIK matches the full-compute PixelDiT on GenEval and its DPG-Bench score drops by only 1.0 point. Compared to diverse efficiency paradigms, including temporal patch scheduling and feature caching, our approach delivers highly competitive performance at moderate budgets and consistently outperforms these baselines in highly constrained compute regimes.
Pixel diffusion models have recently regained attention for visual generation. However, training advanced pixel-space models from scratch demands prohibitive computational and data resources. To address this, we propose the Latent-to-Pixel (L2P) transfer paradigm, an efficient framework that directly harnesses the rich knowledge of pre-trained LDMs to build powerful pixel-space models. Specifically, L2P discards the VAE in favor of large-patch tokenization and freezes the source LDM's intermediate layers, exclusively training shallow layers to learn the latent-to-pixel transformation. By utilizing LDM-generated synthetic images as the sole training corpus, L2P fits an already smooth data manifold, enabling rapid convergence with zero real-data collection. This strategy allows L2P to seamlessly migrate massive latent priors to the pixel space using only 8 GPUs. Furthermore, eliminating the VAE memory bottleneck unlocks native 4K ultra-high resolution generation. Extensive experiments across mainstream LDM architectures show that L2P incurs negligible training overhead, yet performs on par with the source LDM on DPG-Bench and reaches 93% performance on GenEval.
Zhennan Chen, Junwei Zhu, Xu Chen +7
Nanjing University · Tencent Youtu Lab · Hainan-biuh +1