cs.CVNov 5, 2026

Scale as a Modality: Multi-Scale Pixel-Space Diffusion Transformers

Authors: Ruibo Chen, Kaishen Wang, Tianyi Xiong, Chenxi Liu, Yanshuo Chen, Heng Huang, Jian Du

Organizations: TikTok · University of Maryland, College Park

Abstract

Latent diffusion models rely on pretrained autoencoders that introduce lossy compression and constrain spatial granularity. Pixel-space diffusion avoids this bottleneck and enables flexible tokenization at multiple spatial scales, yet existing approaches largely adopt two-level coarse-to-fine architectures with limited cross-scale interaction. We introduce MS-pDiT, a multi-scale pixel-space diffusion transformer that treats spatial scale as a modality. MS-pDiT tokenizes the same noisy image at multiple patch sizes, processes each scale in a dedicated transformer stream, and enables bidirectional information exchange through compacted MM-DiT-style joint attention, with attention cost independent of the finest patch size. This design establishes the number of scales as a new architectural scaling dimension. On ImageNet 256×256256\times256, increasing the number of scales from two to five consistently improves FID from 4.39 to 2.31, 2.26, and 2.19, while adding an intermediate scale substantially outperforms increasing depth under comparable parameter budgets. Further analysis highlights the importance of bidirectional cross-scale interaction and late introduction of fine-grained streams. MS-pDiT achieves FID scores of 1.43 and 1.44 on ImageNet 256×256256\times256 and 512×512512\times512, respectively, and naturally extends to text-to-image generation, achieving 0.86 on GenEval and 83.9 on DPG-Bench.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PixelDiT2: Representation-Grounded Pixel Diffusion Transformers

    Sep 21, 2026Yongsheng Yu, Wei Xiong, Yichen Sheng +2Diffusion Model GuidanceDiffusion Transformer

  2. Pixel-Space Diffusion Transformers

    Jul 20, 2026Renye Yan, Jikang Cheng, You Wu +8Diffusion TransformerPixel-Space Diffusion Models