Scale as a Modality: Multi-Scale Pixel-Space Diffusion Transformers
Organizations: TikTok · University of Maryland, College Park
Abstract
Latent diffusion models rely on pretrained autoencoders that introduce lossy compression and constrain spatial granularity. Pixel-space diffusion avoids this bottleneck and enables flexible tokenization at multiple spatial scales, yet existing approaches largely adopt two-level coarse-to-fine architectures with limited cross-scale interaction. We introduce MS-pDiT, a multi-scale pixel-space diffusion transformer that treats spatial scale as a modality. MS-pDiT tokenizes the same noisy image at multiple patch sizes, processes each scale in a dedicated transformer stream, and enables bidirectional information exchange through compacted MM-DiT-style joint attention, with attention cost independent of the finest patch size. This design establishes the number of scales as a new architectural scaling dimension. On ImageNet , increasing the number of scales from two to five consistently improves FID from 4.39 to 2.31, 2.26, and 2.19, while adding an intermediate scale substantially outperforms increasing depth under comparable parameter budgets. Further analysis highlights the importance of bidirectional cross-scale interaction and late introduction of fine-grained streams. MS-pDiT achieves FID scores of 1.43 and 1.44 on ImageNet and , respectively, and naturally extends to text-to-image generation, achieving 0.86 on GenEval and 83.9 on DPG-Bench.
Figures & tables
| Method | Params | Epochs | NFE | FID | IS | sFID | Prec. | Rec. |
| Latent-space diffusion | ||||||||
| DiT-XL/2 [ 32 ] | 675+49M | 1400 | 2.27 | 278.2 | 4.60 | 0.83 | 0.57 | |
| SiT-XL/2 [ 28 ] | 675+49M | 1400 | 2.06 | 277.5 | 4.50 | 0.80 | 0.64 | |
| REPA-XL/2 [ 45 ] | 675+49M | 800 | 1.42 | 305.7 | 4.70 | 0.80 | 0.64 | |
| DDT-XL/2 [ 42 ] | 675+49M | 400 | – | 1.26 | 310.6 | – | 0.78 | 0.67 |
| RAE-XL/2 [ 48 ] | 839+415M | 800 | – | 1.13 | 262.6 | – | 0.78 | 0.67 |
| Model | Patch sizes | Depths | Stream widths | Heads | Params | |
|---|---|---|---|---|---|---|
| MS-pDiT-L | 1152 | 16 | 788M | |||
| MS-pDiT-XL | 1152 | 16 | 963M | |||
| MS-pDiT-H | 1152 | 16 | 1.4B | |||
| MS-pDiT-T2I | 1536 | 24 | 1.4B |
| GenEval | DPG | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Parameters | Single Object | Two Objects | Counting | Colors | Position | Color Attribution | Overall | Average |
| PixelFlow | 0.9B | – | – | – | – | – | – | 0.60 | 77.9 |
| PixNerd | 1.2B | 0.97 | 0.86 | 0.44 | 0.83 | 0.71 | 0.53 | 0.73 | 80.9 |
| PixelGen | 1.1B | 0.99 | 0.88 | 0.59 | 0.90 | 0.70 | 0.70 | 0.79 | – |
| PixelDiT | 1.3B | 1.00 | 0.94 | 0.70 | 0.90 | 0.53 | 0.65 | 0.78 | 83.7 |
| DeCo | 1.1B | 1.00 | 0.92 | 0.72 | 0.91 | 0.80 | 0.79 | 0.86 | 81.4 |
| Method | FID | sFID | IS | Prec. | Rec. |
|---|---|---|---|---|---|
| Latent-space diffusion | |||||
| DiT-XL [ 32 ] | 3.04 | 5.02 | 240.8 | 0.84 | 0.54 |
| SiT-XL [ 28 ] | 2.62 | 4.18 | 252.2 | 0.84 | 0.57 |
| MaskDiT [ 49 ] | 2.50 | 5.10 | 256.2 | 0.83 | 0.56 |
| U-ViT-H [ 2 ] | 4.05 | – | 263.8 | 0.84 | 0.48 |
| REPA [ 45 ] | 2.08 | 4.19 | 274.6 | 0.83 | 0.58 |
| Stream Depth | Training Settings | Quality | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 16 16 | 8 8 | 4 4 | 2 2 | 1 1 | Params | Epochs | FID | IS | sFID | Prec. | Rec. |
| 22 | - | - | - | 4 | 651M | 80 | 4.39 | 241.5 | 6.23 | 0.73 | 0.59 |
| 22 | - | 8 | - | 4 | 788M | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| 22 | - | 8 | 8 | 4 | 910M | 80 | 2.26 | 298.5 | 4.66 | 0.81 | 0.58 |
| 22 | 8 | 8 | - | 4 | 914M | 80 | 2.35 | 300.0 | 4.67 | 0.81 | 0.59 |
| 22 | 8 | 8 | 8 | 4 | 1.0B | 80 | 2.19 | 296.4 | 4.73 | 0.81 | 0.59 |
| Stream Depth | Training Settings | Quality | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 16 16 | 8 8 | 4 4 | 2 2 | 1 1 | Params | Epochs | FID | IS | sFID | Prec. | Rec. |
| 22 | - | 8 | - | 4 | 788M | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| 13 | - | 13 | - | 13 | 768M | 80 | 3.74 | 237.8 | 5.63 | 0.76 | 0.59 |
| 26 | - | - | - | 4 | 758M | 80 | 3.63 | 254.6 | 5.72 | 0.76 | 0.58 |
| 22 | - | - | 8 | 4 | 773M | 80 | 2.54 | 292.5 | 5.26 | 0.80 | 0.58 |
| 22 | 8 | - | - | 4 | 777M | 80 | 2.42 | 298.2 | 5.12 | 0.80 | 0.59 |
| Method | FID | IS | sFID | P | R |
|---|---|---|---|---|---|
| Last Layers (Ours) | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| + Scale Causal Mask | 3.47 | 262.2 | 5.58 | 0.76 | 0.58 |
| + Strict Scale Causal Mask | 2.55 | 299.4 | 5.18 | 0.79 | 0.59 |
| Middle Layers | 2.31 | 277.4 | 4.99 | 0.80 | 0.59 |
| + Scale Causal Mask | 2.50 | 282.6 | 5.10 | 0.80 | 0.58 |
| + Strict Scale Causal Mask | 3.24 | 265.2 | 5.70 | 0.78 | 0.58 |
| Stage | Res. | Data | Steps | GPUs | Global batch | LR | Shift |
|---|---|---|---|---|---|---|---|
| 1. Pretrain | BLIP3o | 200K | 24 | 1536 | 2.0 | ||
| 2. High-res. | BLIP3o ( 512px) | 80K | 32 | 512 | 4.0 | ||
| 3. SFT | BLIP3o-60k | 40K | 32 | 512 | 4.0 |
| Attention Patch Size | Epochs | FID | IS | sFID | Prec. | Rec. |
|---|---|---|---|---|---|---|
| 16 (ours) | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| 32 | 80 | 3.00 | 252.6 | 5.53 | 0.77 | 0.60 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Epochs | FID | IS | sFID | Prec. | Rec. |
|---|---|---|---|---|---|---|
| MS-pDiT | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| w/o Scale Identity Embedding | 80 | 3.29 | 265.2 | 6.02 | 0.76 | 0.59 |
| Method | Epochs | FID | IS | sFID | Prec. | Rec. |
|---|---|---|---|---|---|---|
| MS-pDiT | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| + | 80 | 2.40 | 300.0 | 4.95 | 0.81 | 0.58 |
| Method | Epochs | FID | IS | sFID | Prec. | Rec. |
|---|---|---|---|---|---|---|
| Largest Patch (Ours) | 80 | 2.31 | 302.1 | 4.77 | 0.81 | 0.58 |
| No REPA | 80 | 5.30 | 195.3 | 5.63 | 0.83 | 0.47 |
| Registers | 80 | 2.56 | 312.5 | 4.43 | 0.83 | 0.56 |
| Method | Params | GFLOPs |
|---|---|---|
| ADM-G [ 8 ] | 554M | 1120 |
| RIN [ 19 ] | 410M | 334 |
| VDM++ [ 20 ] | 2B | 555 |
| PixelFlow-XL/4 [ 4 ] | 677M | 2909 |
| PixNerd-XL/16 [ 41 ] | 700M | 134 |
| JiT-G/16 [ 22 ] | 2B | 383 |
| # | Prompt |
|---|---|
| 1 | a painting of a fox in the style of starry night. |
| 2 | a close-up of the eyes of an owl. |
| 3 | A fluffy baby sloth with a knitted hat trying to figure out a laptop, close up, highly detailed, studio lighting, screen reflecting in its eyes. |
| 4 | an old man with a long grey beard and green eyes. |
| 5 | A photo of a light bulb in outer space traveling the galaxy with a sailing boat inside the light bulb. |
| 6 | a sport car melting into a clock, surrealist painting in the style of Salvador Dali. |
| # | Prompt |
|---|---|
| 1 | A smiling sloth is wearing a leather jacket, a cowboy hat, a kilt and a bowtie. The sloth is holding a quarterstaff and a big book. The sloth is standing on grass a few feet in front of a shiny VW van with flowers painted on it. wide-angle lens from below. |
| 2 | a painting of street in Paris |
| 3 | A teddy bear wearing a motorcycle helmet and cape is standing in front of Loch Awe with Kilchurn Castle behind him |
| 4 | a portrait of young girl |
| 5 | A punk rock squirrel in a studded leather jacket shouting into a microphone while standing on a stump and holding a beer on dark stage. |
| 6 | Snow mountain and tree reflection in the lake |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,266 papers