cs.CVOct 16, 2026

Rethinking the design space of Vanilla Pixel Diffusion Transformers

Authors: Soumik Mukhopadhyay, Long Mai, Abhinav Shrivastava, Aniruddha Mahapatra

Organizations: University of Maryland, College Park · Adobe Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pixel-space diffusion models have recently re-emerged as a compelling alternative to latent diffusion, avoiding the lossy latent bottleneck and separately trained autoencoder. JiT shows that a vanilla diffusion transformer can generate directly in pixel space by replacing the commonly used velocity prediction (u\boldsymbol{u}-prediction) with clean image prediction (x0\boldsymbol{x_0}-prediction). In this work, we revisit this choice and introduce MFlow, an adaptive parameterization that learns to interpolate between x0\boldsymbol{x_0}- and u\boldsymbol{u}-prediction across principal directions of the data, learned end-to-end with diffusion training. We find that the preferred parameterization varies systematically with the data representation and model capacity -- notably, increasing model capacity shifts preference toward velocity prediction. We evaluate MFlow across ImageNet, UCF-101 video, and large-scale text-to-image generation, demonstrating competitive or improved generative performance across diverse settings. In addition, we find that vanilla pixel diffusion transformers, irrespective of parameterization, retain a persistent high-frequency deficit relative to real images. To address this, we introduce ConvRefiner, a lightweight convolutional module trained post-hoc with an adversarial loss while keeping the diffusion backbone frozen. With only a small amount of additional training, ConvRefiner restores the high-frequency statistics of generated images while preserving FID. Project webpage: https://soumik-kanad.github.io/mflow-diffusion/.

Figures & tables

Figure 1: Sample 512px text-to-image generations with our MFlow -XXL/16 model.
Figure 2: Additional Text-to-Image Results. Sample 512px text-to-image generations with our MFLOW-XXL/16 model.
Figure 3: Overview of MFlow Formulation. We propose a principal data direction-wise learnable parameterization as a generalized framework in contrast with previous fixed parameterizations.
Figure 4: Toy manifold experiment. A 2-D spiral is embedded in D∈{2,8,16,512}D\in\{2,8,16,512\} dimensions and projected back for visualization. Full u{\bm{u}} -prediction degrades with DD , while x0{\bm{x}}_{0} -prediction, rank-matched AsymFlow, and fixed or learned MFlow recover the spiral, including in the undercomplete D=512D=512 setting.
Figure 7: Overview of ConvRefiner Formulation. We propose a residual convolutional refiner post hoc training for a few epochs, with a fixed backbone, to regain the high-frequency similar to real images spectrum in vanilla vision transformers.
MethodFID↓HF-deficit (dB)
JiT-H/160 1.82-6.69
MFlow -H/16 MFlow0 1.81-6.63
MFlow -H/16 + ConvRefiner (ours)0 1.81-1.29
Table 1: Summary of Results. Our MFlow + ConvRefiner provides a flexible parameterization, saving expensive hyperparameter sweeps while having better FID, as well as solving the high-frequency deficit (measured in mean radial power deficit compared to real images) on ImageNet-1K 256 ×\times 256.
MethodFID↓
w/o LPIPSw/ LPIPS
JiT-B/16 ( Li and He (2026) )0 3.703.42
AsymFlow-B/16 ( Chen et al. (2026a) )0 3.593.40
MFlow -B/16 (ours)0 3.743.33
Table 2: Effect of LPIPS on ImageNet 256×256 generation. We see that LPIPS helps all the models in FID (evaluated on base JiT-B evaluation hyperparameters). However, it improves MFlow performance the most, beating the baselines.
Table 8
Figure 8: (Left) ConvRefiner recovers high frequency across JiT/ MFlow and model sizes B/H. (Right) The generative performance remains almost the same in this post-training.
Figure 9: Visualizing effect of ConvRefiner for text-to-image generation. Here we visualize the effect of adding our post-training to MFlow -XXL on text-to-image generations. (left) We see improved details in the leaf veins in the generated image. (right) We also note the average power gain in the high-frequency (0.5–0.7) region. We see overall improvement in texture details comparable to real image spectrum.
Table 11
Figure 11: (Left) Increasing blocks of Conv-refiner and the weight of GAN loss helps recover high frequency. (Right) Other baselines refining does not help except DiP patch detailer( Chen et al. (2026b) ). This probably reflects a convolutions bias.
Figure 12: Learned M{\bm{M}} values. Variations in the MFlow gain curves across multiple axes: objectives – w/ vs w/o LPIPS, domains – images vs videos, representations – pixel vs latent x0{\bm{x}}_{0} -prediction models, model capacities – 1B vs 13B-MoE.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

 Image (ImageNet-256)Video (UCF-101, 16×128216{\times}128^{2} )Text-to-Image
Architecture
BackboneJiT-B/H (JiT Tab. 9)VideoJiT-B/L (= JiT depth/width)MMDiT
Params131M / 953M131M / 458M1.1B / 13B-MoE 2.9B act
Patch16×1616\times 164×8×84{\times}8{\times}816×1616\times 16 (PixelVAE)
Sequence length256 ( +32+32 in-context)1024res-dep. ( 10241024 @ 512512 px)
Bottleneck128 (B), 256 (H)256– (none)
Appendix
Table 7: Configuration for our JiT / AsymFlow / MFlow models. Architecture and optimizer follow JiT-B/L/H ( Li and He (2026) Tab. 9) for image/video; the text-to-image (T2I) stack is a PixelVAE-latent MMDiT. We list the settings we update.
TableModelSolverStepsCFGσmin⁡\sigma_{\min}CFG Interval
Table 5 (ImageNet, H, tuned)JiT-H ++ LPIPSHeun502.500.04[0.10,1.0][0.10,1.0]
AsymFlow-H ++ LPIPSHeun502.200.04[0.10,1.0][0.10,1.0]
MFlow -H ++ LPIPSHeun502.350.04[0.10,1.0][0.10,1.0]
Table 5 (UCF, B, tuned)Video JiT-BHeun1003.010−510^{-5}[0,1.0][0,1.0]
Video AsymFlow-BHeun1003.50.04[0,1.0][0,1.0]
 Video MFlow -BHeun1003.50.04[0,1.0][0,1.0]
Appendix
Table 8: Evaluation / sampling hyperparameters for our trained models (Tables 2 , 5 , 5 , 5 ). Table 1 uses a single shared base JiT-B configuration while the rest are tuned per model.
Figure 13: Leading modes of the uncentered patch-PCA basis. We reshape the first 64 eigenvectors of the 16×16×316\times 16\times 3 patch-PCA basis into RGB patches; the value λ\lambda denotes the variance of each direction. The leading ordering interleaves patch-global or very low-frequency modes with spatially varying modes. In particular, PC0 is dominated by low-frequency patch-level tone, PC1 is an approximately uniform chromatic mode, PC2–PC3 encode spatial gradients, and PC4 returns to an approximately uniform chromatic mode. Eigenvector signs are arbitrary, so the interpretation is based on spatial structure rather than the displayed color polarity.
Table 17
Figure 14: ImageNet Loss Curves at H-level. We usually observe that the u{\bm{u}} -loss of MFlow training is very slightly better.
Figure 15: Faster convergence on ImageNet with LPIPS at B-level. This corresponds to the results noted in Table 2 .
Figure 16: Qualitative Text-to-Image Results on DPG-Bench. Images generated by our method and baselines on DPG-Bench prompts.
Figure 17: Qualitative Text-to-Image Results on PRISM-Bench. Images generated by our method and baselines on PRISM-Bench prompts.
Figure 18: Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.
Figure 19: More Class-wise Uncurated Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.
Figure 20: More Class-wise Uncurated Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.
Figure 21: Qualitative UCF-101 Results. Frames generated by MFlow +post-hoc FW-LPIPS L-level model.
Figure 22: Visualizing effect of ConvRefiner . Here we visualize the effect of adding our post-training to MFlow -H+LPIPS on ImageNet generations. We also note the average power gain in the high-frequency(0.5–0.7) region. We see overall improvement in texture details.
Figure 23: Qualitative Ablation for ConvRefiner . Here we visualize the effect of adding our post-training to JiT-B on ImageNet generations. We also note the average power gain in the high-frequency(0.5–0.7) region. We also show the visual improvements – we see that b4 adds additional texture details, while b8 sometimes overshoots in texture.
More in this topic

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 312 papers

Explore similar work

CardsList
Jul 20, 2026

Pixel-Space Diffusion Transformers

Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.

Renye Yan, Jikang Cheng, You Wu +8

Peking University, China · Nanjing University, China · Stanford University, USA +3

Diffusion TransformerPixel-Space Diffusion ModelsDiffusion-Based Image GenerationMultimodal Diffusion Models
AskPDF
Jun 26, 2026

PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion

End-to-end pixel-space diffusion models bypass the lossy compression of Latent Diffusion Models (LDMs) but struggle to jointly model low-frequency semantics and high-frequency signals in high-dimensional space. Existing works heavily rely on complex pixel decoders to alleviate this issue. In this paper, we challenge this trend by revealing that these decoders primarily compensate for the optimization difficulties inherent to velocity prediction (vv-prediction). Under the clean data paradigm (xx-prediction), they are redundant. Motivated by this insight, we advocate for simplicity over complexity and introduce PixelU, a minimalist, single-stage U-shaped Diffusion Transformer tailored for pixel space. PixelU abandons auxiliary decoders in favor of zero-cost skip connections, which provide an "information highway" that directly routes uncorrupted high-frequency spatial details from shallow to deep layers. To further enable the backbone to focus exclusively on modeling low-frequency semantics, we introduce a constant-channel spatial down-sampling mechanism as a natural low-pass filter, which compresses deep features into a compact, low-frequency semantic manifold. Extensive experiments demonstrate that this decoupling of frequencies could outperform the strong baseline (JiT-G) with only about 1/3 of its computation cost. On ImageNet 256×\times256 and 512×\times512, PixelU achieves FID of 1.63 and 1.92 respectively, surpassing recent pixel-space methods and establishing a simple yet powerful new paradigm for end-to-end diffusion models.

Zipeng Guo, Lichen Ma, Yu He +4

Xi’an Jiaotong University

Diffusion TransformerPixel-Space Diffusion ModelsDiffusion-Based Image GenerationDiffusion Model AccelerationDiffusion Models
AskPDF
Sep 21, 2026

PixelDiT2: Representation-Grounded Pixel Diffusion Transformers

Recent advances in pixel-space diffusion models have narrowed the image quality gap with latent-space diffusion, but still converge more slowly and lag behind in final image quality. We argue that a key reason is the lack of an explicit representation prior: unlike latent diffusion, which usually denoises in a compact and structured latent space, pixel diffusion needs to learn denoising-friendly representations and pixel generation simultaneously from raw RGB space. To address this problem, we propose PixelDiT2, an end-to-end pixel-space diffusion model designed to decouple representation learning from pixel generation without introducing an autoencoder or latent reconstruction bottleneck. We propose representation grounding that uses a frozen pretrained vision foundation model to provide explicit per-patch representation guidance throughout denoising, allowing the pixel diffusion transformer to focus more on pixel generation. On ImageNet-256x256, PixelDiT2 achieves an FID of 1.46 after 600 epochs; at 512x512 resolution, PixelDiT2 achieves an FID of 1.48 after 680 epochs. Project page: https://pixeldit.github.io/pixeldit2/

Yongsheng Yu, Wei Xiong, Yichen Sheng +2

NVIDIA · University of Rochester

Diffusion Model GuidanceDiffusion TransformerPixel-Space Diffusion ModelsDiffusion-Based Image GenerationDiffusion-Based Denoising
AskPDF