Rethinking the design space of Vanilla Pixel Diffusion Transformers

Soumik Mukhopadhyay; Long Mai; Abhinav Shrivastava; Aniruddha Mahapatra

cs.CVOct 16, 2026

Rethinking the design space of Vanilla Pixel Diffusion Transformers

Authors: Soumik Mukhopadhyay, Long Mai, Abhinav Shrivastava, Aniruddha Mahapatra

Organizations: University of Maryland, College Park · Adobe Research

Abstract

Pixel-space diffusion models have recently re-emerged as a compelling alternative to latent diffusion, avoiding the lossy latent bottleneck and separately trained autoencoder. JiT shows that a vanilla diffusion transformer can generate directly in pixel space by replacing the commonly used velocity prediction ( $\boldsymbol{u}$ -prediction) with clean image prediction ( $\boldsymbol{x_0}$ -prediction). In this work, we revisit this choice and introduce MFlow, an adaptive parameterization that learns to interpolate between $\boldsymbol{x_0}$ - and $\boldsymbol{u}$ -prediction across principal directions of the data, learned end-to-end with diffusion training. We find that the preferred parameterization varies systematically with the data representation and model capacity -- notably, increasing model capacity shifts preference toward velocity prediction. We evaluate MFlow across ImageNet, UCF-101 video, and large-scale text-to-image generation, demonstrating competitive or improved generative performance across diverse settings. In addition, we find that vanilla pixel diffusion transformers, irrespective of parameterization, retain a persistent high-frequency deficit relative to real images. To address this, we introduce ConvRefiner, a lightweight convolutional module trained post-hoc with an adversarial loss while keeping the diffusion backbone frozen. With only a small amount of additional training, ConvRefiner restores the high-frequency statistics of generated images while preserving FID. Project webpage: https://soumik-kanad.github.io/mflow-diffusion/.

Figures & tables

Figure 1: Sample 512px text-to-image generations with our MFlow -XXL/16 model.

Figure 2: Additional Text-to-Image Results. Sample 512px text-to-image generations with our MFLOW-XXL/16 model.

Figure 3: Overview of MFlow Formulation. We propose a principal data direction-wise learnable parameterization as a generalized framework in contrast with previous fixed parameterizations.

Figure 4: Toy manifold experiment. A 2-D spiral is embedded in $D\in\{2,8,16,512\}$ dimensions and projected back for visualization. Full ${\bm{u}}$ -prediction degrades with $D$ , while ${\bm{x}}_{0}$ -prediction, rank-matched AsymFlow, and fixed or learned MFlow recover the spiral, including in the undercomplete $D=512$ setting.

Panel of Figure 7 — Figure 7: Overview of ConvRefiner Formulation. We propose a residual convolutional refiner post hoc training for a few epochs, with a fixed backbone, to regain the high-frequency similar to real images spectrum in vanilla vision transformers.

Method	FID↓	HF-deficit (dB)
JiT-H/16	0 1.82	-6.69
MFlow -H/16 MFlow	0 1.81	-6.63
MFlow -H/16 + ConvRefiner (ours)	0 1.81	-1.29

Table 1: Summary of Results. Our MFlow + ConvRefiner provides a flexible parameterization, saving expensive hyperparameter sweeps while having better FID, as well as solving the high-frequency deficit (measured in mean radial power deficit compared to real images) on ImageNet-1K 256

\times

256.

Method	FID↓
Method	w/o LPIPS	w/ LPIPS
JiT-B/16 ( Li and He (2026) )	0 3.70	3.42
AsymFlow-B/16 ( Chen et al. (2026a) )	0 3.59	3.40
MFlow -B/16 (ours)	0 3.74	3.33

Table 2: Effect of LPIPS on ImageNet 256×256 generation. We see that LPIPS helps all the models in FID (evaluated on base JiT-B evaluation hyperparameters). However, it improves MFlow performance the most, beating the baselines.

Table 8

Panel of Figure 8 — Figure 8: (Left) ConvRefiner recovers high frequency across JiT/ MFlow and model sizes B/H. (Right) The generative performance remains almost the same in this post-training.

Figure 9: Visualizing effect of ConvRefiner for text-to-image generation. Here we visualize the effect of adding our post-training to MFlow -XXL on text-to-image generations. (left) We see improved details in the leaf veins in the generated image. (right) We also note the average power gain in the high-frequency (0.5–0.7) region. We see overall improvement in texture details comparable to real image spectrum.

Figure 11: (Left) Increasing blocks of Conv-refiner and the weight of GAN loss helps recover high frequency. (Right) Other baselines refining does not help except DiP patch detailer( Chen et al. (2026b) ). This probably reflects a convolutions bias.

Figure 12: Learned ${\bm{M}}$ values. Variations in the MFlow gain curves across multiple axes: objectives – w/ vs w/o LPIPS, domains – images vs videos, representations – pixel vs latent ${\bm{x}}_{0}$ -prediction models, model capacities – 1B vs 13B-MoE.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Image (ImageNet-256)	Video (UCF-101, $16{\times}128^{2}$ )	Text-to-Image
Architecture
Backbone	JiT-B/H (JiT Tab. 9)	VideoJiT-B/L (= JiT depth/width)	MMDiT
Params	131M / 953M	131M / 458M	1.1B / 13B-MoE 2.9B act
Patch	$16\times 16$	$4{\times}8{\times}8$	$16\times 16$ (PixelVAE)
Sequence length	256 ( $+32$ in-context)	1024	res-dep. ( $1024$ @ $512$ px)
Bottleneck	128 (B), 256 (H)	256	– (none)

Appendix

Table 7: Configuration for our JiT / AsymFlow / MFlow models. Architecture and optimizer follow JiT-B/L/H ( Li and He (2026) Tab. 9) for image/video; the text-to-image (T2I) stack is a PixelVAE-latent MMDiT. We list the settings we update.

Table	Model	Solver	Steps	CFG	$\sigma_{\min}$	CFG Interval
Table 5 (ImageNet, H, tuned)	JiT-H $+$ LPIPS	Heun	50	2.50	0.04	$[0.10,1.0]$
	AsymFlow-H $+$ LPIPS	Heun	50	2.20	0.04	$[0.10,1.0]$
	MFlow -H $+$ LPIPS	Heun	50	2.35	0.04	$[0.10,1.0]$
Table 5 (UCF, B, tuned)	Video JiT-B	Heun	100	3.0	$10^{-5}$	$[0,1.0]$
Table 5 (UCF, B, tuned)	Video AsymFlow-B	Heun	100	3.5	0.04	$[0,1.0]$
	Video MFlow -B	Heun	100	3.5	0.04	$[0,1.0]$

Appendix

Table 8: Evaluation / sampling hyperparameters for our trained models (Tables 2 , 5 , 5 , 5 ). Table 1 uses a single shared base JiT-B configuration while the rest are tuned per model.

Figure 13: Leading modes of the uncentered patch-PCA basis. We reshape the first 64 eigenvectors of the $16\times 16\times 3$ patch-PCA basis into RGB patches; the value $\lambda$ denotes the variance of each direction. The leading ordering interleaves patch-global or very low-frequency modes with spatially varying modes. In particular, PC0 is dominated by low-frequency patch-level tone, PC1 is an approximately uniform chromatic mode, PC2–PC3 encode spatial gradients, and PC4 returns to an approximately uniform chromatic mode. Eigenvector signs are arbitrary, so the interpretation is based on spatial structure rather than the displayed color polarity.

Table 17

Figure 14: ImageNet Loss Curves at H-level. We usually observe that the ${\bm{u}}$ -loss of MFlow training is very slightly better.

Figure 15: Faster convergence on ImageNet with LPIPS at B-level. This corresponds to the results noted in Table 2 .

Figure 16: Qualitative Text-to-Image Results on DPG-Bench. Images generated by our method and baselines on DPG-Bench prompts.

Figure 17: Qualitative Text-to-Image Results on PRISM-Bench. Images generated by our method and baselines on PRISM-Bench prompts.

Figure 18: Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.

Figure 19: More Class-wise Uncurated Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.

Figure 20: More Class-wise Uncurated Qualitative ImageNet Results. Images generated by MFlow + ConvRefiner H-level model.

Figure 21: Qualitative UCF-101 Results. Frames generated by MFlow +post-hoc FW-LPIPS L-level model.

Figure 22: Visualizing effect of ConvRefiner . Here we visualize the effect of adding our post-training to MFlow -H+LPIPS on ImageNet generations. We also note the average power gain in the high-frequency(0.5–0.7) region. We see overall improvement in texture details.

Figure 23: Qualitative Ablation for ConvRefiner . Here we visualize the effect of adding our post-training to JiT-B on ImageNet generations. We also note the average power gain in the high-frequency(0.5–0.7) region. We also show the visual improvements – we see that b4 adds additional texture details, while b8 sometimes overshoots in texture.

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 312 papers

Pixel-Space Diffusion Transformers
Jul 20, 2026Renye Yan, Jikang Cheng, You Wu +8Diffusion Transformer Pixel-Space Diffusion Models
Peking University, China · Nanjing University, China · Stanford University, USA +3
PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion
Jun 26, 2026Zipeng Guo, Lichen Ma, Yu He +4Diffusion Transformer Pixel-Space Diffusion Models
Xi’an Jiaotong University
PixelDiT2: Representation-Grounded Pixel Diffusion Transformers
Sep 21, 2026Yongsheng Yu, Wei Xiong, Yichen Sheng +2Diffusion Model Guidance Diffusion Transformer
NVIDIA · University of Rochester

Jul 20, 2026J/K move · Enter open · S save

Pixel-Space Diffusion Transformers

Renye Yan, Jikang Cheng, You Wu, Ling Liang+7

Peking University, China · Nanjing University, China · Stanford University, USA+3

Diffusion Transformer Pixel-Space Diffusion Models Diffusion-Based Image Generation Multimodal Diffusion Models

Open paper AskPDF arXiv

Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.

Abstract

Diffusion-Based Image Generation

Explore similar work

Pixel-Space Diffusion Transformers

PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion

PixelDiT2: Representation-Grounded Pixel Diffusion Transformers