DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

Dahye Kim; Deepti Ghadiyaram; Raghudeep Gadde

cs.CVFeb 19, 2026

DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

Authors: Dahye Kim, Deepti Ghadiyaram, Raghudeep Gadde

Organizations: Boston University · Amazon

Abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to $3.52\times$ and $3.2\times$ speedup on FLUX-1.Dev and Wan $2.1$ , respectively, without compromising the generation quality and prompt adherence.

Figures & tables

Figure 2: Main idea: dynamic tokenization during denoising . Current methods use the same patch size for all denoising steps during inference time. Instead, DDiT adapts the patch size at each timestep according to the latent complexity, allocating fewer tokens for certain timesteps and more tokens for certain others. While DiT divides VAE latents into patches, for illustrative purposes, we use a real image in pixel space.

Figure 3: Revised patch-embedding layer to support patches of varied resolutions. We modify the standard patch-embedding layer, designed for a fixed patch size $p$ , to additionally support patch sizes $p_{\text{new}}$ .

Figure 4: Inference speed vs. patch size. Inference speed measured over

50

denoising steps for generating

1024\times 1024

images using FLUX-1.Dev [ 54 ] , where every timestep uses a fixed patch size. As the patch size increases from

p

\rightarrow

2p

\rightarrow

4p

, the number of tokens decreases quadratically (4096

\rightarrow

1024

\rightarrow

256), resulting in approximately

3\times

and

4\times

faster inference for

2p

and

4p

, respectively, compared to

p

Figure 5: Given

\Delta^{(3)}\mathbf{z}_{t-1}

, we divide it into patches of size

p_{i}\times p_{i}

, compute within-patch standard deviation

\boldsymbol{\sigma}_{t-1}^{p_{i}}

of the acceleration.

Figure 6: Visualization of $\boldsymbol{\sigma}_{t-1}^{2p,(\rho)}$ for two prompts (log scale). Prompt 1: “Several zebras are standing together behind a fence.” Prompt 2: “A simple red apple on a black background.” Prompts requiring different levels of spatial granularity exhibit distinct $\boldsymbol{\sigma}_{t-1}^{2p,(\rho)}$ patterns across timesteps. For the fine-grained zebra pattern, $\boldsymbol{\sigma}_{t-1}^{2p,(\rho)}$ remains higher, indicating higher detail sensitivity, whereas for the simpler apple scene, $\boldsymbol{\sigma}_{t-1}^{2p,(\rho)}$ is lower, thus we can seamlessly use larger patch sizes during generation.

Model	Speed $\downarrow$	COCO		DrawBench				PartiPrompts
Model	(secs/image)	FID $\downarrow$	CLIP $\uparrow$	CLIP $\uparrow$	ImgR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	ImgR $\uparrow$
FLUX-1.Dev (50 steps)	12.0	33.07	0.314	0.3156	1.0291	–	–	0.3197	1.192
FLUX-1.Dev (28 steps)	6.72	33.35	0.312	0.3140	1.0107	0.739 $\pm$ 0.155	0.175 $\pm$ 0.126	0.3118	1.115
FLUX-1.Dev (15 steps)	3.6	34.02	0.311	0.3121	0.9865	0.591 $\pm$ 0.155	0.298 $\pm$ 0.128	0.3072	0.9613
TeaCache ( $\delta=0.6$ )	6.0	34.95	0.303	0.3071	0.9968	0.654 $\pm$ 0.145	0.241 $\pm$ 0.114	0.3018	0.9701
TeaCache ( $\delta=0.8$ )	5.33	35.57	0.303	0.3075	0.9780	0.612 $\pm$ 0.146	0.279 $\pm$ 0.114	0.2991	0.9699

Table 1: Quantitative comparison of text-to-image generation performance with state-of-the-art methods on COCO, DrawBench, and PartiPrompts. If not specified, all results are reported using 50 inference steps by default. Each color ( Yellow , Blue ) indicates methods operating at similar inference speeds. As highlighted in Blue , our method achieves the best overall image quality, evidenced by the lowest FID scores, strong prompt alignment (CLIP and ImageReward), and high perceptual similarity (SSIM and LPIPS). Bold : best. Underline : second-best.

Figure 7: Qualitative comparisons with the base model [ 54 ] , TeaCache [ 61 ] , TaylorSeer [ 62 ] , and DDiT under similar speedups on DrawBench. DDiT effectively preserves fine-grained details, pose, spatial layout, and overall color distribution of the generated images.

Figure 8: Qualitative comparison on DrawBench with the baseline and TaylorSeer [ 62 ] . Our method remains robust even for complex prompts that require a deeper understanding of semantic content.

Figure 9: Qualitative comparison of text-to-video generation between DDiT and the baseline. DDiT produces videos with comparable visual quality to the baseline while achieving significant speedup.

Model	Speed ( $\uparrow$ )	VBench ( $\uparrow$ )
Wan-2.1 (Baseline)	1.0 $\times$	81.24
Ours ( $\tau{=}0.004$ )	1.6 $\times$	81.17
Ours ( $\tau{=}0.001$ )	2.1 $\times$	80.97
Ours ( $\tau{=}0.001$ ) + TeaCache ( $\delta{=}0.05$ )	3.2 $\times$	80.53

Table 2: Quantitative results on V-Bench [ 43 ] . Comparison of DDiT under different threshold settings (

\tau

) and its combination with TeaCache [ 61 ] .

Method	FID↓	CLIP↑	ImageReward↑
DDiT ( $n=1$ )	34.71	0.2927	0.9782
DDiT ( $n=2$ )	34.28	0.3082	1.0128
DDiT ( $n=3$ )	33.42	0.3136	1.0284

Table 3: Effect of the

n

-th order difference on generation quality. Higher-order terms capture more informative temporal dynamics, improving both FID and CLIP scores. The third-order term (

n=3

) achieves the best overall performance.

Figure 10: Sample patch schedules for 3 different prompts. Our dynamic patch scheduler seamlessly adapts to each prompt’s complexity and detail, thereby allocating more computation (aka higher percentage of smaller patch sizes) to images with highly detailed textures compared to simpler ones, thereby balancing efficiency and visual quality.

Method	Speed ( $\times$ )	CLIP	ImageReward
DDiT ( $\tau{=}0.004$ )	1.88	0.3148	1.0271
DDiT ( $\tau{=}0.001$ )	2.18	0.3136	1.0284
DDiT ( $\tau{=}0.01$ )	3.52	0.3082	1.0124

Table 4: Effect of the threshold

\tau

on DrawBench. Higher

\tau

values yield faster inference at very mild dip in generation quality.

Explore similar work

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking
Mar 6, 2026Akash Haridas, Utkarsh Saxena, Parsa Ashrafi Fashi +3Diffusion Transformer Diffusion-Based Image Generation
Advanced Micro Devices Inc. (AMD)
Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
May 7, 2026Ernie Chu, Vishal M. PatelVideo Diffusion Models Diffusion Transformer
Johns Hopkins University Baltimore, MD 21218
AViTS: Adaptive Spatiotemporal Token Selection for Efficient Dynamic-Resolution Generation
Aug 18, 2026Haoran Qin, Zhengan Yan, Shikang Zheng +9Diffusion-Based Image Generation Diffusion Model Acceleration
Shanghai Jiao Tong University, China · Shandong University, China · Terminal Intelligent Computing Division, Alibaba Cloud, China +2

Mar 6, 2026J/K move · Enter open · S save

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking

Akash Haridas, Utkarsh Saxena, Parsa Ashrafi Fashi, Mehdi Rezagholizadeh+2

Advanced Micro Devices Inc. (AMD)

Diffusion Transformer Diffusion-Based Image Generation Diffusion Model Inference Acceleration Visual Token Compression

Open paper AskPDF arXiv

Diffusion Transformers rely on static patchify tokenization, assigning the same token budget to smooth backgrounds, detailed object regions, noisy early timesteps, and late-stage refinements. We introduce the Dynamic Chunking Diffusion Transformer (DC-DiT), which replaces fixed patchification with a learned encoder-router-decoder scaffold that adaptively compresses the 2D input into a shorter token sequence through a chunking mechanism learned end-to-end with diffusion training. DC-DiT allocates fewer tokens to predictable regions and noisy timesteps, and more tokens to detailed regions and later refinement stages, yielding meaningful spatial segmentations and timestep-adaptive compression schedules without supervision. Furthermore, the router provides an importance ordering over retained tokens, enabling elastic inference: a single checkpoint can be evaluated at flexible compute budgets with a smooth quality-compute tradeoff. Additionally, DC-DiT can be upcycled from pretrained DiT checkpoints and is also compatible with orthogonal dynamic computation approaches. On class-conditional ImageNet generation, DC-DiT reduces inference FLOPs by up to 36.8% and improves FID by up to 37.8% over DiT baselines, yielding a stronger quality--compute Pareto frontier across model scales, resolutions, and guidance settings. More broadly, these results suggest that adaptive tokenization is a general mechanism for making visual generation both more efficient and more flexible at inference time.

Abstract

Explore similar work

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation

AViTS: Adaptive Spatiotemporal Token Selection for Efficient Dynamic-Resolution Generation