Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to 3.52× and 3.2× speedup on FLUX-1.Dev and Wan 2.1, respectively, without compromising the generation quality and prompt adherence.
Figures & tables
Figure 2: Main idea: dynamic tokenization during denoising . Current methods use the same patch size for all denoising steps during inference time. Instead, DDiT adapts the patch size at each timestep according to the latent complexity, allocating fewer tokens for certain timesteps and more tokens for certain others. While DiT divides VAE latents into patches, for illustrative purposes, we use a real image in pixel space.
Figure 3: Revised patch-embedding layer to support patches of varied resolutions. We modify the standard patch-embedding layer, designed for a fixed patch size p , to additionally support patch sizes pnew .
Figure 4: Inference speed vs. patch size. Inference speed measured over 50 denoising steps for generating 1024×1024 images using FLUX-1.Dev [ 54 ] , where every timestep uses a fixed patch size. As the patch size increases from p→2p→4p , the number of tokens decreases quadratically (4096 → 1024 → 256), resulting in approximately 3× and 4× faster inference for 2p and 4p , respectively, compared to p .
Figure 5: Given Δ(3)zt−1 , we divide it into patches of size pi×pi , compute within-patch standard deviation σt−1pi of the acceleration.
Figure 6: Visualization of σt−12p,(ρ) for two prompts (log scale). Prompt 1: “Several zebras are standing together behind a fence.” Prompt 2: “A simple red apple on a black background.” Prompts requiring different levels of spatial granularity exhibit distinct σt−12p,(ρ) patterns across timesteps. For the fine-grained zebra pattern, σt−12p,(ρ) remains higher, indicating higher detail sensitivity, whereas for the simpler apple scene, σt−12p,(ρ) is lower, thus we can seamlessly use larger patch sizes during generation.
Model
Speed ↓
COCO
DrawBench
PartiPrompts
(secs/image)
FID ↓
CLIP ↑
CLIP ↑
ImgR ↑
SSIM ↑
LPIPS ↓
CLIP ↑
ImgR ↑
FLUX-1.Dev (50 steps)
12.0
33.07
0.314
0.3156
1.0291
–
–
0.3197
1.192
FLUX-1.Dev (28 steps)
6.72
33.35
0.312
0.3140
1.0107
0.739 ± 0.155
0.175 ± 0.126
0.3118
1.115
FLUX-1.Dev (15 steps)
3.6
34.02
0.311
0.3121
0.9865
0.591 ± 0.155
0.298 ± 0.128
0.3072
0.9613
TeaCache ( δ=0.6 )
6.0
34.95
0.303
0.3071
0.9968
0.654 ± 0.145
0.241 ± 0.114
0.3018
0.9701
TeaCache ( δ=0.8 )
5.33
35.57
0.303
0.3075
0.9780
0.612 ± 0.146
0.279 ± 0.114
0.2991
0.9699
Table 1: Quantitative comparison of text-to-image generation performance with state-of-the-art methods on COCO, DrawBench, and PartiPrompts. If not specified, all results are reported using 50 inference steps by default. Each color ( Yellow , Blue ) indicates methods operating at similar inference speeds. As highlighted in Blue , our method achieves the best overall image quality, evidenced by the lowest FID scores, strong prompt alignment (CLIP and ImageReward), and high perceptual similarity (SSIM and LPIPS). Bold : best. Underline : second-best.
Figure 7: Qualitative comparisons with the base model [ 54 ] , TeaCache [ 61 ] , TaylorSeer [ 62 ] , and DDiT under similar speedups on DrawBench. DDiT effectively preserves fine-grained details, pose, spatial layout, and overall color distribution of the generated images.
Figure 8: Qualitative comparison on DrawBench with the baseline and TaylorSeer [ 62 ] . Our method remains robust even for complex prompts that require a deeper understanding of semantic content.
Figure 9: Qualitative comparison of text-to-video generation between DDiT and the baseline. DDiT produces videos with comparable visual quality to the baseline while achieving significant speedup.
Model
Speed ( ↑ )
VBench ( ↑ )
Wan-2.1 (Baseline)
1.0 ×
81.24
Ours ( τ=0.004 )
1.6 ×
81.17
Ours ( τ=0.001 )
2.1 ×
80.97
Ours ( τ=0.001 ) + TeaCache ( δ=0.05 )
3.2 ×
80.53
Table 2: Quantitative results on V-Bench [ 43 ] . Comparison of DDiT under different threshold settings ( τ ) and its combination with TeaCache [ 61 ] .
Method
FID↓
CLIP↑
ImageReward↑
DDiT ( n=1 )
34.71
0.2927
0.9782
DDiT ( n=2 )
34.28
0.3082
1.0128
DDiT ( n=3 )
33.42
0.3136
1.0284
Table 3: Effect of the n -th order difference on generation quality. Higher-order terms capture more informative temporal dynamics, improving both FID and CLIP scores. The third-order term ( n=3 ) achieves the best overall performance.
Figure 10: Sample patch schedules for 3 different prompts. Our dynamic patch scheduler seamlessly adapts to each prompt’s complexity and detail, thereby allocating more computation (aka higher percentage of smaller patch sizes) to images with highly detailed textures compared to simpler ones, thereby balancing efficiency and visual quality.
Method
Speed ( × )
CLIP
ImageReward
DDiT ( τ=0.004 )
1.88
0.3148
1.0271
DDiT ( τ=0.001 )
2.18
0.3136
1.0284
DDiT ( τ=0.01 )
3.52
0.3082
1.0124
Table 4: Effect of the threshold τ on DrawBench. Higher τ values yield faster inference at very mild dip in generation quality.
Diffusion Transformers rely on static patchify tokenization, assigning the same token budget to smooth backgrounds, detailed object regions, noisy early timesteps, and late-stage refinements. We introduce the Dynamic Chunking Diffusion Transformer (DC-DiT), which replaces fixed patchification with a learned encoder-router-decoder scaffold that adaptively compresses the 2D input into a shorter token sequence through a chunking mechanism learned end-to-end with diffusion training. DC-DiT allocates fewer tokens to predictable regions and noisy timesteps, and more tokens to detailed regions and later refinement stages, yielding meaningful spatial segmentations and timestep-adaptive compression schedules without supervision. Furthermore, the router provides an importance ordering over retained tokens, enabling elastic inference: a single checkpoint can be evaluated at flexible compute budgets with a smooth quality-compute tradeoff. Additionally, DC-DiT can be upcycled from pretrained DiT checkpoints and is also compatible with orthogonal dynamic computation approaches. On class-conditional ImageNet generation, DC-DiT reduces inference FLOPs by up to 36.8% and improves FID by up to 37.8% over DiT baselines, yielding a stronger quality--compute Pareto frontier across model scales, resolutions, and guidance settings. More broadly, these results suggest that adaptive tokenization is a general mechanism for making visual generation both more efficient and more flexible at inference time.
Diffusion Transformers (DiTs) have achieved state-of-the-art video generation quality, but they incur immense computational cost because standard inference applies the same number of denoising steps uniformly to every token in the sequence. It is well known that human vision ignores vast amounts of redundant motion. Why, then, do our densest models treat every spatiotemporal token with equal priority? In this paper, we introduce Heterogeneous Step Allocation (HSA), a training-free inference algorithm that assigns varying step budgets to different spatiotemporal tokens based on their velocity dynamics. To resolve the resulting sequence-length mismatch without sacrificing global context, HSA introduces a KV-cache synchronization mechanism that allows active tokens to attend to the full sequence while entirely bypassing inactive tokens. Furthermore, we derive a cached Euler update that advances the latent states of skipped tokens in a single operation without additional model evaluations. We evaluate HSA on the Wan-2 and LTX-2 models for both text-to-video (T2V) and image-to-video (I2V) generation. Our results demonstrate that HSA significantly outperforms previous state-of-the-art caching methods and the vanilla Flow Matching baseline, especially at aggressive acceleration regimes (e.g., 50% and 25% runtimes). Crucially, HSA achieves a superior quality-runtime Pareto frontier without the need for expensive offline profiling, robustly preserving structural integrity and generation quality even under tight computational budgets. Project page: https://ernestchu.github.io/hsa
Diffusion Transformers (DiTs) achieve high-quality generation but are costly due to iterative sampling. Dynamic-resolution sampling reduces early-stage cost by denoising at low resolution; however, uniformly upsampling all latent tokens at resolution transitions incurs redundant computation and may degrade fine-detail consistency. Existing partial upsampling strategies typically rely on local latent structure cues or single-step statistics, making it difficult to jointly capture token-text semantic relevance and token-wise representation dynamics across diffusion steps. We propose AViTS, an adaptive spatiotemporal token selection framework for dynamic-resolution DiTs. AViTS models spatial importance via latent-text attention and temporal importance via token-level feature variation across diffusion timesteps, and fuses them to enable spatiotemporal importance-aware selective upsampling: it prioritizes resolution refinement for critical tokens while deferring less important ones, thereby reducing redundant high-resolution computation and improving the quality-efficiency trade-off. AViTS achieves up to 6.34x on FLUX and nearly 9x FLOPs reduction on Qwen-Image-Edit and FLUX.1-Kontext-dev, orthogonal to distillation, quantization, and feature caching, and reaching 14.76x with distilled models. Code: https://github.com/QHR69/AViTS
Haoran Qin, Zhengan Yan, Shikang Zheng +9
Shanghai Jiao Tong University, China · Shandong University, China · Terminal Intelligent Computing Division, Alibaba Cloud, China +2