Analyzing and Improving Fast Sampling of Text-to-Image Diffusion Models
Authors: Zhenyu Zhou, Defang Chen, Siwei Lyu, Chun Chen, Can Wang
Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · University of California, Berkeley · State University of New York at Buffalo
Text-to-image diffusion models have achieved unprecedented success but still struggle to produce high-quality results under limited sampling budgets. Existing training-free sampling acceleration methods are typically developed independently, leaving the overall performance and compatibility among these methods unexplored. In this paper, we bridge this gap by systematically elucidating the design space, and our comprehensive experiments identify the sampling time schedule as the most pivotal factor. Inspired by the geometric properties of diffusion models revealed through the Frenet-Serret formulas, we propose constant total rotation schedule (TORS), a scheduling strategy that ensures uniform geometric variation along the sampling trajectory. TORS outperforms previous training-free acceleration methods and produces high-quality images with 10 sampling steps on Flux.1-Dev and Stable Diffusion 3.5. Extensive experiments underscore the adaptability of our method to unseen models, hyperparameters, and downstream applications. Code is available at https://github.com/zju-pi/TORS.
Figures & tables
Figure 1: Text-to-image generation using Flux.1-Dev [ 1 ] . Our proposed TORS attains image quality comparable to the 50-step baseline using only 10 steps.
Figure 2: (a) Diffusion sampling follows a predefined outer schedule {tn}n=0N , while an inner schedule determines whether each step performs computation or feature reuse. A sample xtn+1 transitions from tn+1 to tn with the solver’s update rule, utilizing stored historical velocities. Various types of cache objects can be selected, and a feature predictor is employed to estimate the current feature Δtn . This perspective also applies to U-Net architectures. (b) Four types of cache objects: velocity cache (V. cache), transformer cache (T. cache), block cache (B. cache), and operation cache (O. cache).
Figure 3: Comprehensive experiments examining the impact of each acceleration method on sampling acceleration. The experiments are conducted on Flux.1-Dev [ 1 ] . The outer schedule is identified as the most influential factor (see the vertical scale range).
Figure 4: Text-to-image generation with Flux.1-Dev [ 1 ] using the default uniform outer schedule. As the number of sampling steps increases, the image structure continues to change and only stabilizes after around 30 steps.
Figure 5: We extend the insights of sampling regularity from diffusion models to flow-based models and propose TORS by utilizing the geometric properties of sampling trajectories. (a) Visualization of 100 sampling trajectories generated by Flux.1-Dev reveals a strong trajectory regularity. All sampling trajectories starting from Gaussian noise are uniformly shifted to the origin. (b) A 10-step example of our proposed TORS, which ensures a constant total rotation change ∫0S∣ω(s)∣ds along the sampling trajectory. (c) TORS yields considerably faster structural convergence compared with the uniform outer schedule ( Figure 4 ).
Table 1: Results on text-to-image generation. Our proposed TORS achieves state-of-the-art results among sampling acceleration methods.
Model
Steps
Single Object
Two Object
Counting
Colors
Attribute Binding
Position
Overall ↑
Baseline
50
0.99
0.81
0.69
0.81
0.41
0.20
0.65
Baseline
10
0.96
0.71
0.59
0.69
0.28
0.20
0.57
TORS (ours)
10
1.00
0.82
0.67
0.80
0.49
0.22
0.67
Table 2: GenEval evaluation on Flux.1-dev.
Figure 8Table 9
Table 5: Ablation studies evaluating the robustness of TORS schedules on prompt distribution, sample sizes and step counts.
Method
Structure
Background Preservation
CLIP Similarity
Dist. ×103↓
PSNR ↑
LPIPS ×103↓
MES ×104↓
SSIM ×102↑
Whole ↑
Edit ↑
Baseline N=50
47.52
28.33
59.32
90.17
91.67
25.71
22.65
Baseline N=10
65.29
25.05
75.99
112.10
87.69
25.64
22.59
TORS N=10
52.46
27.59
64.87
90.90
90.37
25.76
22.70
Table 6: Performance of image editing evaluated on PIE-Bench using Flux.1-Kontext.
Figure 9: Qualitative results of image editing using Flux.1-Kontext.
Figure 10: Compatibility evaluation on acceleration methods using Image Reward. The performance can be enhanced by integrating appropriate acceleration strategies.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
∫0si∣ω(s)∣ds=NiΘ[0,S],for i=0,…,N.
Appendix
Algorithm 1 Generate TORS Schedule
Figure 15
Cache Object
Latency (s) ↓
VRAM (GB) ↓
OF=1/2/3
OF=1/2/3
V. cache
9.85 /
– /
–
36.32 /
– /
–
T. cache
9.90 /
9.92 /
9.94
36.38 /
36.44 /
38.61
B. cache
10.03 /
10.41 /
10.64
37.94 /
39.56 /
41.17
O. cache
10.47 /
10.87 /
11.19
38.48 /
40.63 /
42.78
Appendix
Table 7: Latency and VRAM requirements on Flux.1-Dev with a batch size of 1. All statistics are measured on a single A6000 GPU. Velocity cache requires 38.88 GB, 41.44 GB and 44.00 GB of VRAM for batch sizes of 2, 3 and 4, respectively.
Figure 17
Figure 15: Comprehensive experiments examining the impact of each acceleration method on sampling acceleration. The experiments are conducted on Stable Diffusion 3.5 medium model [ 6 ] . The outer schedule is identified as the most influential factor.
Figure 16: Text-to-image generation with Stable Diffusion 3.5 medium [ 6 ] using the default uniform outer schedule.
Figure 17: Trajectory regularity and the effectiveness of our proposed TORS on Stable Diffusion 3.5 medium.
Sampling from a diffusion model typically requires many forward passes through a large neural network, making generation computationally expensive. While much work has focused on efficient solvers and samplers, comparatively little attention has been paid to selecting the sampling timesteps themselves. A recent line of work optimizes theoretically derived surrogates for sample quality rather than the quality metric itself. We propose Optimizing Your Sampling (OYS), which instead treats timestep selection as a black-box optimization problem, optimizing the target metric directly with Bayesian optimization. OYS outperforms both the default schedules and those of Align Your Steps on text-to-image generation, and improves over the default schedules on inpainting and other image tasks, in both quantitative and human evaluations. OYS requires no additional training, is applicable even to distilled models, and improves both simple and sophisticated samplers such as Euler and DPM-Solver++. A 5-step OYS schedule retains 89%-94% of the quality of a 50-step schedule while reducing inference cost by 10x.
Travis Zhang, Christian Belardi, Justin Lovelace +4
Hardware-agnostic strategies for accelerating text-to-image diffusion, such as timestep distillation and feature caching, can reduce inference time without custom kernels or system-level optimization. Among them, multi-resolution generation strategies have recently received broad attention, attaining more than 5x speedup without any training. However, the design of performing upsampling in the latent space, together with the selective modification of partial regions, causes these methods to exhibit noticeable blurring or artifacts. To this end, we propose MrFlow, a training-free multi-resolution acceleration strategy for pretrained flow-matching models built upon a staged low-to-high-resolution pipeline. MrFlow first rapidly generates the main structure at low resolution, then performs super-resolution in the pixel space using a lightweight pretrained GAN-based model, subsequently injects low-strength noise to enable high-frequency resampling, and finally refines the details at high resolution. Quantitative and qualitative results on FLUX.1-dev and Qwen-Image show that MrFlow exploits the quadratic token reduction and reduced step requirement of low-resolution sampling to achieve 10x end-to-end acceleration while keeping OneIG within a 1% gap relative to that before acceleration, significantly surpassing other training-free acceleration strategies, and requiring no training or runtime dynamic identification whatsoever. MrFlow can further be directly combined orthogonally with pre-trained timestep distillation strategies, achieving even higher generation acceleration of up to 25x.
Xingyu Zheng, Xianglong Liu, Yifu Ding +4
State Key Laboratory of Complex & Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University · Nanyang Technological University +5
Text-to-image diffusion models expose many inference-time sampling parameters, including prompts, negative prompts, classifier-free guidance scales, and noise schedules. These parameters are typically manually chosen once and then held fixed across prompts and denoising timesteps, even though different prompts and stages of generation can benefit from different parameter values. We introduce LeSAMP, a framework for learning prompt-conditioned, timestep-varying sampling parameters. We formulate parameter selection as a reinforcement learning problem: Given a user prompt, a large language model is trained to emit schedules for the chosen sampling parameters. We optimize our model using rewards from human preference models and VLM-as-a-judge. We evaluate our model on Flux.1 [dev] and Stable Diffusion 3.5, and find that compared to baselines, LeSAMP has a win rate of up to 68.12% using human preference scores and 73.37% using VLM-as-a-judge. These gains are validated in a user study where we achieve win rates of up to 59.46% over previous baselines. Our results suggest that learned sampling-parameter policies provide a complementary approach to existing post-training methods for improving diffusion model outputs.