DRiffusion: Draft-and-Refine Process Parallelizes Diffusion Models with Ease
Authors: Runsheng Bai, Chengyu Zhang, Yangdong Deng
Organizations: CSAIL, MIT · IST, Nanjing University · School of Software, Tsinghua University; FuturististAI Lab, Shanghai Tsinghua International Innovation Center
Diffusion models have achieved remarkable success in generating high-fidelity content but suffer from slow, iterative sampling, resulting in high latency that limits their use in interactive applications. We introduce DRiffusion, a parallel sampling framework that parallelizes diffusion inference through a draft-and-refine process. DRiffusion employs skip transitions to generate multiple draft states for future timesteps and computes their corresponding noises in parallel, which are then used in the standard denoising process to produce refined results. Theoretically, our method achieves an acceleration rate of n1 or n+12, depending on whether the conservative or aggressive mode is used, where n denotes the number of devices. Empirically, DRiffusion attains 1.4×-3.7× speedup across multiple diffusion models while incur minimal degradation in generation quality: on MS-COCO dataset, both FID and CLIP remain largely on par with those of the original model, while PickScore and HPSv2.1 show only minor average drops of 0.17 and 0.43, respectively. These results verify that DRiffusion delivers substantial acceleration and preserves perceptual quality.
Figures & tables
Figure 1: Comparison across different prompts. Top row: original outputs; bottom row: results of 3-device aggressive parallelization.
Figure 2: Temporal dependencies of different methods. In DDPM, timesteps are sequential. DDIM and Euler Solvers allow global re-selection, but timestep traverse pattern within the subsequence or discretization remains sequential. The skip transitions allow direct moves between arbitrary pairs of timesteps, offering local temporal flexibility and enabling parallelization.
Figure 3: Computation map of different versions. Conservative one takes a standalone timestep for computing the noise for extra steps predicting while aggressive one directly takes the noise computed from draft states.
Figure 4: Qualitative comparison at 50 steps on MS-COCO using Stable Diffusion 2.1. The first row displays conservative results and the second row shows aggressive ones. Our parallelization method accelerates the sampling process with minimal impact on sample quality. The prompt used for generation is: A photo of a cat sitting on a wooden table in sunlight.
Model
Mode
#Devices
Latency (s)
Speed Up
FID ↓
CLIP ↑
Pick ↑
HPSv2.1 ↑
SD2.1
original
1
5.96
–
23.63
26.29
21.79
26.62
conservative
2
4.09
1.5 ×
23.60
26.32
21.78
26.61
3
3.12
1.9 ×
23.50
26.32
21.77
26.56
4
2.42
2.5 ×
23.47
26.30
21.75
26.49
aggressive
2
3.06
1.9 ×
23.39
26.31
21.77
26.56
3
2.11
2.8 ×
23.26
26.29
21.73
26.47
Table 1: Quantitative results of DRiffusion under different models on the MS-COCO dataset. Latency is averaged over steady runs. Speedup is measured against the 1-device original run of the same model. Number of samples per generation is 1.
Figure 5: Latency scaling of DRiffusion with respect to the number of devices (N) on Stable Diffusion 2.1 base using 50 sampling steps. Both modes closely follow their theoretical lower bounds.
Method
Devices
SpeedUp ↑
ExtraMem. ↓
FID ↓
CLIP ↑
Pick ↑
Stable Diffusion
1
–
+0MB
24.02
26.67
22.43
AsyncDiff (N=2, S=1)
2
1.6 ×
+494MB
24.13
26.60
22.34
DRiffusion (consv., 2 dev)
2
1.5 ×
+186MB
23.87
26.60
22.37
Stable Diffusion (step=25)
1
2.0 ×
+0MB
24.51
26.64
22.27
AsyncDiff (N=3, S=1)
3
2.1 ×
+502MB
24.17
26.56
22.30
DRiffusion (aggr., 2 dev)
2
1.9 ×
+186MB
24.08
26.61
22.33
Table 2: Comparison with baselines on MS-COCO dataset using SDXL. Speedups and ExtraMems are measured relatively to the 1-device SDXL baseline, and one sample per generation. We set the number of warm-up steps of AysncDiff to 5 to match our acceleration levels.
Model
Mode
#Devices
Latency (s)
Speed Up
FID ↓
CLIP ↑
Pick ↑
HPSv2.1 ↑
SD1.5
original
1
2.68
–
25.91
26.49
21.49
26.07
conservative
2
2.07
1.3 ×
25.95
26.60
21.51
25.98
3
1.43
1.9 ×
25.67
26.60
21.51
25.95
4
1.10
2.4 ×
25.57
26.61
21.50
25.92
aggressive
2
1.41
1.9 ×
26.04
26.55
21.50
25.94
3
1.00
2.7 ×
25.48
26.58
21.49
25.90
Table 3: Quantitative results of DRiffusion under SD1.5 and SD2.1-base on the MS-COCO dataset.
Figure 6: Comparison with LCM. We apply DRiffusion to LCM’s teacher. A cozy cabin in a snowy forest at night, glowing windows.