Diffusion models have achieved remarkable progress in high-fidelity image, video, and audio generation, yet inference remains computationally expensive. Nevertheless, current diffusion acceleration methods based on distributed parallelism suffer from noticeable generation artifacts and fail to achieve substantial acceleration proportional to the number of GPUs. Therefore, we propose a hybrid parallelism framework that combines a novel data parallel strategy, condition-based partitioning, with an optimal pipeline scheduling method, adaptive parallelism switching, to reduce generation latency and achieve high generation quality in conditional diffusion models. The key ideas are to (i) leverage the conditional and unconditional denoising paths as a new data-partitioning perspective and (ii) adaptively enable optimal pipeline parallelism according to the denoising discrepancy between these two paths. Our framework achieves 2.31× and 2.07× latency reductions on SDXL and SD3, respectively, using two NVIDIA RTX~3090 GPUs, while preserving image quality. This result confirms the generality of our approach across U-Net-based diffusion models and DiT-based flow-matching architectures. Our approach also outperforms existing methods in acceleration under high-resolution synthesis settings. Code is available at https://github.com/kaist-dmlab/Hybridiff.
Figures & tables
Figure 2: Comparison of parallel strategies for diffusion inference. (a) Patch-based data parallel frameworks suffer from bottlenecks caused by all-gather operations and artifacts at patch boundaries, leading to limited acceleration and quality degradation. (b) Pipeline parallel frameworks incur excessive asynchronous communication overhead and accumulate estimate errors. (c) Our hybrid parallelism , which incorporates condition-based data parallelism, adaptively combines both paradigms to achieve high fidelity and fast generation.
Figure 3: Overview of the proposed diffusion inference hybrid parallel framework. Our method adaptively switches parallelism modes at τ1 and τ2 , optimizing the trade-off between computational efficiency and consistency of conditional guidance, and demonstrates superior inference acceleration performance while preserving high generation quality.
Figure 4: Illustration of the rel-MAEt(ϵc,ϵu) curve. The rel-MAEt(ϵc,ϵu) value is relatively large before τ1 and after τ2 , while it converges near zero between them, indicating stable alignment between conditional and unconditional branches during the parallelism phase.
Base Model
Devices
Methods
Latency (s) ↓
Speed-Up ↑
Comm. (GB) ↓
FID ↓
LPIPS ↓
PSNR ↑
w/ G. T.
w/ Orig.
w/ G. T.
w/ Orig.
w/ G. T.
w/ Orig.
Stable Diffusion XL
1
Original Model
16.49
-
-
23.977
-
0.797
-
9.618
-
2
DistriFusion [ 11 ]
13.53
1.22 ×
0.525
24.164
4.864
0.7978
0.146
9.597
24.634
AsyncDiff [ 2 ] (stride=1)
12.54
1.31 ×
9.830
23.941
4.103
0.797
0.108
9.586
26.387
Ours ( k =5)
7.12
2.31 ×
0.516
23.831
4.100
0.796
0.107
9.665
26.640
Stable Diffusion 3
1
Original Model
19.36
-
-
33.433
-
0.810
-
8.086
-
Table 1: Quantitative comparison of parallelism methods on the Stable Diffusion XL and Stable Diffusion 3 models. We compare our method with existing distributed inference techniques under 1- and 2-GPU. We report both the baseline latency and the corresponding acceleration ratio ( Speed-Up ), Communication efficiency (Comm.) , and quantitative metrics assessing generation fidelity. Here, w/ G.T. denotes comparison with the ground-truth image, and w/ Orig. indicates comparison with the original (single-GPU) model output.
Figure 5: Qualitative results of the main experiments. We compare 1024 × 1024 image generations from the SDXL model. Our method achieves the best acceleration and FID performance, while producing visuals most similar to the original.
Figure 6: Visualization of speed–quality trade-off across different parallelism intervals k . Smaller k values preserve higher fidelity, whereas larger k achieve greater acceleration. Our method consistently dominates prior works across the trade-off frontier. All experiments were conducted on 2 GPUs.
Methods
Latency (s) ↓
Speed-Up ↑
FID ↓
(w/ Orig.)
Original Model
16.49
-
-
Full Condition-Based Partitioning
9.24
1.78 ×
3.623
Ours (Hybrid Parallelism)
7.12
2.31 ×
4.100
Table 2: Ablation on hybrid parallel components. All experiments are conducted on the SDXL model at 1024 × 1024 resolution, comparing the single-GPU baseline, full condition-based partitioning, and our hybrid parallelism framework.
Figure 7: Comparison of high-resolutions tasks. We compare different parallel inference methods on the SDXL model using NVIDIA H200 GPUs across 1024 × 1024, 2048 × 2048, and 2560 × 2560 high-resolutions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Methods
Speed-Up ↑
Image Quality ↑
Model General. ↑
High-res Synth. ↑
Comm. Efficiency ↑
Distrifusion
2.5
3.5
2.5
3.3
5.0
AsyncDiff
3.0
4.5
5.0
3.5
1.0
Ours
4.7
4.5
5.0
4.4
5.0
Appendix
Table 3: Quantitative metrics comparison across five evaluation aspects. Scores are normalized to a 5-point scale. Higher values ( ↑ ) indicate better performance.
Figure 8: Empirical visualization of denoising discrepancy curve.
Figure 9: Extensibility to many GPU configurations structures. This figure illustrates two strategies for scaling the proposed hybrid parallelism framework to larger GPU configurations. These structures demonstrate how the proposed framework naturally generalizes from the 2 GPUs setting to both batch-level and layer-wise many GPU configurations.
Parallelism Interval k
Latency (s) ↓
Speed-Up ↑
FID ↓
(w/ Orig.)
k =5
7.12
2.31 ×
4.100
k =10
6.89
2.39 ×
5.942
k =20
6.44
2.56 ×
7.966
k =30
5.94
2.78 ×
9.191
Appendix
Table 4: Effect of speed-quality trade-off across different parallelism intervals k . All experiments are conducted on the SDXL model at 1024 × 1024 resolution with various parallelism intervals.
Figure 10: Additional qualitative results of the main experiments. We compare 1024 × 1024 image generations from the SDXL model. Our method achieves the best acceleration and FID performance, while producing visuals most similar to the original.
Figure 11: Additional qualitative comparisons across different k values. We compare 1024 × 1024 image generations from the SDXL model across various parallelism intervals. Smaller k values preserve higher visual fidelity, whereas larger k gradually reduce local detail due to the extended parallelism window. Although the overall appearance remains similar, fine-grained conditional attributes become subtly blurred as k increases.