Diffusion Models (DMs) have achieved state-of-the-art generative performance across multiple modalities, yet their sampling process remains prohibitively slow due to the need for hundreds of function evaluations. Recent progress in multi-step ODE solvers has greatly improved efficiency by reusing historical gradients, but existing methods rely on handcrafted coefficients that fail to adapt to the non-stationary dynamics of diffusion sampling. To address this limitation, we propose Dynamic Gradient Weighting (DyWeight), a lightweight, learning-based multi-step solver that introduces a streamlined implicit coupling paradigm. By relaxing classical numerical constraints, DyWeight learns unconstrained time-varying parameters that adaptively aggregate historical gradients while intrinsically scaling the effective step size. This implicit time calibration accurately aligns the solver's numerical trajectory with the model's internal denoising dynamics under large integration steps, avoiding complex decoupled parameterizations and optimizations. Extensive experiments on CIFAR-10, FFHQ, AFHQv2, ImageNet64, LSUN-Bedroom, Stable Diffusion and FLUX.1-dev demonstrate that DyWeight achieves superior visual fidelity and stability with significantly fewer function evaluations, establishing a new state-of-the-art among efficient diffusion solvers. Code is available at https://github.com/Westlake-AGI-Lab/DyWeight
Figures & tables
Figure 2: Overview of DyWeight . Our method dynamically aggregates buffered historical gradients ( wtn,j⋅vbuffer ) using unconstrained learned weights ( i.e . , not requiring ∑w=1 ). The unnormalized sum part adjusts the effective step size via time calibration ( t_{n}\rightarrow{\color[rgb]{0.9727,0.4805,0.4648}\tilde{t}_{n}} ). As illustrated on the right, DyWeight exhibits a distinct advantage in few-step generation scenarios.
Figure 3: L2 error under different complexity and steps settings. (a) Error vs. Complexity: We fix the solver steps ( S ) and increase the problem complexity (polynomial order K ). (b) Error vs. Steps: We fix the complexity and decrease the number of solver steps.
Figure 4: Effect of Time Shifting on Training Stability. We compare the (a) standard deviation of a gradient term specific to an intermediate point (expected =1 ) and (b) training loss for a 5-NFE student solver on CIFAR-10 [ 21 ] . The solver w/o time shifting ( red ) exhibits extreme variance and an unstable loss. In contrast, our proposed time shifting ( blue ) stabilizes the gradient variance around the expected value of 1.0 and achieves a smoother, lower-variance, and faster-converging training loss.
Table 1: Image generation FID ↓ [ 11 ] results across six datasets, pixel space : (a) CIFAR-10 ( 32×32 ) [ 21 ] , (b) FFHQ ( 64×64 ) [ 16 ] , (c) AFHQv2 ( 64×64 ) [ 4 ] , (d) ImageNet ( 64×64 ) [ 39 ] , and latent space : (e) LSUN-Bedroom ( 256×256 ) [ 59 ] , (f) MS-COCO ( 512×512 ) [ 26 ] with Stable-Diffusion-v1.5 (SD-v1.5) [ 37 ] . We compared our DyWeight with (1) Training-free multi-step solvers: iPNDM [ 60 , 28 ] , DPM-Solver++ [ 30 ] , and UniPC [ 61 ] , (2) Learnable single-step and multi-step solvers: AMED-Solver and AMED-Plugin [ 63 ] , EPD-Solver and EPD-Plugin [ 64 ] , LD3 [ 50 ] (we report the best FID results achieved by LD3 across different multi-step solvers) and DLMS [ 25 ] . The best results are in bold , the second best are underlined . See Appendix E.1 for complete results.
Metrics
Method
NFE=3
NFE=5
NFE=7
FID ↓
DPM-Solver++(2M) [ 30 ]
63.64
21.81
22.18
iPNDM(2M) [ 28 , 60 ]
48.46
25.03
22.22
DyWeight (ours)
42.69
21.02
20.69
CLIP ↑
DPM-Solver++(2M) [ 30 ]
24.40
26.38
26.20
iPNDM(2M) [ 28 , 60 ]
24.83
26.37
26.17
DyWeight (ours)
25.43
26.73
26.49
Table 2: Performance comparison on FLUX.1-dev [ 23 ] for text-to-image generation, evaluated on MS-COCO (512 × 512) [ 26 ] . DyWeight consistently outperforms competitive solvers in both FID [ 11 ] and CLIP score [ 35 ] .
Metrics
Method
NFE=5
NFE=7
NFE=9
CLIP ↑
DPM-Solver++(2M) [ 30 ]
26.95
27.35
27.36
iPNDM(2M) [ 28 , 60 ]
26.95
27.10
27.14
DyWeight (ours)
27.48
27.75
27.63
HPSv2.1 ↑
DPM-Solver++(2M) [ 30 ]
0.271
0.286
0.291
iPNDM(2M) [ 28 , 60 ]
0.273
0.286
0.291
DyWeight (ours)
0.279
0.288
0.293
Table 3: Performance comparison on FLUX.1-dev [ 23 ] for text-to-image generation, evaluated on Drawbench [ 42 ] . DyWeight consistently outperforms competitive solvers in CLIP score [ 35 ] , HPSv2.1 [ 55 ] and ImageReward [ 57 ] .
Figure 5: Training convergence on CIFAR-10 (200K training images). We track training loss (left) and generation FID (right) against optimization progress under various solver configurations. As shown, our framework exhibits extreme sample efficiency, achieving massive performance gains with merely ∼600 images.
Figure 6: Ablation studies. (a, b) Impact of time calibration components on CIFAR-10 [ 21 ] and FFHQ [ 16 ] . Both time shifting and time scaling are essential for optimal performance. (c) Sensitivity to multi-step order K on CIFAR-10 [ 21 ] , showing K=3 is optimal.
Teacher Solver ( M steps)
NFE
3
5
7
9
iPNDM (18) [ 60 , 28 ]
9.06
3.33
2.47
2.28
iPNDM (35) [ 60 , 28 ]
8.16
3.02
2.40
2.13
iPNDM (50) [ 60 , 28 ]
8.70
3.23
2.39
2.29
DPM-Solver-2 (35) [ 31 ]
8.77
3.17
2.44
2.24
Heun (35) [ 15 ]
8.80
3.09
2.28
2.12
Table 5: Robustness to teacher solver on CIFAR-10 [ 21 ] . Numbers in parentheses denote teacher steps. Student performance is stable across different high-fidelity teacher solvers and step counts.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Visualization of the learned coefficients across different datasets at 3 and 4 NFEs. Each row represents the learned parameters for a specific denoising step, starting with the Analytical First Step (AFS) [ 5 ] in the first row. The columns explicitly detail the unconstrained historical gradient weights ( wn,i ), followed by their unnormalized sum ( ∑wn,i ) and the time scaling factor ( sn ).
Dataset / Coeff.
C.-CIFAR
C.-ImgNet
C.-FFHQ
C.-LSUN
iPNDM
CIFAR-10 [ 21 ]
8.16
14.04
16.41
14.86
24.55
ImageNet [ 39 ]
25.18
17.37
21.20
20.11
34.81
FFHQ [ 16 ]
33.89
16.64
16.78
21.49
27.72
LSUN-Bed [ 59 ]
56.52
27.58
54.53
12.65
34.81
Appendix
Table A.1 : Cross-dataset evaluation of learned coefficients. We evaluate the transferability of 3-NFE parameters trained on one dataset (columns, denoted by “C.-”) across different evaluation distributions (rows). Native coefficients generally provide the most stable performance (diagonal).
Supervision strategy
NFE
3
5
7
9
CIFAR-10 [ 21 ]
iPNDM [ 28 , 60 ]
24.55
7.77
4.04
2.83
DyWeight - path
11.74
4.44
2.91
2.38
DyWeight - endpoint
8.16
3.02
2.40
2.13
ImageNet-64 [ 39 ]
Appendix
Table A.2 : Comparison of supervision strategies for training. We compare trajectory-based ( path ) supervision against terminal-state ( endpoint ) supervision . Path supervision computes the loss at every intermediate steps, forcing the student to strictly mimic the teacher’s trajectory. In contrast, endpoint supervision only supervises the final output, consistently yielding better results.
Dataset / NFE
CIFAR-10 [ 21 ]
AFHQv2 [ 4 ]
4
6
8
4
6
8
Weights initialization
Adams-Bashforth [ 9 , 1 ]
3.92
2.61
2.18
5.38
2.66
2.23
Euler [ 9 , 1 ]
6.46
3.92
2.80
8.26
4.19
2.77
Uniform
9.40
5.40
4.39
10.25
7.86
5.76
Time schedule initialization
Appendix
Table A.3 : Comparison of different initialization strategies for DyWeight . We report the FID [ 11 ] scores of different initialization methods across varying NFEs and datasets, all optimized using 10K training pairs. Top: Comparison of gradient weights initialization strategies. Adams-Bashforth [ 9 , 1 ] initializes weights using standard coefficients (iPNDM [ 28 , 60 ] ); Euler [ 9 , 1 ] sets the current step weight to 1 and others to 0; Uniform distributes weights evenly. Bottom: Comparison of time schedule initialization methods. We set ρ=7 for the EDM (Polynomial) [ 15 ] schedule and ρ=1 for the Time Uniform schedule.
Size
CIFAR-10 [ 21 ]
ImageNet [ 39 ]
NFE=4
NFE=6
NFE=8
NFE=4
NFE=6
NFE=8
10K
3.92
2.63
2.26
11.48
6.15
4.69
20K
4.09
2.61
2.18
12.94
5.80
5.02
30K
4.35
2.68
2.41
13.32
6.16
4.68
Appendix
Table A.4 : Impact of distillation dataset size (number of teacher samples) on CIFAR-10 [ 21 ] and ImageNet [ 39 ] . Performance saturates at 10K samples, demonstrating data efficiency.
Figure A.2 : Complete results of the numerical study. L2 error under different complexity and steps settings. (a) – (h) Error vs. Complexity: We fix the solver steps ( S ) and increase the problem complexity (polynomial order K ). (i) – (p) Error vs. Steps: We fix the complexity ( K ) and decrease the number of solver steps ( S ).
Method
NFE
3
4
5
6
7
8
9
iPNDM [ 60 , 28 ]
24.55
13.92
7.77
5.07
4.04
3.22
2.83
DPM-Solver++(3M) [ 30 ]
55.76
22.41
9.94
5.97
4.29
3.37
2.99
UniPC [ 61 ]
109.60
45.20
23.98
11.14
5.83
3.99
3.21
AMED-Solver [ 63 ]
18.49
17.18
7.59
7.04
4.36
5.56
3.67
AMED-Plugin [ 63 ]
10.81
-
6.61
-
3.65
-
2.63
Appendix
Table A.5 : Image generation FID ↓ [ 11 ] on CIFAR-10( 32×32 ) [ 21 ] . The best results are in bold , the second best are underlined .
Method
NFE
3
4
5
6
7
8
9
iPNDM [ 60 , 28 ]
27.72
20.07
13.80
8.61
7.16
5.43
4.98
DPM-Solver++(3M) [ 30 ]
66.07
30.06
13.47
8.25
6.20
5.20
4.77
UniPC [ 61 ]
86.43
44.78
21.40
12.85
7.44
5.50
4.47
AMED-Solver [ 63 ]
47.31
26.89
14.80
9.97
8,82
7.86
6.31
AMED-Plugin [ 63 ]
26.87
-
12.49
-
6.64
-
4.24
Appendix
Table A.6 : Image generation FID ↓ [ 11 ] on FFHQ( 64×64 ) [ 16 ] . The best results are in bold , the second best are underlined .
Method
NFE
3
4
5
6
7
8
9
iPNDM [ 60 , 28 ]
15.53
8.73
5.58
3.81
3.19
2.72
2.48
DPM-Solver++(3M) [ 30 ]
35.05
21.04
10.63
6.24
4.47
3.86
3.44
UniPC [ 61 ]
60.89
33.78
13.01
8.27
5.07
4.60
4.46
AMED-Solver [ 63 ]
31.82
18.99
7.34
8.19
4.39
5.55
3.53
LD3 [ 50 ]
17.94
9.96
6.09
3.63
2.97
2.63
2.27
Appendix
Table A.7 : Image generation FID ↓ [ 11 ] on AFHQv2( 64×64 ) [ 4 ] . The best results are in bold , the second best are underlined .
Method
NFE
3
4
5
6
7
8
9
iPNDM [ 60 , 28 ]
34.81
21.33
15.54
10.27
8.64
6.60
5.64
DPM-Solver++(3M) [ 30 ]
65.19
30.56
16.87
11.38
8.68
7.12
6.25
UniPC [ 61 ]
91.38
55.63
24.36
14.30
9.57
7.52
6.34
AMED-Solver [ 63 ]
38.10
32.69
10.74
10.63
6.66
7.71
5.44
AMED-Plugin [ 63 ]
28.06
-
13.83
-
7.81
-
5.60
Appendix
Table A.8 : Image generation FID ↓ [ 11 ] on ImageNet( 64×64 ) [ 39 ] . The best results are in bold , the second best are underlined .
Method
NFE
3
4
5
6
7
8
9
iPNDM [ 60 , 28 ]
43.31
26.14
18.46
12.81
11.03
9.49
7.89
DPM-Solver++(3M) [ 30 ]
111.90
48.49
18.44
8.39
5.18
4.12
3.77
UniPC [ 61 ]
112.30
39.66
13.76
6.46
4.52
3.96
3.72
AMED-Solver [ 63 ]
58.21
15.67
13.20
8.92
7.10
4.19
5.65
AMED-Plugin [ 63 ]
101.5
-
25.68
-
8.63
-
7.82
Appendix
Table A.9 : Image generation FID ↓ [ 11 ] on LSUN-Bedroom( 256×256 ) [ 59 ] . The best results are in bold , the second best are underlined .
Figure A.3 : Qualitative comparison on CIFAR-10 ( 32×32 ) [ 21 ] at 3 NFEs. DyWeight exhibits significantly reduced noise and better structure compared to iPNDM [ 28 , 60 ] .
Figure A.4 : Qualitative comparison on CIFAR-10 ( 32×32 ) [ 21 ] at 4 NFEs. DyWeight exhibits significantly reduced noise and better structure compared to iPNDM [ 28 , 60 ] .
Figure A.5 : Qualitative comparison on CIFAR-10 ( 32×32 ) [ 21 ] at 5 NFEs. DyWeight exhibits significantly reduced noise and better structure compared to iPNDM [ 28 , 60 ] .
Figure A.6 : Qualitative comparison on FFHQ ( 64×64 ) [ 16 ] at 3 NFEs. DyWeight yields fewer artifacts and clearer facial details compared with iPNDM [ 28 , 60 ] .
Figure A.7 : Qualitative comparison on FFHQ ( 64×64 ) [ 16 ] at 4 NFEs. DyWeight yields fewer artifacts and clearer facial details compared with iPNDM [ 28 , 60 ] .
Figure A.8 : Qualitative comparison on FFHQ ( 64×64 ) [ 16 ] at 5 NFEs. DyWeight yields fewer artifacts and clearer facial details compared with iPNDM [ 28 , 60 ] .
Figure A.9 : Qualitative comparison on AFHQv2 ( 64×64 ) [ 4 ] at 3 NFEs. DyWeight produces animal features with higher fidelity and noticeably less blurring compared with iPNDM [ 28 , 60 ] .
Figure A.10 : Qualitative comparison on AFHQv2 ( 64×64 ) [ 4 ] at 4 NFEs. DyWeight produces animal features with higher fidelity and noticeably less blurring compared with iPNDM [ 28 , 60 ] .
Figure A.11 : Qualitative comparison on AFHQv2 ( 64×64 ) [ 4 ] at 5 NFEs. DyWeight produces animal features with higher fidelity and noticeably less blurring compared with iPNDM [ 28 , 60 ] .
Figure A.12 : Qualitative comparison on ImageNet ( 64×64 ) [ 39 ] at 3 NFEs. DyWeight generates recognizable object classes with reduced noise compared to iPNDM [ 28 , 60 ] .
Figure A.13 : Qualitative comparison on ImageNet ( 64×64 ) [ 39 ] at 4 NFEs. DyWeight generates recognizable object classes with reduced noise compared to iPNDM [ 28 , 60 ] .
Figure A.14 : Qualitative comparison on ImageNet ( 64×64 ) [ 39 ] at 5 NFEs. DyWeight generates recognizable object classes with reduced noise compared to iPNDM [ 28 , 60 ] .
Figure A.15 : Qualitative comparison on LSUN-Bedroom ( 256×256 ) [ 59 ] at 3 NFEs. DyWeight preserves global room layout and structure, reduces blur, and enhances object details compared with iPNDM [ 28 , 60 ] .
Figure A.16 : Qualitative comparison on LSUN-Bedroom ( 256×256 ) [ 59 ] at 4 NFEs. DyWeight preserves global room layout and structure, reduces blur, and enhances object details compared with iPNDM [ 28 , 60 ] .
Figure A.17 : Qualitative comparison on LSUN-Bedroom ( 256×256 ) [ 59 ] at 5 NFEs. DyWeight preserves global room layout and structure, reduces blur, and enhances object details compared with iPNDM [ 28 , 60 ] .
Figure A.18 : Qualitative comparison on SD v1.5 [ 37 ] (MS-COCO [ 26 ] ) at 8 NFEs (4 Steps). Our method generates more detailed textures and coherent objects with reduced noise compared to iPNDM [ 28 , 60 ] . Prompts : (1) A small bird standing on top of a sidewalk. (2) a close up of a cat on the ground near a chair. (3) Teddy bear in front of a holly covered object. (4) An empty boat in the water near a tree.
Figure A.19 : Qualitative comparison on SD v1.5 [ 37 ] (MS-COCO [ 26 ] ) at 12 NFEs (6 Steps). Our method generates more detailed textures and coherent objects with reduced noise compared to iPNDM [ 28 , 60 ] . Prompts : (1) A small bird standing on top of a sidewalk. (2) a close up of a cat on the ground near a chair. (3) Teddy bear in front of a holly covered object. (4) An empty boat in the water near a tree.
Figure A.20 : Qualitative comparison on SD v1.5 [ 37 ] (MS-COCO [ 26 ] ) at 16 NFEs (8 Steps). Our method generates more detailed textures and coherent objects with reduced noise compared to iPNDM [ 28 , 60 ] . Prompts : (1) A small bird standing on top of a sidewalk. (2) a close up of a cat on the ground near a chair. (3) Teddy bear in front of a holly covered object. (4) An empty boat in the water near a tree.
Figure A.21 : Qualitative comparison on FLUX.1-dev [ 23 , 22 ] at 5 NFEs. From top to bottom, the rows show results from DPM-Solver++(2M) [ 30 ] , iPNDM(2M) [ 28 , 60 ] , and our DyWeight . Our method delivers superior visual fidelity, prompt alignment, and structural coherence.
Figure A.22 : Qualitative comparison on FLUX.1-dev [ 23 , 22 ] at 5 NFEs. From top to bottom, the rows show results from DPM-Solver++(2M) [ 30 ] , iPNDM(2M) [ 28 , 60 ] , and our DyWeight . Our method delivers superior visual fidelity, prompt alignment, and structural coherence.
Figure A.23 : Qualitative comparison on FLUX.1-dev [ 23 , 22 ] at 7 NFEs. From top to bottom, the rows show results from DPM-Solver++(2M) [ 30 ] , iPNDM(2M) [ 28 , 60 ] , and our DyWeight . Our method delivers superior visual fidelity, prompt alignment, and structural coherence.
Figure A.24 : Qualitative comparison on FLUX.1-dev [ 23 , 22 ] at 7 NFEs. From top to bottom, the rows show results from DPM-Solver++(2M) [ 30 ] , iPNDM(2M) [ 28 , 60 ] , and our DyWeight . Our method delivers superior visual fidelity, prompt alignment, and structural coherence.
Figure A.25 : Comprehensive list of text prompts employed for the FLUX.1-dev [ 23 , 22 ] qualitative comparisons.
Diffusion models achieve state-of-the-art image quality. However, sampling is costly at inference time because it requires a large number of function evaluations (NFEs). To reduce NFEs, classical ODE numerical methods have been adopted. Yet, the choice of prediction type and integration domain leads to different sampling behaviors. To address these issues, we introduce Dual-Solver, which generalizes multistep samplers through learnable parameters that continuously (i) interpolate among prediction types, (ii) select the integration domain, and (iii) adjust the residual terms. It retains the standard predictor-corrector structure while preserving second-order local accuracy. These parameters are learned via a classification-based objective using a frozen pretrained classifier (e.g., MobileNet or CLIP). For ImageNet class-conditional generation (DiT, GM-DiT) and text-to-image generation (SANA, PixArt-α), Dual-Solver improves FID and CLIP scores in the low-NFE regime (3≤ NFE ≤9) across backbones.
Diffusion-based generative models have achieved remarkable performance across various domains, yet their practical deployment is often limited by high sampling costs. While prior work focuses on training objectives or individual solvers, the broader sampling design problem, specifically solver selection and scheduling, remains largely governed by static heuristics. We propose SDM, a principled, training-free sampling framework that adapts both the numerical solver and the timestep schedule to the intrinsic properties of the diffusion trajectory. By analyzing the PF-ODE dynamics, we show that velocity variation is small in high-noise stages and increases near the data manifold, identifying intervals where solver order is most consequential. In parallel, we introduce an offline-calibrated adaptive scheduling method that explicitly controls the local Wasserstein discretization error and projects the calibrated trajectory to a prescribed NFE budget. We further extend the formulation to a mixed-transition Wasserstein error bound, providing a unified error-propagation view of adaptive scheduling and solver selection within the overall SDM framework. Across standard benchmarks, with extensions to modern ODE samplers, high-resolution synthesis, and text-to-image generation, SDM achieves improved sample quality compared to baseline methods, attaining an FID of 1.93 on CIFAR-10, 2.41 on FFHQ, and 1.98 on AFHQv2, with a reduced number of function evaluations compared to existing samplers. Our code is available at https://github.com/aiimaginglab/sdm.
Sangwoo Jo, Sungjoon Choi
Department of Artificial Intelligence, Korea University, Seoul, South Korea.
Diffusion models achieve state-of-the-art generative performance but suffer from high computational costs during inference due to the repeated evaluation of a heavy neural network. In this work, we propose Dual-Rate Diffusion, a method to accelerate sampling by interleaving the execution of a heavy high-capacity context encoder and a light efficient denoising model. The context encoder is evaluated sparsely to extract high-dimensional features, which are effectively reused by the light denoising model at every step to refine the sample efficiently. This approach significantly accelerates inference without compromising sample quality. On ImageNet benchmarks, Dual-Rate Diffusion matches the performance of standard baselines while reducing computational cost by a factor of 2-4. Furthermore, we demonstrate that our method is compatible with distillation techniques, such as Moment Matching Distillation, enabling further efficiency gains in few-step generation.
Grigory Bartosh, David Ruhe, Emiel Hoogeboom +3
*Work done while interning at Google DeepMind Amsterdam, 1University of Amsterdam, 2Google DeepMind Amsterdam · 2Google DeepMind Amsterdam