Sig-DEG for Distillation: Making Diffusion Models Faster and Lighter

Wen Ge; Lei Jiang; Lingyi Yang; Leonardo Roveri; Mingxuan Yi; Po-Yu Chen; Niels Cariou-Kotlarek; Francois Buet-Golfouse

cs.LGAug 23, 2025

Sig-DEG for Distillation: Making Diffusion Models Faster and Lighter

Authors: Wen Ge, Lei Jiang, Lingyi Yang, Leonardo Roveri, Mingxuan Yi, Po-Yu Chen, Niels Cariou-Kotlarek, Francois Buet-Golfouse, +2 more

Organizations: University College London · University of Oxford · JPMorganChase · AIML Global Markets, Barclays

Abstract

Diffusion models achieve state-of-the-art generative performance but remain computationally expensive at inference, often requiring many discretization steps. We propose Sig-DEG (Signature-based Differential Equation Generator), a generator for distilling pre-trained diffusion models. Inspired by high-order stochastic Taylor expansions, Sig-DEG combines partial signatures that summarize Brownian motion over subintervals with a recurrent structure to approximate stochastic differential equation solutions to achieve universality. We formulate distillation as supervised learning, matching fine-resolution teacher trajectories on a coarse time grid. At inference, Sig-DEG enables fast generation, as the partial signatures can be simulated exactly from their known distributions without requiring fine-grained Brownian paths. Experiments on image and time-series generation demonstrate competitive sample quality with inference speedups of up to several hundred times over the teacher. A Schrödinger bridge example further demonstrates Sig-DEG's applicability beyond diffusion distillation.

Figures & tables

Figure 1: Overview of Sig-DEG for distilling diffusion models.

Panel of Figure 2 — Figure 2: Qualitative generation results. (a) Sig-DEG on a 1D Gaussian mixture: backward trajectories (top) and final-sample histograms (bottom). (b–d) Teacher samples, Sig-DEG driven by the teacher’s Brownian path (Supervised), and Sig-DEG with independent noise (Generated): (b) rBerg price process (top) and variance process (bottom) paths; (c) MNIST; (d) CIFAR-10. Sig-DEG uses 5 coarse steps in (a–c) and 16 in (d).

Figure 3: Sample quality at matched sampling budgets. Sig-DEG and CM at the same number of steps on (a) MNIST (1500-step teacher), measured by FID, and (b) rBerg , measured by histogram loss. Lines indicate teacher performance; values below each step report Sig-DEG’s speed-up over its teacher.

Method	FID $\downarrow$	Time (s) $\downarrow$	$\times$ Speed $\uparrow$
Teacher (2048)	$3.63{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.05}$	$4517.0{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu5.9}$	–
DPM-Solver++ (8)	$10.45{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.07}$	26.3	172
DPM-Solver++ (32)	$3.25{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.03}$	105.2	43
Sig-DEG (64)	$3.42{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.04}$	$314.22{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.31}$	$14$
Sig-DEG (32)	$3.65{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$	$157.24{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.05}$	$29$
Sig-DEG (16)	$3.93{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.03}$	$78.55{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.19}$	$58$

Table 1: CIFAR-10 generation quality and inference speed. FID on 50,000 samples, mean

\pm

SD over five sampling seeds where given; time for 79 batches of 128 images on the same GPU. Teacher: Euler–Maruyama. DPM-Solver++ ( Lu et al., 2025 ) : training-free ODE solver (2M) on the same network as the teacher, better of two step schedules per budget. Sig-DEG (16, 8): resolution curriculum; Sig-DEG-A: adaptive mesh.

Panel of Figure 4 — Figure 4: CIFAR-10 training cost . (a) Sig-DEG (8) trained on a curriculum vs. on full-resolution from the start, up to epoch $285$ (total time: $16.2$ vs. $65.5$ h). (b) FID of the PD chain $64\to 32\to 16$ per epoch (one evaluation each) vs. cumulative training time; stars: direct training (five-seed means).

Panel of Figure 5 — Figure 5: V-neck task. Left : supervised trajectories (teacher’s are shaded); connecting segments are shown only for visualisation and do not represent trajectories. Right : performance of different coarse samplers.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Approximates	Supervision	Stochastic input
PF-ODE target
EDM Heun [ Karras et al., 2022 ]	PF-ODE (second-order Heun)	none (training-free)	initial noise only
DPM-Solver++ [ Lu et al., 2025 ]	PF-ODE	none (training-free)	initial noise only
PD [ Salimans and Ho, 2022 ]	PF-ODE (DDIM sampler)	one student step matches two teacher steps	initial noise only
Consistency models [ Song et al., 2023 ]	PF-ODE solution map	self-consistency on adjacent teacher ODE steps	fresh re-noising between steps
CTM [ Kim et al., 2024 ]	PF-ODE map, any $t\to s$	soft consistency, DSM, adversarial	optional fresh re-noising

Appendix

Table 2: Accelerated samplers for pre-trained diffusion models, grouped by their target (PF-ODE: probability-flow ODE). Sig-DEG is the only one that regresses fine-resolution teacher trajectories under the teacher’s own Brownian path, which enters as partial signatures sampled exactly from their Gaussian law (Lemmas 4 and 20 ).

	Teacher	10 steps		5 steps
Metric	(300)	SD	CM	SD	CM
W 1 $\downarrow$	$0.217{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.090}$	$0.191{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.079}$	$0.217{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.087}$	$0.208{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.087}$	$0.218{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.089}$
Variance $\downarrow$	$0.166{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.103}$	$0.171{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.107}$	$0.239{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.116}$	$0.219{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.118}$	$0.204{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.118}$
Time (ms) $\downarrow$	$163.76{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.52}$	$5.062{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.019}$	$6.680{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.016}$	$2.665{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.034}$	$3.347{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.018}$
$\times$ Speed $\uparrow$	–	$32$	$25$	$61$	$49$

Appendix

Table 3: 1D Gaussian Mixture (300-step teacher): Sig-DEG (SD) vs. CM. Mean over 5 training seeds

\pm

standard deviation over the test batches; time for 1,000 samples. Bold: the better of SD and CM.

Panel of Figure 6 — Figure 6: 1D Gaussian Mixture : 300-step teacher (green) and Sig-DEG (5) (purple), generating from $t=1$ (right) to $t=0$ (left). (a) Two teacher paths and Sig-DEG driven by the same Brownian paths (dots: coarse times). (b) Marginals at each coarse time, Sig-DEG with independent noise. Bottom: 2,000 final samples, (a) with the same Brownian paths, (b) with independent noise.

	VQ-VAE	Teacher	10 steps		5 steps
Metric	recon.	(1500)	SD	CM	SD	CM
FID $\downarrow$	$7.85^{\dagger}$	$8.94{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.07}$	$8.28{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.06}$	$10.27{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.01}$	$7.96{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.03}$	$10.04{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.06}$
IS $\uparrow$	–	$9.02{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$	$9.06{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.01}$	$8.88{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.01}$	$9.18{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$	$9.05{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.01}$
Time (s) $\downarrow$	–	$886.3$	$5.935{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.003}$	$5.994{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.002}$	$2.986{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.001}$	$2.990{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.001}$
$\times$ Speed $\uparrow$	–	–	$149$	$148$	$297$	$296$

Appendix

Table 4: MNIST (1500-step teacher): Sig-DEG (SD) vs. CM. FID (50,000 samples) and IS over three evaluations; time for 79 batches of 128 images. Bold: the better of SD and CM. † FID of the 10,000 test images after VQ-VAE encoding and decoding, without any generative model.

Teacher mesh (fine steps)	32	63	125	249	497	993	1985	2048
Sig-DEG (64), direct	–	–	–	–	–	–	–	508
Sig-DEG (32), direct	–	–	–	–	–	–	–	885
Sig-DEG (16), curriculum	106	138	181	277	543	977	1802	–
Sig-DEG (8), curriculum	126	185	287	573	1005	1858	3640	–

Appendix

Table 5: CIFAR-10 : training time per epoch (seconds, batch size 128), for the resolution curriculum on each rung of its teacher mesh and for direct training on the full 2048-step mesh. The cost grows with the fine steps per coarse step,

N_{f}/C

32

and

64

for Sig-DEG (64) and (32) on the 2048-step mesh, but about

124

and

248

for Sig-DEG (16) and (8) on the 1985-step rung. The curriculum rows are the runs of Sig-DEG (16) and (8) in Table 1 (Appendix D.4.4 ).

	Sampling steps
Chain	64	32	16	8
From Sig-DEG (64)	$3.42{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.04}$ (direct, $62.1$ h)	$3.63{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.04}$ ( $25.0$ h)	$4.07{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.05}$ ( $9.9$ h)	$5.18{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.04}$ ( $4.5$ h)
From Sig-DEG (32)	–	$3.65{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$ (direct, $108.2$ h)	$4.05{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.04}$ ( $9.9$ h)	$5.28{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$ ( $4.5$ h)

Appendix

Table 6: CIFAR-10 : PD chains from the directly trained Sig-DEG (64) and (32). Each stage halves the steps in 12 epochs on the frozen parent’s rollouts. FID on 50,000 samples, mean

\pm

standard deviation over five sampling seeds; in brackets: training time without validation, on one H100 NVL (direct: 440 epochs at the times of Table 5 ; PD: that stage).

Figure 7: Accumulation error for Sig-DEG (8). The figure shows the FID difference between a teacher completing a student-generated partial trajectory vs. the student doing it. Both start from the same student-generated states. $\sigma$ values are also reported, highlighting how its $\Delta\sigma$ varies enormously over intervals.

Method	FID $\downarrow$	Time (s) $\downarrow$	$\times$ Speed $\uparrow$
Sig-DEG-A (8)	$3.88{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.03}$	$39.33{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.02}$	115
Sig-DEG (8)	$4.47{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.05}$	$39.21{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.20}$	115
DPM-Solver++, ODE (8)	$10.45{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.07}$	26.3	172
DPM-Solver++, SDE (8)	26.63	26.4	171
EDM Heun (8)	6.62	50.8	89
Sig-DEG (16)	$3.93{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.03}$	$78.55{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.19}$	58

Appendix

Table 7: CIFAR-10 : Sig-DEG against training-free solvers run on the teacher’s network: DPM-Solver++(2M) [ Lu et al., 2025 ] as an ODE and an SDE solver, with the better of two step schedules (

\rho=7

and uniform in log-SNR) per budget, and the Heun sampler of Karras et al. [2022] , which evaluates the network twice per step (once on the last), so that EDM Heun (8) and (18) cost 15 and 35 evaluations. Rows are grouped by sampling cost. FID on 50,000 samples, mean

\pm

SD over five sampling seeds where given; time and speed-up as in Table 1 .

1000-step teacher
	Teacher	10 steps		5 steps		1 step
Metric		SD	CM	SD	CM	SD	CM
Hist. $\downarrow$	$0.0341{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0013}$	$0.0323{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.1340{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0015}$	$0.0348{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.1114{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.0467{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0008}$	$0.0943{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$
CrossCorr $\downarrow$	$0.1368{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0081}$	$0.1191{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0095}$	$0.1187{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0091}$	$0.1147{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0124}$	$0.1111{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0092}$	$0.0731{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0184}$	$0.1080{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0133}$
AutoCorr $\downarrow$	$0.2548{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0168}$	$0.2058{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0341}$	$0.1945{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0482}$	$0.1943{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0334}$	$0.1566{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0316}$	$0.1415{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0437}$	$0.1542{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0465}$
D-score $\downarrow$	$0.2293{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0079}$	$0.2446{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0150}$	$0.4441{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0155}$	$0.2557{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0128}$	$0.4161{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0304}$	$0.2191{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0134}$	$0.4281{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0116}$

Appendix

Table 8: rBerg : Sig-DEG (SD) vs. CM at the same number of steps, with a 1000-step and a 1500-step teacher (mean

\pm

SD over 10 sampling runs of 20k paths; time for 20k paths). P-scores are comparable only within a block (different cut-offs).

Figure 8: rBerg : Table 8 as best value / model value per metric and number of steps (rim: best; larger is better), with (a) the 1000-step and (b) the 1500-step teacher.

	Teacher, SDE on $N$ steps		Teacher, PF-ODE on $N$ steps		Sig-DEG ( $N$ )
$N$	Hist. $\downarrow$	D-score $\downarrow$	Hist. $\downarrow$	D-score $\downarrow$	Hist. $\downarrow$
1	$0.7818{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$0.4997{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$0.3041{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0016}$	$0.4897{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0030}$	$0.0467$
2	$0.8725{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0004}$	$0.4999{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$0.2403{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0015}$	$0.4862{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0023}$	$0.0384$
5	$0.6342{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0012}$	$0.4949{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0012}$	$0.0699{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0006}$	$0.3433{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0263}$	$0.0348$
10	$0.2276{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0013}$	$0.4307{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0094}$	$0.0557{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0011}$	$0.2615{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0192}$	$0.0323$
20	$0.1006{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.3265{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0215}$	$0.0553{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0011}$	$0.2623{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0176}$	–

Appendix

Table 9: rBerg : the 1000-step teacher sampled on

N

steps with its reverse SDE (Euler–Maruyama) or its probability-flow ODE, vs. Sig-DEG (

N

) (10 sampling runs;

N=1000

: an independent re-evaluation of the teacher).

Steps	Features	Hist. $\downarrow$	CrossCorr $\downarrow$	AutoCorr $\downarrow$	D-score $\downarrow$
10	full	$0.0323{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.119{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.010}$	$0.206{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.034}$	$0.245{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.015}$
	$\Delta W$ only	$0.0355{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$0.109{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.017}$	$0.176{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.065}$	$0.248{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.013}$
5	full	$0.0348{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.115{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.012}$	$0.194{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.033}$	$0.256{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.013}$
	$\Delta W$ only	$0.0369{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.108{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.022}$	$0.175{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.059}$	$0.250{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.016}$
2	full	$0.0384{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0008}$	$0.130{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.009}$	$0.238{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.029}$	$0.267{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.035}$
	$\Delta W$ only	$0.0397{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0008}$	$0.117{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.019}$	$0.203{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.050}$	$0.243{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.008}$

Appendix

Table 10: rBerg : Sig-DEG with the full partial signature

(\Delta W,\int(W-W_{s})\,du)

vs.

\Delta W

alone (10 sampling runs; bold: the better one).

		Hist. $\downarrow$		D-score $\downarrow$		Time (ms)
Width	# Params (SD/CM)	SD	CM	SD	CM	SD	CM
64	66k / 60k	$0.1344{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.1387{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$0.4561{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0144}$	$0.4596{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0134}$	$2.759{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.029}$	$3.369{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.013}$
128	223k / 211k	$0.0354{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.1224{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0012}$	$0.2775{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0234}$	$0.4530{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0097}$	$2.776{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.018}$	$3.269{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.020}$
192	471k / 451k	$0.0348{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.1114{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.2557{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0128}$	$0.4161{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0304}$	$3.380{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.015}$	$3.937{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.033}$
256	808k / 782k	$0.0335{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0008}$	$0.0903{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.2453{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0138}$	$0.4079{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0163}$	$3.956{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.013}$	$4.268{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.037}$
384	1753k / 1714k	$0.0306{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.0892{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0011}$	$0.2402{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0071}$	$0.3957{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0263}$	$5.453{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.020}$	$5.771{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.018}$

Appendix

Table 11: rBerg : network width at 5 sampling steps, Sig-DEG (SD) vs. CM.

Steps	Training	Hist. $\downarrow$	CrossCorr $\downarrow$	AutoCorr $\downarrow$	D-score $\downarrow$	P-score $\downarrow$	Train (h) $\downarrow$
10	Direct	$0.0323{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.1191{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0095}$	$0.2058{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0341}$	$0.2446{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0150}$	$0.0575{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$4.83$
	Curriculum	$0.0332{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.1148{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0116}$	$0.2070{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0380}$	$0.2489{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0169}$	$0.0572{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$4.46$
5	Direct	$0.0348{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0010}$	$0.1147{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0124}$	$0.1943{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0334}$	$0.2557{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0128}$	$0.0578{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0004}$	$6.24$
	Curriculum	$0.0347{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.1066{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0143}$	$0.1767{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0391}$	$0.2536{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0130}$	$0.0575{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0004}$	$4.98$
	PD $10\to 5$ , direct parent	$0.0316{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0009}$	$0.1101{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0140}$	$0.1763{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0401}$	$0.2670{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0164}$	$0.0571{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$5.49$
	PD $10\to 5$ , curriculum parent	$0.0324{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0006}$	$0.1063{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0160}$	$0.1689{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0460}$	$0.2345{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0074}$	$0.0572{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$5.12$

Appendix

Table 12: rBerg : training strategies at the same number of sampling steps: direct training on the 1000-step mesh, the resolution curriculum, and PD from a directly trained or curriculum-trained parent (10 sampling runs; bold: best per number of steps). Train: total training time, including validation and the parents.

Panel of Figure 9 — Figure 9: rBerg : validation histogram loss (2,048 paths) against training time, validation included (seed 42).

Figure 10: Marginals of generated rBerg paths at path times 1, 5, 10, 15, 20 and 25 (of 25; final samples): standardised price log-returns (top) and variance log-increments (bottom). Filled green: 1000-step teacher; blue: Sig-DEG (10).

		terminal $\mathcal{S}$ ( $\downarrow$ )		joint $\mathcal{S}$ ( $\downarrow$ )		path MSE ( $\downarrow$ )
teacher	$C$	increment-only	Sig-DEG	increment-only	Sig-DEG	increment-only	Sig-DEG
V-neck	2	$0.0124{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.0025{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0001}$	$0.0451{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0004}$	$0.0367{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0001}$	$0.5930{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0113}$	$0.4143{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0045}$
	4	$0.0063{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$0.0021{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$0.0400{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.0368{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0006}$	$0.3274{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0018}$	$0.1847{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0003}$
	10	$0.0021{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0002}$	$0.0016{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0001}$	$0.0371{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.0371{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$	$0.1353{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0007}$	$0.0458{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0013}$
	floor	$0.0016{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0001}$		$0.0366{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.0005}$		–
GMM with obstacles	2	$1.49{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.15}$	$1.10{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.09}$	$3.50{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.87}$	$2.73{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu0.23}$	$27.5{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu2.3}$	$23.2{\scriptstyle{}\mkern-1.0mu\pm\mkern-1.0mu1.0}$

Appendix

Table 13: Schrödinger-bridge teachers : increment-only student against Sig-DEG for

C

coarse steps (teacher:

200

steps), mean

\pm

SD over three training seeds. Terminal Sinkhorn divergence to the teacher (within-mode for the GMM teachers) and joint divergence of the

(X_{0},X_{T})

pairs (pooled), each with its floor (Appendix D.6.2 ); path MSE under the teacher’s Brownian path, for which a perfect student scores

0

. Bold: the lower of the two student means when their difference exceeds the larger of their two SDs.

Panel of Figure 11 — Figure 11: Schrödinger-bridge teachers : terminal Sinkhorn divergence to the teacher against the number of coarse steps $C$ (log–log; within-mode for the GMM teachers; error bars are the SD over three seeds). The dashed line is the floor and the grey band its $\pm 2$ SD. Teal: Sig-DEG (labelled SigDEG in the panels); orange: increment-only student; red: coarse Euler with the teacher’s drift. Panel titles give the checkpoint names.

Figure 12: Sig-DEG (4) distilled from the V-neck teacher. Left : the $200$ -step teacher, with source (grey) and terminal (teal) samples and $25$ paths. Center : $12$ teacher paths (faint) and the student driven by their Brownian paths, starting from the start points; the dots are the student’s states at the coarse times. Right : free generation of the student from $\mu_{0}$ with fresh noise, and $25$ of its paths. Segments between coarse states are drawn for legibility and are not paths: only the dots are states.

Panel of Figure 13 — Figure 13: V-neck teacher , states of $4{,}096$ freely generated paths at the three interior coarse times of the $4$ -step students (colour) over the teacher’s states at the same times (grey). Top: increment-only; bottom: Sig-DEG (4). At $t=T/2$ the teacher’s crowd is split on either side of the neck; both students put mass in the neck between the two lobes, the increment-only student more so.

Explore similar work

May 8, 2026

Stochastic Transition-Map Distillation for Fast Probabilistic Inference

Diffusion models achieve strong generation quality, diversity, and distribution coverage, but their performance often comes with expensive inference. In this work, we propose Stochastic Transition-Map Distillation (STMD), a teacher-free framework for accelerating diffusion model inference while preserving probabilistic sample generation. In contrast to score-based diffusion models, whose denoising parametrization models the mean of the posterior distribution, STMD distills the full transition map associated with the sampling stochastic differential equation (SDE). We parameterize these SDE transitions with a conditional Mean Flow model, yielding a one- or few-step stochastic sampler that retains the transition structure of the underlying diffusion process. This perspective is especially useful for downstream tasks that require stochastic inference, such as diffusion posterior sampling, inverse problems, and energy-based fine-tuning. Compared to recent distillation methods, STMD requires no pretrained teacher, bi-level optimization, or trajectory simulation and caching, enabling efficient and scalable training. We derive convergence bounds for our method in the Wasserstein distance, providing a strong theoretical foundation for our approach, and validate STMD on various image generation examples on the MNIST, CIFAR-10, and CelebA datasets.

George Rapakoulias, Peter Garud, Lingjiong Zhu +1

Department of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA · Department of Computer Science, Georgia Institute of Technology, Atlanta, GA · Department of Mathematics, Florida State University, Tallahassee, FL

Diffusion Model Distillation MeanFlow Diffusion Sampling Few-Step Diffusion Sampling Diffusion Model Inference Acceleration

Ask PDF

May 18, 2026

Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network

Diffusion models achieve state-of-the-art generative performance but suffer from high computational costs during inference due to the repeated evaluation of a heavy neural network. In this work, we propose Dual-Rate Diffusion, a method to accelerate sampling by interleaving the execution of a heavy high-capacity context encoder and a light efficient denoising model. The context encoder is evaluated sparsely to extract high-dimensional features, which are effectively reused by the light denoising model at every step to refine the sample efficiently. This approach significantly accelerates inference without compromising sample quality. On ImageNet benchmarks, Dual-Rate Diffusion matches the performance of standard baselines while reducing computational cost by a factor of

2

4

. Furthermore, we demonstrate that our method is compatible with distillation techniques, such as Moment Matching Distillation, enabling further efficiency gains in few-step generation.

Grigory Bartosh, David Ruhe, Emiel Hoogeboom +3

*Work done while interning at Google DeepMind Amsterdam, 1University of Amsterdam, 2Google DeepMind Amsterdam · 2Google DeepMind Amsterdam

Diffusion Sampling Diffusion-Based Image Generation Diffusion Model Inference Acceleration

Ask PDF

Mar 20, 2026

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.

Emiel Hoogeboom, David Ruhe, Jonathan Heek +2

Google DeepMind Amsterdam

Diffusion Model Distillation Diffusion Models Diffusion Language Model Distillation

Ask PDF