Refresh or Realize? Compute Allocation in Drifting Models

Sipeng Chen; Xu Zheng; Shibo Li

cs.LGSep 26, 2026

Refresh or Realize? Compute Allocation in Drifting Models

Authors: Sipeng Chen, Xu Zheng, Shibo Li

Organizations: Department of Computer Science Florida State University · Knight Foundation School of Computing and Information Sciences Florida International University

Abstract

Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet 256x256. Holding the target fixed for k optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, k=2 also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine ~0.3-0.6 against ~0.95), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.

Figures & tables

Figure 1: A field can become easier to realize and still not be the best place to spend compute. (a) Drifting prescribes a motion

V

toward a frozen target; one optimizer step executes a tangent-filtered motion

\Delta H_{1}\approx\eta KV

that need not coincide with

V

; repeated steps against the same frozen target move closer (dotted: remaining residual). (b) Measured on ImageNet

256\times 256

at the state from which all our downstream runs start: net progress toward the frozen target grows with depth apart from a small dip at

k=2

. (c) With the number of refreshes fixed,

k=2

takes two optimizer steps per refresh and reaches lower FID on both training seeds. (d) At approximately matched measured wall-clock,

k=1

affords more refreshes and reaches lower FID on both seeds. Panels (c) and (d) share one FID axis.

Figure 2: Repeated fitting turns the realized motion toward the prescribed one. Swiss-roll Drifting at one frozen training state. Blue: prescribed motion

V

, drawn wide and underneath; red: motion realized after

k

AdamW steps toward the same frozen target. Arrows are normalized for display; cosines use the full displacements of all 120 generated samples, and every panel shows the same 42 samples. The final panel is not an optimizer run but the

k\to\infty

limit

JJ^{\dagger}V

of the fixed-kernel model at the same state, where

K

is full rank and the prescribed motion is realized exactly.

checkpoint	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$	$k{=}32$
5k	$-7.7$	$-4.7$	$-7.1$	$-2.7$	$+11.0$	$+31.9$
10k	$-1.2$	$-4.7$	$-4.6$	$+3.5$	$+19.5$	$+37.5$
20k	$+0.6$	$-1.2$	$+3.3$	$+10.8$	$+23.3$	$+41.2$
30k	$+1.0$	$-0.6$	$+2.5$	$+9.2$	$+22.5$	$+36.9$

Table 1: The depth at which optimization begins to realize the frozen field falls from

k\approx 16

k=1

during training. Net progress

1-\rho_{k}^{2}

(

\times 10^{-3}

B_{\mathrm{eff}}=64

. Shaded: first depth with positive net progress.

budget	seed	$k=1$	$k=2$	better
equal refresh count $k{=}2$ gets $2\times$ updates	42	7.56	7.26	$k=2$
equal refresh count $k{=}2$ gets $2\times$ updates	43	7.60	7.30	$k=2$
$\approx$ matched measured wall-clock	42	6.98 (32.2 h)	7.26 (35.7 h)	$k=1$
$\approx$ matched measured wall-clock	43	6.93 (34.5 h)	7.30 (33.9 h)	$k=1$

Table 2: Extra realization helps when it is free; refreshing wins once it is paid for. FID under two budget definitions (lower is better; bold marks the better arm). Wall-clock is the scheduler-reported elapsed time of each run.

Figure 3: Refreshing the support moves the target far more than a parameter update does, and refreshing wins every local allocation decision. Forty probes, eight per position. The first three positions vary the support size at checkpoint 30k; the last three vary the training stage at

N=256

, sharing the

N=256

, 30k condition. (a) Cosine between the old target and the target rebuilt after one update, from the same support (dots) or a freshly drawn one (triangles); lines pair the two for each probe. (b) Held-out loss reduction of refresh over realize at matched compute; every probe is positive. Probes within a cell share a model state and are not independent.

Figure 4: A clean law in field space does not become a reliable gain in FID. (a) The audit-optimal coefficient follows an approximately

N^{-1/2}

law, stable between checkpoints

30

k and

60

k. (b) Downstream, the calibrated coefficient is not reliably better than the parameter-free

\beta=1

: it is better in three cells and worse in one, and the mean difference is

-0.005

FID. FID averaged over three evaluation seeds; exact values in Appendix H .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Optimizer	Metric	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$
Swiss roll	GD	cosine	0.342	0.448	0.516	0.585	0.651
	GD	residual	0.953	0.926	0.889	0.841	0.785
	AdamW	cosine	0.316	0.423	0.561	0.712	0.893
	AdamW	residual	0.949	0.909	0.834	0.703	0.457
Checkerboard	GD	cosine	0.370	0.469	0.553	0.621	0.694
	GD	residual	0.960	0.933	0.892	0.836	0.765

Appendix

Table 3: Finite-

k

realization under GD and AdamW. Cosine measures directional agreement between the prescribed field

V

and the actual finite output displacement

\Delta X_{k}

; residual is

\|V-\Delta X_{k}\|/\|V\|

. The audit is performed at a fixed intermediate training state with the same frozen target for all

k

. The GD learning rate is chosen from the local stability condition, whereas AdamW uses the practical fixed learning rate; the table is therefore intended to compare the trend with

k

, not the absolute performance of the two optimizers.

Figure 5: How realization depth ranks depends on what is held fixed. Swiss-roll sliced-Wasserstein distance under four budget definitions. Ranking the same runs by outer iterations, optimizer micro-steps, FLOPs and wall-clock gives different orderings, which is why the ImageNet comparison reports measured elapsed time for every arm.

$k$	$\rho_{k}$	$\cos$	$m_{k}$	$a_{k}$	$G_{\parallel,k}$	$P_{\perp,k}$	$1-\rho_{k}^{2}$
1	0.999505	0.03266	0.04146	0.001354	0.002706	0.001717	$+0.000989$
2	1.000303	0.03446	0.07681	0.002647	0.005287	0.005893	$-0.000606$
4	0.998774	0.05935	0.09208	0.005465	0.010900	0.008449	$+0.002451$
8	0.995368	0.09702	0.11004	0.010676	0.021238	0.011995	$+0.009243$
16	0.988698	0.15148	0.12983	0.019666	0.038945	0.016468	$+0.022477$
32	0.981388	0.19359	0.16914	0.032744	0.064415	0.027537	$+0.036878$

Appendix

Table 4: Full decomposition at checkpoint

30

B_{\mathrm{eff}}=64

(the data behind Fig. 1 b).

cell	$\cos(T_{0},T_{1}^{\mathrm{same}})$	$\cos(T_{0},T_{1}^{\mathrm{new}})$	refresh gain	refresh wins
$N=64$ , 30k	0.9790	0.3300	0.474%	8/8
$N=128$ , 30k	0.9780	0.4463	0.472%	8/8
$N=256$ , 30k	0.9767	0.5456	0.482%	8/8
$N=256$ , 20k	0.9704	0.5466	0.678%	8/8
$N=256$ , 10k	0.9476	0.5526	1.321%	8/8
all 40	0.9704	0.4842	0.685% (median 0.562%)	40/40

Appendix

Table 5: Local refresh/realize probes, cell means over eight probes each.

slice	field	floor active	cosine	rel. change	norm ratio
0	$\tau=0.2$	1.27%	0.99976	0.022	1.000
	$\tau=0.05$	49.83%	0.99121	0.133	1.000
	$\tau=0.02$	90.42%	0.96938	0.247	1.000
	aggregated	—	0.98930	0.147	0.975
1	$\tau=0.2$	1.28%	0.99968	0.025	1.000
	$\tau=0.05$	48.49%	0.99583	0.091	1.000

Appendix

Table 6: Affinity-floor diagnostic per slice and pooled. Relative change is

\left\lVert V_{\mathrm{clip}}-V_{\mathrm{noclip}}\right\rVert/\left\lVert V_{\mathrm{clip}}\right\rVert

; norm ratio is

\left\lVert V_{\mathrm{noclip}}\right\rVert/\left\lVert V_{\mathrm{clip}}\right\rVert

. Every temperature branch keeps unit norm ratio because it is RMS-normalized; the change is in direction.

seed	$N$	eval 0	eval 1	eval 2	mean
42	32	$-0.0941$	$-0.0341$	$-0.0455$	$-0.0579$
42	64	$-0.0943$	$-0.0856$	$+0.0524$	$-0.0425$
42	128	$-0.1405$	$+0.1000$	$-0.0690$	$-0.0365$
43	32	$-0.0484$	$+0.0143$	$-0.0899$	$-0.0413$
43	64	$-0.0500$	$-0.0489$	$+0.0026$	$-0.0321$
43	128	$-0.0732$	$+0.0154$	$-0.0362$	$-0.0313$

Appendix

Table 7: Paired

\mathrm{FID}(\beta^{\ast})-\mathrm{FID}(0)

for each evaluation seed.

seed	$N$	$\beta=0$	$\beta=1$	$\beta^{\ast}$	$\beta^{\ast}-\beta{=}1$
42	32	8.0228	7.9947	7.9649	$-0.0298$
43	32	8.0661	8.0434	8.0248	$-0.0186$
42	128	7.5843	7.4728	7.5478	$+0.0750$
43	128	7.6194	7.6329	7.5881	$-0.0448$

Appendix

Table 8: Calibrated

\beta^{\ast}

versus the parameter-free

\beta=1

(FID averaged over three evaluation seeds). The mean difference is

-0.0046

and the sign disagrees across seeds at

N=128

Drifting Models

Drifting models learn transport fields that move generated samples toward a target distribution, often enabling one-step generation.

See all 53 papers

DriftXpress: Faster Drifting Models via Projected RKHS Fields
May 12, 2026Ali Falahati, Elliot Creager, Gautam Kamath +1Image Generation One-Step Generative Modeling
Cheriton School of Computer Science, University of Waterloo and Vector Institute. · Department of Electrical and Computer Engineering, University of Waterloo and Vector Institute. · A*STAR Centre for Frontier AI Research, Singapore.
Learning Discriminative Geometry for Drifting Models
Oct 3, 2026Doudou Zhang, Wenwen Hou, Yilin Chen +1Drifting Models Image Generation
ELLIS Institute Finland · Aalto University
Lookahead Drifting Model
Apr 10, 2026Guoqiang Zhang, Kenta Niwa, W. Bastiaan KleijnDrifting Models Image Generation
University of Exeter · NTT Communication Science Laboratories · Victoria University of Wellington

May 12, 2026J/K move · Enter open · S save

DriftXpress: Faster Drifting Models via Projected RKHS Fields

Ali Falahati, Elliot Creager, Gautam Kamath, Shubhankar Mohapatra

Cheriton School of Computer Science, University of Waterloo and Vector Institute. · Department of Electrical and Computer Engineering, University of Waterloo and Vector Institute. · A*STAR Centre for Frontier AI Research, Singapore.

Image Generation One-Step Generative Modeling Kernel Methods Drifting Models

Open paper AskPDF arXiv

Drifting Models have emerged as a new paradigm for one-step generative modeling, achieving strong image quality without iterative inference. The premise is to replace the iterative denoising process in diffusion models with a single evaluation of a generator. However, this creates a different trade-off: drifting reduces inference cost by moving much of the computation into training. We introduce DriftXpress, an accelerated formulation of drifting models based on projected RKHS fields. DriftXpress approximates the drifting kernel in a low-rank feature space. This preserves the attraction-repulsion structure of the original drifting field while reducing the cost of field evaluation. Across image-generation benchmarks, DriftXpress achieves comparable FID to standard drifting while reducing wall-clock training cost. These results show that the training-inference trade-off of drifting models can be pushed further without giving up their one-step inference advantage.

Abstract

Drifting Models

Explore similar work

DriftXpress: Faster Drifting Models via Projected RKHS Fields

Learning Discriminative Geometry for Drifting Models

Lookahead Drifting Model