Refresh or Realize? Compute Allocation in Drifting Models

Sipeng Chen; Xu Zheng; Shibo Li

cs.LGSep 26, 2026

Refresh or Realize? Compute Allocation in Drifting Models

Authors: Sipeng Chen, Xu Zheng, Shibo Li

Organizations: Department of Computer Science Florida State University · Knight Foundation School of Computing and Information Sciences Florida International University

Abstract

Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet 256x256. Holding the target fixed for k optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, k=2 also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine ~0.3-0.6 against ~0.95), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.

Figures & tables

Figure 1: A field can become easier to realize and still not be the best place to spend compute. (a) Drifting prescribes a motion

V

toward a frozen target; one optimizer step executes a tangent-filtered motion

\Delta H_{1}\approx\eta KV

that need not coincide with

V

; repeated steps against the same frozen target move closer (dotted: remaining residual). (b) Measured on ImageNet

256\times 256

at the state from which all our downstream runs start: net progress toward the frozen target grows with depth apart from a small dip at

k=2

. (c) With the number of refreshes fixed,

k=2

takes two optimizer steps per refresh and reaches lower FID on both training seeds. (d) At approximately matched measured wall-clock,

k=1

affords more refreshes and reaches lower FID on both seeds. Panels (c) and (d) share one FID axis.

Figure 2: Repeated fitting turns the realized motion toward the prescribed one. Swiss-roll Drifting at one frozen training state. Blue: prescribed motion

V

, drawn wide and underneath; red: motion realized after

k

AdamW steps toward the same frozen target. Arrows are normalized for display; cosines use the full displacements of all 120 generated samples, and every panel shows the same 42 samples. The final panel is not an optimizer run but the

k\to\infty

limit

JJ^{\dagger}V

of the fixed-kernel model at the same state, where

K

is full rank and the prescribed motion is realized exactly.

checkpoint	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$	$k{=}32$
5k	$-7.7$	$-4.7$	$-7.1$	$-2.7$	$+11.0$	$+31.9$
10k	$-1.2$	$-4.7$	$-4.6$	$+3.5$	$+19.5$	$+37.5$
20k	$+0.6$	$-1.2$	$+3.3$	$+10.8$	$+23.3$	$+41.2$
30k	$+1.0$	$-0.6$	$+2.5$	$+9.2$	$+22.5$	$+36.9$

Table 1: The depth at which optimization begins to realize the frozen field falls from

k\approx 16

k=1

during training. Net progress

1-\rho_{k}^{2}

(

\times 10^{-3}

B_{\mathrm{eff}}=64

. Shaded: first depth with positive net progress.

budget	seed	$k=1$	$k=2$	better
equal refresh count $k{=}2$ gets $2\times$ updates	42	7.56	7.26	$k=2$
equal refresh count $k{=}2$ gets $2\times$ updates	43	7.60	7.30	$k=2$
$\approx$ matched measured wall-clock	42	6.98 (32.2 h)	7.26 (35.7 h)	$k=1$
$\approx$ matched measured wall-clock	43	6.93 (34.5 h)	7.30 (33.9 h)	$k=1$

Table 2: Extra realization helps when it is free; refreshing wins once it is paid for. FID under two budget definitions (lower is better; bold marks the better arm). Wall-clock is the scheduler-reported elapsed time of each run.

Figure 3: Refreshing the support moves the target far more than a parameter update does, and refreshing wins every local allocation decision. Forty probes, eight per position. The first three positions vary the support size at checkpoint 30k; the last three vary the training stage at

N=256

, sharing the

N=256

, 30k condition. (a) Cosine between the old target and the target rebuilt after one update, from the same support (dots) or a freshly drawn one (triangles); lines pair the two for each probe. (b) Held-out loss reduction of refresh over realize at matched compute; every probe is positive. Probes within a cell share a model state and are not independent.

Figure 4: A clean law in field space does not become a reliable gain in FID. (a) The audit-optimal coefficient follows an approximately

N^{-1/2}

law, stable between checkpoints

30

k and

60

k. (b) Downstream, the calibrated coefficient is not reliably better than the parameter-free

\beta=1

: it is better in three cells and worse in one, and the mean difference is

-0.005

FID. FID averaged over three evaluation seeds; exact values in Appendix H .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Optimizer	Metric	$k=1$	$k=2$	$k=4$	$k=8$	$k=16$
Swiss roll	GD	cosine	0.342	0.448	0.516	0.585	0.651
	GD	residual	0.953	0.926	0.889	0.841	0.785
	AdamW	cosine	0.316	0.423	0.561	0.712	0.893
	AdamW	residual	0.949	0.909	0.834	0.703	0.457
Checkerboard	GD	cosine	0.370	0.469	0.553	0.621	0.694
	GD	residual	0.960	0.933	0.892	0.836	0.765

Appendix

Table 3: Finite-

k

realization under GD and AdamW. Cosine measures directional agreement between the prescribed field

V

and the actual finite output displacement

\Delta X_{k}

; residual is

\|V-\Delta X_{k}\|/\|V\|

. The audit is performed at a fixed intermediate training state with the same frozen target for all

k

. The GD learning rate is chosen from the local stability condition, whereas AdamW uses the practical fixed learning rate; the table is therefore intended to compare the trend with

k

, not the absolute performance of the two optimizers.

Figure 5: How realization depth ranks depends on what is held fixed. Swiss-roll sliced-Wasserstein distance under four budget definitions. Ranking the same runs by outer iterations, optimizer micro-steps, FLOPs and wall-clock gives different orderings, which is why the ImageNet comparison reports measured elapsed time for every arm.

$k$	$\rho_{k}$	$\cos$	$m_{k}$	$a_{k}$	$G_{\parallel,k}$	$P_{\perp,k}$	$1-\rho_{k}^{2}$
1	0.999505	0.03266	0.04146	0.001354	0.002706	0.001717	$+0.000989$
2	1.000303	0.03446	0.07681	0.002647	0.005287	0.005893	$-0.000606$
4	0.998774	0.05935	0.09208	0.005465	0.010900	0.008449	$+0.002451$
8	0.995368	0.09702	0.11004	0.010676	0.021238	0.011995	$+0.009243$
16	0.988698	0.15148	0.12983	0.019666	0.038945	0.016468	$+0.022477$
32	0.981388	0.19359	0.16914	0.032744	0.064415	0.027537	$+0.036878$

Appendix

Table 4: Full decomposition at checkpoint

30

B_{\mathrm{eff}}=64

(the data behind Fig. 1 b).

cell	$\cos(T_{0},T_{1}^{\mathrm{same}})$	$\cos(T_{0},T_{1}^{\mathrm{new}})$	refresh gain	refresh wins
$N=64$ , 30k	0.9790	0.3300	0.474%	8/8
$N=128$ , 30k	0.9780	0.4463	0.472%	8/8
$N=256$ , 30k	0.9767	0.5456	0.482%	8/8
$N=256$ , 20k	0.9704	0.5466	0.678%	8/8
$N=256$ , 10k	0.9476	0.5526	1.321%	8/8
all 40	0.9704	0.4842	0.685% (median 0.562%)	40/40

Appendix

Table 5: Local refresh/realize probes, cell means over eight probes each.

slice	field	floor active	cosine	rel. change	norm ratio
0	$\tau=0.2$	1.27%	0.99976	0.022	1.000
	$\tau=0.05$	49.83%	0.99121	0.133	1.000
	$\tau=0.02$	90.42%	0.96938	0.247	1.000
	aggregated	—	0.98930	0.147	0.975
1	$\tau=0.2$	1.28%	0.99968	0.025	1.000
	$\tau=0.05$	48.49%	0.99583	0.091	1.000

Appendix

Table 6: Affinity-floor diagnostic per slice and pooled. Relative change is

\left\lVert V_{\mathrm{clip}}-V_{\mathrm{noclip}}\right\rVert/\left\lVert V_{\mathrm{clip}}\right\rVert

; norm ratio is

\left\lVert V_{\mathrm{noclip}}\right\rVert/\left\lVert V_{\mathrm{clip}}\right\rVert

. Every temperature branch keeps unit norm ratio because it is RMS-normalized; the change is in direction.

seed	$N$	eval 0	eval 1	eval 2	mean
42	32	$-0.0941$	$-0.0341$	$-0.0455$	$-0.0579$
42	64	$-0.0943$	$-0.0856$	$+0.0524$	$-0.0425$
42	128	$-0.1405$	$+0.1000$	$-0.0690$	$-0.0365$
43	32	$-0.0484$	$+0.0143$	$-0.0899$	$-0.0413$
43	64	$-0.0500$	$-0.0489$	$+0.0026$	$-0.0321$
43	128	$-0.0732$	$+0.0154$	$-0.0362$	$-0.0313$

Appendix

Table 7: Paired

\mathrm{FID}(\beta^{\ast})-\mathrm{FID}(0)

for each evaluation seed.

seed	$N$	$\beta=0$	$\beta=1$	$\beta^{\ast}$	$\beta^{\ast}-\beta{=}1$
42	32	8.0228	7.9947	7.9649	$-0.0298$
43	32	8.0661	8.0434	8.0248	$-0.0186$
42	128	7.5843	7.4728	7.5478	$+0.0750$
43	128	7.6194	7.6329	7.5881	$-0.0448$

Appendix

Table 8: Calibrated

\beta^{\ast}

versus the parameter-free

\beta=1

(FID averaged over three evaluation seeds). The mean difference is

-0.0046

and the sign disagrees across seeds at

N=128

Drifting Models

Drifting models learn transport fields that move generated samples toward a target distribution, often enabling one-step generation.

See all 53 papers

May 12, 2026

DriftXpress: Faster Drifting Models via Projected RKHS Fields

Drifting Models have emerged as a new paradigm for one-step generative modeling, achieving strong image quality without iterative inference. The premise is to replace the iterative denoising process in diffusion models with a single evaluation of a generator. However, this creates a different trade-off: drifting reduces inference cost by moving much of the computation into training. We introduce DriftXpress, an accelerated formulation of drifting models based on projected RKHS fields. DriftXpress approximates the drifting kernel in a low-rank feature space. This preserves the attraction-repulsion structure of the original drifting field while reducing the cost of field evaluation. Across image-generation benchmarks, DriftXpress achieves comparable FID to standard drifting while reducing wall-clock training cost. These results show that the training-inference trade-off of drifting models can be pushed further without giving up their one-step inference advantage.

Ali Falahati, Elliot Creager, Gautam Kamath +1

Cheriton School of Computer Science, University of Waterloo and Vector Institute. · Department of Electrical and Computer Engineering, University of Waterloo and Vector Institute. · A*STAR Centre for Frontier AI Research, Singapore.

Image Generation One-Step Generative Modeling Kernel Methods Drifting Models

Ask PDF

Oct 3, 2026

Learning Discriminative Geometry for Drifting Models

Recently proposed Drifting Models shift iterative distribution refinement from inference to training, enabling effective one-step generation. However, their performance on complex image datasets depends strongly on the representation used to construct the drifting field: pixel-space drifting performs poorly, whereas pretrained feature spaces substantially improve sample quality for reasons that remain unclear. We trace this gap to the discriminative geometry of the representation, which determines sample weighting in kernel density estimation (KDE) and, consequently drift. We introduce persistent representation learning, which continuously learns a more discriminative representation geometry as the generator evolves across batches. We further establish a current-step gradient equivalence between the KDE ratio loss and drift regression loss under matched conditions, connecting density-ratio-based generator optimization to empirical drifting and motivating direct control of the drifting velocity. Across multiple datasets, our method learns effective discriminative representations directly from pixels and reduces FID by approximately

82-95\%

over the original pixel-space Drifting Models, without pretrained encoders. Adapting pretrained representations and applying velocity clipping provide further gains.

Doudou Zhang, Wenwen Hou, Yilin Chen +1

ELLIS Institute Finland · Aalto University

Drifting Models Image Generation Representation Learning

Ask PDF

Apr 10, 2026

Lookahead Drifting Model

Recently, a new paradigm named drifting model has been proposed for mapping distributions, which achieves the SOTA image generation performance over ImageNet via one-step neural functional evaluation (NFE). The basic idea is to compute a drifting term at each training iteration and then push the output of the model towards the direction of the drifting term. In this paper, we propose a lookahead drifting model. At each training iteration, we compute a set of drifting terms sequentially. Each drifting term is calculated by making use of previously computed ones as well as the positive samples and the output of the model. %One key step is to properly scale the drifting terms so that their magnitudes are in a comparable range. In principle, the drifting terms obtained at a later stage capture higher order gradient information towards the positive samples. At each training iteration, the model is optimized by pushing its output towards the direction of the (weighted) summation of the drifting terms. Experimental results on toy examples and CIFAR10 demonstrate the better performance of the new method than the baseline.

Guoqiang Zhang, Kenta Niwa, W. Bastiaan Kleijn

University of Exeter · NTT Communication Science Laboratories · Victoria University of Wellington

Drifting Models Image Generation One-Step Generative Modeling

Ask PDF

Abstract

Drifting Models

Explore similar work

DriftXpress: Faster Drifting Models via Projected RKHS Fields

Learning Discriminative Geometry for Drifting Models

Lookahead Drifting Model