The Limits of Closed-Form Couplings for Few-Step Generation
Authors: Tobias Höppe, Stefan Bauer, Qiang Liu, Andrea Dittadi, Kirill Neklyudov
Organizations: Technical University of Munich · Helmholtz AI · Munich Center for Machine Learning · University of Texas at Austin · Mila – Quebec AI Institute · Université de Montréal · Institut Courtois
Few-step generators via flow models require complicated technical solutions such as a pretrained flow model or a consistency loss. While rectified flows have gained popularity for their success in few-step sampling, they pay a cost of needing a pre-trained model and sampling a large number of couplings per dataset. Yet for a finite dataset and a Gaussian probability path, the probability-flow vector field of the empirical distribution is available in closed form, so one could use this field as the teacher and skip pretraining altogether. We demonstrate that, surprisingly, this approach is impossible for problems of practical relevance. In particular, the analytic solution of the diffusion process in high dimensions becomes almost a linear transformation. The integration returns the noise vector drawn at initialisation, plus a drift that is empirically a faint copy of the data point, almost independent of the noise. The coupling therefore differs from an independent pairing only through this drift. As a result, training a model on these couplings is almost equivalent to training the model on the independent draws of the noise and the data, which does not provide a useful teacher. We highlight that this is the direct consequence of the high-dimensionality of the data, and that for low enough dimensions or large enough datasets, analytic couplings become more efficient teachers. We quantify this effect by the fraction of endpoint energy carried by the drift, which scales as dlogN for a dataset of size N in a d-dimensional space. Finally, we provide an empirical study confirming all the theoretical results presented in the paper.
Figures & tables
Figure 1: FID against sampling steps for students trained on πPF , πFM and πRandom . For a fair comparison, each model is trained exactly on the N forward couplings permitted by the datasets. This limits the FID value, but enables us to provide a clean comparison between couplings. Although the gap narrows as the number of integration steps increases, it is substantial in the few-step regime.
Figure 2: Endpoint decomposition of an ImageNet-64 integration ( λ=1 , ϵ=10−3 , closed-form velocity against all 1.28 M training images). The endpoint x1 looks like pure Gaussian noise. Subtracting the injected noise λz reveals the drift Δ(xϵ) , a faint copy of the datapoint that keeps most of its structure. The drift is well aligned with the data, with cos(Δ(xϵ),x0)=0.88 for this image ( 0.86 on average). The drift of the noise-free PF-ODE (λ=0) integration, Δ(αϵx0) , is almost identical to it, with cos(Δ(xϵ),Δ(αϵx0))=0.96 ( 0.95 on average). The injected noise therefore barely changes the information the drift carries about x0 . The datapoint x0 on the right is shown for reference. All panels use the same colour scale.
Figure 3: Closed-form couplings against the dimension. Left: Share of the endpoint’s squared norm carried by the drift, E∥Δ(xϵ)∥2/E∥x1∥2 with Δ(xϵ)=x1−z . The dashed lines show d∥Δ(xϵ)∥2 with the drift norm held fixed at its measured value. Each dataset follows this scaling closely. Right: Gap score G ( Equation 14 ) of the closed-form students at 1 and 2 Euler steps. G=0 means the closed-form student is no better than the random one, and G=1 means it matches the student trained on teacher couplings. πPF improves as the dimension shrinks on all datasets, but does not reach πFM . Note that G is not comparable across datasets. On ImageNet, the random student cannot fit its 1.28 M random training pairs, which inflates G ( Appendix H ).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Many-to-one structure of the closed-form probability-flow ODE. Left: An empirical two-dimensional dataset p^0 , with each colour denoting one data point. Right: The corresponding partition of the Gaussian prior space p1 induced by the closed-form PF-ODE . Each coloured region contains all prior points that are mapped back to the same data point when the flow is integrated in reverse. The matching colours indicate the associated data point and region. Stars mark the endpoints obtained by forward integration from the conditional mean, xϵ=αϵx0 ( λ=0 ). Thus, a single data point corresponds to an entire region of prior space rather than a unique endpoint.
Figure 5: Prior-space partition of a symmetric dataset. Left: Eight equally spaced data points on a circle, which satisfy the assumptions of Propositions A.1 and A.2 . Right: The partition of the prior induced by the closed-form PF-ODE , computed as in Figure 4 . The regions coincide with the Voronoi cones of the data (dashed lines), as predicted by Proposition A.1 . The noise-free endpoints ( λ=0 , stars) lie on the rays through their data points (dotted lines), i.e. they are rescaled copies of the data points, as predicted by Proposition A.2 .
Figure 6: Noisy versus noise-free initialisation on ImageNet-64 ( ϵ=10−3 ). The two initial conditions differ only by σϵz and are visually indistinguishable. Their endpoints, however, differ completely. Starting from the noisy initialisation (a) produces endpoints that resemble Gaussian noise, whereas starting from the conditional mean (b) produces faint, low-contrast copies of the data points, from which each pair can be identified by eye.
Figure 7: Visualising the endpoint decomposition on ImageNet-64 ( λ=1 , ϵ=10−3 ). The endpoints x1 resemble pure Gaussian noise, yet subtracting the injected noise λz reveals the drift Δ(xϵ) , a faint, structure-preserving copy of each data point that closely matches the PF-ODE (λ=0) endpoint Δ(αϵx0) of a noise-free integration. The two drifts differ mainly in magnitude, with Δ(xϵ)≈1.2Δ(αϵx0) , while their directions agree closely, with an average cosine similarity of 0.95 ( Table 2 ).
dataset
coupling
noise share
cos(x1,x1′)
NN match
CIFAR-10
teacher
6.1%
0.94
100%
closed-form
99.6%
0.004
0.05%
CelebA
teacher
8.2%
0.92
100%
closed-form
99.9%
0.001
0.05%
ImageNet
teacher
4.3%
0.96
100%
closed-form
99.8%
0.002
0.03%
Appendix
Table 1: Teacher couplings select a narrow set of endpoints within each region. For the first 2000 training images on each data point, the endpoints from three independent noise draws ( λ=1 , ϵ=10−3 ) are integrated with 250 Euler steps using vFM and vPF . Then, we measure the noise share, i.e. fraction of the endpoint variance explained by the injected noise, the cosine similarity between two endpoints of the same data point and fraction of endpoints whose nearest neighbour among the endpoints of another noise draw belongs to the same data point (chance level 0.05% ).
Figure 8: Evaluation of the drift bound. We evaluate both terms of Proposition 5.2 on all three datasets for different values of the cutoff τ , using the entire training set. The bound becomes increasingly tight as τ increases, showing that Proposition 5.2 provides a practical bound on the drift of the PF-ODE trajectory. Furthermore, the tracking error ∥rτ∥L2 remains negligible for small τ , supporting the assumption in Equation 12 .
Figure 9: Empirical validation of the scaling argument. Left: The posterior collapse time τ as the data dimension is varied by downsampling. Dashed lines show the prediction (στατ)2V=ClogN from Equation 11 , with a single constant C fitted per dataset, which closely tracks the measured transition across CIFAR-10, CelebA and ImageNet-64. Right: Drift magnitude as a function of the dataset size N . Solid lines show fits of the form E∥Δ(xϵ)∥=alogN+b , confirming the logN scaling predicted by Equation 12 . Dashed lines extrapolate the fits beyond the full training set, and error bars are 95% confidence intervals.
dataset
px
d
64 steps
128 steps
CIFAR-10
16
768
2.6
2.1
CIFAR-10
24
1728
3.4
3.1
CIFAR-10
32
3072
3.6
3.1
CelebA
16
768
1.5
1.2
CelebA
32
3072
2.6
2.7
CelebA
64
12288
4.4
3.4
Appendix
Table 3: FID of the flow-matching teachers used to generate πFM . FID of the best checkpoint at 64 and 128 Euler steps.
πRandom
πPF
πFM
dataset
px
1 step
2 steps
1 step
2 steps
1 step
2 steps
CIFAR-10
16
61.0 ± 1.9
38.1 ± 0.7
31.3 ± 1.2
20.2 ± 1.1
4.6 ± 0.0
3.9 ± 0.0
CIFAR-10
24
72.7 ± 0.9
49.1 ± 0.6
51.5 ± 1.5
36.5 ± 1.1
9.3 ± 0.2
8.3 ± 0.3
CIFAR-10
32
82.9 ± 0.7
60.7 ± 6.1
68.0 ± 2.3
46.5 ± 2.5
15.2 ± 0.5
12.4 ± 0.3
CelebA
16
38.3 ± 1.0
29.2 ± 0.8
21.3 ± 0.1
14.7 ± 0.1
2.0 ± 0.1
2.0 ± 0.0
CelebA
32
43.1 ± 2.1
29.6 ± 1.1
29.1 ± 1.8
20.2 ± 0.7
4.8 ± 0.0
6.4 ± 0.2
Appendix
Table 4: FID of the students presented in Figure 3 . One- and two-step FID of the students trained on πRandom , πPF and πFM , from which the gap score G ( Equation 14 ) is computed. For CIFAR-10 and CelebA we report mean ± standard deviation over three seeds.
dataset
px
d
πRandom
πPF
πFM
cos(Δ(xϵ),x0)
CIFAR-10
16
768
0.000
0.120
0.335
0.852
CIFAR-10
24
1728
0.000
0.082
0.284
0.861
CIFAR-10
32
3072
0.000
0.062
0.230
0.867
CelebA
16
768
0.000
0.103
0.242
0.703
CelebA
32
3072
0.000
0.054
0.171
0.730
CelebA
64
12288
0.000
0.028
0.108
0.746
Appendix
Table 5: Alignment between data points and their coupled endpoints. Cosine similarity cos(x0,x1) for the three couplings, averaged over data points. The last column is the alignment of the closed-form drift with its data point, cos(Δ(xϵ),x0) .
πRandom
πPF
πFM
dataset
px
d
train
test
train
test
train
test
CIFAR-10
16
768
0.01
0.00
0.54
0.52
0.86
0.85
CIFAR-10
24
1728
0.03
-0.01
0.52
0.48
0.84
0.82
CIFAR-10
32
3072
0.05
-0.01
0.52
0.45
0.81
0.78
CelebA
16
768
0.00
0.00
0.60
0.60
0.84
0.84
CelebA
32
3072
0.02
-0.01
0.55
0.53
0.79
0.78
Appendix
Table 6: Linear read-out of the data point from the coupled endpoint. Held-out and training R2 of a ridge regression x0−xˉ≈W⊤x1 , i.e. the fraction of the data variance a linear map recovers from the endpoint x1 (0: predicts the mean image, 1: exact recovery). Fitted on the first 40000 (CIFAR-10) or 100000 (CelebA, ImageNet) pairs, tested on the next 10000 .
πRandom
πPF
πFM
dataset
px
d
train
test
train
test
train
test
CIFAR-10
16
768
0.96
− 0.54
0.97
0.44
1.00
0.98
CIFAR-10
24
1728
0.95
− 0.51
0.96
0.40
1.00
0.96
CIFAR-10
32
3072
0.92
− 0.59
0.92
0.33
0.99
0.92
CelebA
16
768
0.95
− 0.52
0.97
0.61
1.00
0.99
CelebA
32
3072
0.89
− 0.50
0.92
0.52
1.00
0.98
Appendix
Table 7: Reconstruction of held-out images. R2=1−E∥x^0−x0∥2/V of the one-step student output x^0 against the paired image, on the student’s own training couplings (train) and on couplings built the same way for held-out images (test). V is the variance of the training images, so R2=0 is the accuracy of predicting the mean image. Test couplings of πPF integrate the closed-form field of the training images together with the held-out images.
πPF
πFM
dataset
px
d
directions
variance
directions
variance
CIFAR-10
16
768
5
58%
238
98%
CIFAR-10
24
1728
3
48%
174
94%
CIFAR-10
32
3072
3
47%
154
85%
CelebA
16
768
6
70%
254
97%
CelebA
32
3072
6
64%
425
89%
Appendix
Table 8: Strong directional readability is concentrated in a few leading directions for closed-form couplings. We report the number of principal directions with ρk2>21 and the share of data variance they hold, for closed-form and teacher endpoints. This threshold selects directions in which a linear predictor using the corresponding endpoint coordinate recovers more than half of the component’s variance.
πRandom
πPF
πFM
G
dataset
pairs
1 step
2 steps
1 step
2 steps
1 step
2 steps
1 step
2 steps
ImageNet-16
1.28 M
430.1
158.0
97.5
44.2
2.3
1.6
0.78
0.73
ImageNet-16
50 k
53.8
28.2
35.3
25.1
12.1
11.6
0.44
0.18
Appendix
Table 9: Controlling for the number of pairs. ImageNet-16 students trained on all pairs ( 80 epochs) and on the first 50 k pairs ( 2048 epochs), with the same network, batch size and number of steps (best checkpoint).
πRandom
πPF
πFM
G
d
latent
rFID
1 step
2 steps
1 step
2 steps
1 step
2 steps
1 step
2 steps
256
4×8×8
0.95
69.3
41.7
43.8
32.4
17.7
17.7
0.49
0.39
512
8×8×8
0.55
71.1
40.3
49.4
34.9
20.8
20.9
0.43
0.28
1024
16×8×8
0.37
71.2
56.0
64.8
48.2
24.1
23.1
0.14
0.24
Appendix
Table 10: Dimension reduction with a learned autoencoder (CIFAR-10). One- and two-step FID of the students trained on πRandom , πPF and πFM in the latent space of the autoencoder (mean over three seeds), and the resulting gap score G ( Equation 14 ). rFID is the FID of the autoencoder’s reconstructions of the training set.
px
d
width
parameters
per dimension
πRandom
πPF
πFM
G
32
3072
128
56.4M
18.4k
85.1
69.2
14.7
0.23
24
1728
128
56.4M
32.6k
74.3
50.8
9.5
0.36
24
1728
96
31.7M
18.4k
84.8
58.5
9.1
0.35
16
768
128
56.4M
73.4k
62.9
33.0
4.7
0.51
16
768
64
14.1M
18.4k
93.9
52.1
4.7
0.47
Appendix
Table 11: Students at equal capacity per dimension (CIFAR-10). One-step FID of the students trained on πRandom , πPF and πFM with the base width of the network either fixed ( 128 ) or scaled with d , and the resulting gap score G ( Equation 14 ).
dataset
px
d
πFM
πPF
CIFAR-10
16
768
2.1%
98.2%
CIFAR-10
24
1728
2.7%
99.2%
CIFAR-10
32
3072
6.1%
99.6%
CelebA
16
768
1.2%
98.2%
CelebA
32
3072
3.8%
99.6%
CelebA
64
12288
8.2%
99.9%
Appendix
Table 12: Noise share across resolutions. Fraction of the endpoint variance explained by the injected noise for teacher and closed-form couplings, measured as in Table 1 on the first 2000 training images with three independent noise draws ( λ=1 , ϵ=10−3 ).
Flow-based generative modeling learns probability transports from simple noise to data for fast sampling, density estimation, and controllable generation.