Modern generative models based on diffusion or flow matching transport noise toward the data manifold, without representing the data manifold explicitly. We propose Distance Marching that explores an alternative design by explicitly modeling a distance field of the data manifold. A distance field offers a geometric view of generation as a descent process toward the data manifold using only the current position. The negative gradient of the distance field provides the direction, and the distance value can optionally determine the step size. However, sparse training data in high-dimensional space make it difficult to identify the closest points on the data manifold needed to train the distance field. To address this challenge, we design new loss functions that emphasize denoising targets from closer data points. Our analysis shows that, without time input, emphasizing closer targets reduces denoising ambiguity and the bias toward the data mean. This distance-field perspective naturally motivates two principled deterministic samplers, showing fast convergence in inference. As a result, our model is the only one without time input that remains competitive with time-conditioned models across datasets and architectures. Moreover, our distance prediction is also helpful for early stopping during sampling and for out-of-distribution detection. We hope distance-field modeling can open a promising direction for exploring generative modeling from a geometric perspective.
Figures & tables
Figure 1: We propose Distance Marching to generate images without time input. Inspired by distance-field modeling, our method learns a denoising direction that focuses on closer targets. In contrast, the flow matching loss favors a direction biased toward the data mean in the early stages of generation. We compare outputs after 20% of the sampling steps to show that our sampling trajectories wander less. See Sec. 4 for a detailed analysis.
Figure 2: In 2D, we learn a distance-like scalar field uθ(x) ( a and b ) that transports samples onto the support of the data distribution ( a ). In (a–d), darker shading indicates larger scalar values, with each field normalized separately. In contrast, prior energy-based methods ( c , d ), even when using minibatch nearest-neighbor during training to disambiguate targets, do not capture energy landscape details. Because they inherit the flow-matching loss, when rematching is impractical due to high-dimensional hubness, they become unreliable for determining denoising directions, as analyzed in Sec. 4 . See Figs. 13 and C.1 for quantitative sampling comparisons.
Figure 3: 1D d^(x) to the origin; lighter line, larger c0 .
Figure 4: The leftmost image alone accounts for 62.2% of nearest-neighbor selections due to its grayness and low contrast.
Figure 5
Figure 7: With an extra condition on the interpolation coefficient, the directions become meaningful and consistent. Both panels use the same coordinate scale and a common arrow scale.
Figure 10: Denoising paths.
(a) The pairwise length factor ℓ varies widely, acting as a random reweighting in denoising.
(b) For a fixed noise sample x , distances from real images to x also exhibit high variance.
Figure 10
Figure 14: With step sizes tuned for best 250-step performance, gradient descent and sphere tracing produce perceptually better images earlier than flow matching (SiT+dopri5). We reach flow matching’s final FID within <60% of the steps. FID uses no CFG (B/2 base); sequences use CFG =4 (XL/2 base, every 10% steps). See Sec. C.4 for more details.
Figure 12
Figure 16: CIFAR-10 geometry in a PCA plane. Across 800 queries from 32 seeds, the scalar-field gradient and vector head are closely aligned in the PCA plane, with a mean projected angle discrepancy of only 8.90∘ .
Method
η
Avg. Steps ↓
FID ↓
sFID ↓
IS ↑
original GD
0.64
250.0
32.16
7.52
45.16
adaptive GD
0.64
175.8
32.73
6.70
45.86
Table 4: Adaptive stopping for gradient descent sampling using uθ on ImageNet generation.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 17: Distance predictor architecture.
Metric
SiT
EqM
Ours
Training throughput (steps/sec)
2.19
2.18
2.18
Inference time w/ CFG (sec)
83.36
83.30
84.32
Appendix
Table 6: Training and CFG inference speed comparison on ImageNet. For training, we use 4 Nvidia A100 (80G) GPUs. For inference, we use one Nvidia A10 to generate 64 images in one batch for 250 steps.
Setting
Value
U-Net Architecture
input size
3×32×32
base channels
128
channel multipliers
[2, 2, 2]
ResBlocks per level
4
attention resolutions
[2]
Appendix
Table 7: Configuration for the CIFAR-10 Distance Marching run. U-Net architecture entries follow the CIFAR-10 preset used in the codebase [ 30 ] . Our losses in Eq. 2 and Eq. 5 supervise a quasi-normalized denoising direction for vθ , which induces a different target scale than flow matching. We therefore rescale the network output vθ to align with the loss-defined magnitude.
model
S/2
B/2
L/2
XL/2
SiT Architecture
latent space size
4×32×32
params (M)
33
130
458
675
depth
12
12
24
28
hidden dim
384
768
1024
1152
patch size
2
2
2
2
Appendix
Table 8: Configuration for the ImageNet 256×256 Distance Marching run. Transformer architecture entries follow the SiT preset used in the codebase [ 51 ] .
Figure 18: Energy Matching fails to capture fine-grained energy variations near the data manifold and is inconsistent across two locations that are equally distant from the data manifold.
Figure 19: Equilibrium Matching [ 7 ] learns 4× flow matching denoising vectors from source to target, with a decaying coefficient when t>0.8 ; for visualization clarity, we divide the negative gradient vectors by 4 when rendering. Equilibrium Matching misses fine-grained energy structure near the data manifold and is inconsistent across two locations that are equally distant from the data manifold.
Figure 20: We compare different generation methods on the Equilibrium Matching model, but none of them yields competitive results on this task. Steps are recorded as: init (black), mid (yellow), and final (blue). 1,024 samples for visualizations.
Table 9: 2D point-cloud distance metrics of Fig. 20 between 10 k generated samples and 10 k target two-moons samples for the 8-Gaussians → two-moons task (lower is better). We report the original gradient descent as their best in Fig. 13 .
Figure 21: In our 2D toy model, we use a single sphere tracing step to jump to the target region; then Hamiltonian Monte Carlo explores low-potential areas. Final samples are presented in Fig. 2 . We achieve the lowest Chamfer and Hausdorff distances in Fig. 13 with a meaningful energy landscape. 1,024 samples for visualizations.
Figure 22: We use LDA to visualize the 8D initial distribution ( blue ) and the target distribution ( orange ) in 2D. The data mean of the target distribution is marked using the orange star.
Figure 23: Flow matching losses have minimizers ( green reweighted, red original) that are strongly biased toward the data mean ( orange star). In contrast, one-step loss ( orange ) and directional eikonal loss ( blue ) yield minimizers pointing to closer target data, less affected by the data mean. Global and zoom perspectives shown.
Figure 24: Flow matching produces more curved trajectories and yields slower improvement in target likelihood than our minimizers. We visualize target log-density along the path and report a Mahalanobis-based negative log p-value as a scale-robust outlierness score.
Figure 25: Our one-step loss provides robust guidance across step sizes. It converges to near-zero outlierness over a 4× range of step sizes.
Figure 26: None of the other losses provides reliable guidance over a 4× range of step sizes, as OSL does. Red : FM loss; Green : (1−t)−2 -reweighted FM loss; Blue : directional eikonal loss.
Figure 27: Uncurated samples for time- and class-unconditional generation on CIFAR-10. Distance Marching achieves the lowest FID among time-unconditional methods on CIFAR-10.
Figure 28: Curated ImageNet samples generated by our XL model (patch size 2 ) with CFG scale 4.0 . We apply the CFG formula [ 18 ] on vθ in gradient descent and uθvθ in sphere tracing.
Figure 29: We run the original SiT model [ 51 ] with a flow matching setting using their recommended ODE solver dopri5 for 250 steps, matching our step number, and report the curves of inception score as a supplement of Fig. 14 .
Figure 30: Our model can move samples initialized from uniform noise in [−1,1] , despite the mismatch with the Gaussian noise distribution used during training, toward the image manifold. Individual outputs are visually consistent with the low-contrast subset of CIFAR-10. However, without MCMC, this initialization does not guarantee diverse outputs that cover the full image manifold. Both methods use similar model architectures and the same sampling budget.
Figure 31: With MCMC, poor initializations (uniform distribution in [−1,1] ) can be spread more effectively over the image manifold, producing more diverse categories. Under the same sampling budget, the FID improves from 65.48 to 25.73 and then to 11.37.
Figure 32: MCMC provides limited visual improvements under a fixed 250-step sampling budget when initialized from either the uniform distribution over [−1.7,1.7] or the standard Gaussian distribution.
Method
1.25
1.5
1.75
2
2.25
2.5
2.75
3
4
SiT
0.9019
0.9488
0.9731
0.9858
0.9924
0.9940
0.9968
0.9971
0.9949
EqM
0.9311
0.9659
0.9850
0.9917
0.9943
0.9966
0.9972
0.9973
0.9963
Ours (Sphere Tracing)
0.9524
0.9799
0.9930
0.9937
0.9965
0.9974
0.9975
0.9977
0.9961
Ours (Gradient Descent)
0.9545
0.9810
0.9930
0.9937
0.9964
0.9974
0.9974
0.9980
0.9960
Appendix
Table 12: Coverage across guidance scales. Higher is better. Bold marks the best result at each scale, including ties.
Method
1.25
1.5
1.75
2
2.25
2.5
2.75
3
4
SiT
0.6921
0.8657
1.0231
1.1570
1.2571
1.3328
1.3972
1.4443
1.5366
EqM
0.7682
0.9454
1.0994
1.2231
1.3152
1.3955
1.4509
1.4913
1.5513
Ours (Sphere Tracing)
0.8270
1.0516
1.2264
1.3603
1.4487
1.5112
1.5501
1.5759
1.5838
Ours (Gradient Descent)
0.8246
1.0443
1.2161
1.3461
1.4330
1.4955
1.5272
1.5507
1.5586
Appendix
Table 13: Density across guidance scales. Higher is better. Bold marks the best result at each scale.
Figure 33: Nearest-neighbor analysis of the sphere tracing method in Fig. 28 . Each group shows a generated sample and its top-3 nearest training images by image-space ℓ2 distance. The closest distance is bolded. The large distances and visually distinct content confirm that our model does not memorize training data.
Figure 34: Distance estimation MAPE. Blue: uθ vs. ∥x−s∥22+c0 . Orange: ∥uθvθ∥2 vs. ∥x−s∥2 . Shaded: ±1 standard deviation within each bin.
Figure 35: Generation trajectory in the PCA plane for CIFAR-10. The blue curve traces generation from the Gaussian initialization to the orange endpoint. Thumbnails show the actual images every 50 steps. Gray lines connect them to their projected positions. The inset magnifies the final steps. Both views use equal axis scales.
Figure 36: Nearby inputs share a common denoising direction in the PCA plane. (a) Gray contours show the predicted distance uθ . Thumbnails at nine evenly spaced locations show the corresponding image-space inputs, clipped to the display range. (b) We initialize 25 trajectories on a 5×5 grid and run sphere tracing in the full image space, then project the resulting states onto the same plane. The nine trajectories starting at the thumbnail locations in (a) are blue, with open start markers and orange endpoints. The remaining trajectories are gray. Both panels use the same coordinates as in Fig. 35 .
Figure 37: Local geometry of the learned distance field on CIFAR-10. (a) Gray contours show uθ in the same PCA plane as in Figs. 35 and 36 . The black curve shows the same generation trajectory, from the filled marker to the open marker. Blue solid arrows show −PT∇uθ and orange dashed arrows show −PTvθ , each normalized to the same display length. The axes have equal scale to preserve angles. (b) Histogram of the projected angles across all 800 sample points from 32 seeds, with 5∘ bins. The dashed vertical line marks the mean angle of 8.90∘ . The PC1 sign agreement is 98.75%, and PC1 explains 99.40% of trajectory variance on average.
Figure 38: During ImageNet generation, the distance predicted by uθ , as smoothed by Eq. 4 , converges to the nearby c0 as predicted by our analysis. We sample the same 64 Gaussian noise vectors and run gradient descent and sphere tracing in parallel; the bold lines show their mean. Gradient descent provides a linear descent pattern, while sphere tracing shows an exponential descent.
Figure 39: In the first 40%, gradient descent maintains a steady denoising pace and then rapidly converges to zero modification. Sphere tracing still follows an exponential pattern and has the largest modifications at the beginning, consistent with our analysis in Sec. C.4 .
Figure 40: Distance Marching is robust to step-size choices. We plot results for our backbone SiT [ 51 ] and the second-best EqM [ 7 ] in gray as references, using the same base model size, training epochs, and evaluation protocol (250 steps, 50k samples). The best step size is highlighted with an additional circle.
Method
Inception Score ↑
FID ↓
sFID ↓
no coupling
41.77
71.23
206.2
w/ OT coupling
36.96
77.19
208.2
w/ minibatch closest coupling
26.31
90.79
217.6
Appendix
Table 14: Quick test on ImageNet generation with 1024 samples. We keep sphere-tracing sampling unchanged (same step budget) and only vary the training-time matching on a B/2 SiT model with 80 epochs: random linear interpolation as in Def. 3.1 , OT coupling [ 31 ] (i.e., nearest-target matching without replacement), or minibatch closest coupling (i.e., nearest-target matching within each minibatch with replacement).
Figure 41: Comparison of generative models trained by different rematching algorithms, highlighting both condition mismatch and mode collapse: ( b ) OT coupling introduces more unnatural artifacts in conditional generation; ( c ) minibatch closest coupling makes the outputs noticeably low-contrast and grayish. We keep CFG scale =1 (i.e., no CFG) to align with other experiments.
Sampler
w/ DEL
w/o DEL
Sphere tracing
2.23
2.33
Gradient descent
2.54
2.38
Appendix
Table 15: CIFAR-10 FID with and without directional eikonal loss. Sphere tracing achieves lower FID than gradient descent both with and without DEL. All results use 50k generated samples and 250 sampling steps. We tune η for each method and report the lowest FID achieved.
Dataset
w/ DEL
w/o DEL
CIFAR-10
2.23
2.33
ImageNet
33.44
37.01
Appendix
Table 16: FID under sphere tracing with and without directional eikonal loss, using the evaluation protocol in Tab. 15 . Adding directional eikonal loss improves FID on both datasets, with a larger reduction on ImageNet.
Method
FID ↓ (ST)
FID ↓ (GD)
original OSL, c0=ϵ=4 (baseline)
2.23
2.54
original OSL, c0=ϵ=0
2.37
3.84
unnormalized OSL w/o denominator, c0=4
3.88
4.63
Appendix
Table 17: Ablation study on CIFAR-10 of different loss designs. We tune η for each method and report the lowest FID achieved.
Figure 42: We perform 250-step generation using SiT-B/2 models trained for 80 epochs under their respective losses, starting from the same initial noise at inference. For each method, the step size is chosen to maximize the final performance after 250 steps. We visualize intermediate outputs at 10%–100% progress in 10% increments.
Figure 43: We bin the interpolation coefficient t∈[0,1] uniformly; in each bin, we draw 50k Gaussian noise samples, form linear interpolations, and select their nearest neighbors in the training set, and then compute the coverage. Coverage rate changes abruptly in the middle, remaining near 0 on the noisy side and saturating near 1 on the data side, indicating a highly non-uniform nearest-neighbor visitation across the interpolation.
Figure 44: Full FID curves across diffusion time. We partition t∈[0.0,0.95] into 20 uniform bins (same bin count as Fig. 9 , which is restricted to t∈[0.0,0.25] ). For each bin, we sample T uniformly within the bin and generate 2048 noisy samples following Def. 3.1 ; FID is computed on the corresponding denoised outputs for each objective. We exclude directional eikonal loss from this comparison because it does not specify a denoising destination; instead, we report in Fig. 9 that its solution lies between the one-step loss minimizer and the flow matching minimizer. Beyond the small- t regime, the performance gaps shrink, and the FID curves across objectives become much less distinguishable. The red and green curves always overlap so that they are hard to distinguish.
Flow Matching (FM) is a simulation-free method for learning a continuous, invertible flow that interpolates between two distributions, and in particular generates data from noise. Inspired by the variational nature of the diffusion process as a gradient flow, we introduce a stepwise FM model, Local Flow Matching (LFM), which sequentially learns a sequence of FM submodels, each matching a diffusion process up to the time-step size in the data-to-noise direction. In each step, the two distributions to be interpolated by the sub-flow model are closer than those in the full-flow matching model, which interpolates data to noise distributions, enabling smaller models with more efficient training. This variational perspective also allows us to prove a theoretical generation guarantee for the proposed flow model in terms of the χ2-divergence between the generated and true data distributions, leveraging the contraction property of the diffusion process. In practice, the stepwise structure of LFM is naturally amenable to model distillation, and various distillation techniques can be applied to accelerate generation. We empirically demonstrate that LFM achieves competitive generative performance compared to FM on unconditional generation of tabular and image datasets, and on conditional generation of robotic manipulation policies.
Chen Xu, Xiuyuan Cheng, Yao Xie
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology. · Department of Mathematics, Duke University
We propose the Monge Inception Distance (MIND), a metric for evaluating generative models that addresses key limitations of the widely adopted Fréchet Inception Distance (FID). The MIND metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. This approach circumvents the estimation of high-dimensional means and covariance matrices, which underlie FID's poor sample complexity and vulnerability to adversarial attacks. We empirically demonstrate three primary advantages: (i) it is more sample-efficient by one order of magnitude, (ii) it is faster to compute by two orders of magnitude, (iii) it is more robust to adversarial attacks such as moment-matching. We show that MIND with 5k samples can replace the evaluation performance of FID with 50k samples, providing high correlation with this standard benchmark and superior discriminative performance. We further demonstrate that even smaller sample sizes (e.g., 1k or 2k) remain highly informative for rapid model iteration.
We introduce Midpoint Generative Models (MGM), a principled framework for training one-step generative models. MGM is based on a simple symmetry of Flow Matching with linear interpolation: when the two endpoint distributions coincide, the corresponding drift field vanishes at the midpoint time, t=1/2. We show that the norm of this field defines a valid discrepancy between distributions, which we call the Midpoint Divergence. We extend this discrepancy beyond the midpoint by introducing randomly flipped interpolations and further generalize it by replacing deterministic linear Flow Matching interpolations with symmetric stochastic interpolants, yielding a generalized Midpoint Divergence. Finally, we derive a variational formulation of our generalized divergence, yielding a tractable objective for training a one-step generator. The resulting MGM algorithm offers an effective and theoretically grounded approach to generative modeling, achieving competitive performance against existing one-step generative modeling methods.
Daniil Shlenskii, Nikita Gushchin, Lev Novitskiy +2
1AXXX, Russia · 2Applied AI Institute, Russia · 3Kandinsky Lab, Russia