Regularized Wasserstein Distance for Discrete Generative Modeling and Clustering Analysis
Authors: Senmao Ye, Fei Liu, Dawei Zhang, Yuqian Zheng, Jun Zhang, Zhonglong zheng
Organizations: Zhejiang Normal University, Jinhua, 321004, Zhejiang, China · South China University of Technology, Guangzhou, 510006, Guangdong, China · Jinhua University of Vocational Technology, Jinhua, 321007, Zhejiang, China
In this paper, we formulate clustering analysis as discrete generative modeling. Specifically, when the generator is restricted to a finite candidate space, the generated samples naturally emerge as the most representative prototypes of the underlying data distribution. To model such a discrete candidate space, we employ a Discrete Distribution Network (DDN). However, DDN suffers from unstable training due to its hard nearest-neighbor selection. To stabilize training and encourage geometric consistency, we replace DDN's l2 loss between images with regularized Wasserstein distance between the empirical distributions of real and generated images. The resulting framework, termed Discrete Generative Clustering (DGC), offers four notable advantages: 1) the learned cluster centers are directly interpretable, as they are realistic synthetic samples; 2) the inference cost scales logarithmically with the number of clusters, i.e., O(log K), making the method scalable to extremely large K; 3) the regularized Wasserstein distance provides a smooth and well-behaved optimization objective that facilitates stable convergence, whereas DDN's l2 loss tends to converge to poor local optima; and 4) DGC replaces DDN's nearest-neighbor selection with random-neighbor selection, which avoids the density shift problem during multi-step forward. Extensive experiments on several benchmarks demonstrate the effectiveness and desirable properties of DGC. The code is publicly available at GitHub.
Figures & tables
Figure 1: Clustering analysis is a special case of generative modeling.
Figure 2: Illustration of the proposed Discrete Generative Clustering (DGC) framework. The purple components denote the generative modeling module, while the blue components correspond to the clustering module.
Dataset
Samples
Original size
Classes
MNIST
70,000
28×28
10
Fashion-MNIST
70,000
28×28
10
CIFAR-10
60,000
32×32
10
COIL20
1,440
128×128
20
Table 1: Summary of datasets used in experiments. Original size refers to the native resolution.
Method
MNIST
F-MNIST
CIFAR-10
COIL20
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
Classical clustering vs. DGC
K-means
53.24
0.499
51.08
0.392
23.60
0.102
47.33
0.480
GMM
53.72
0.501
52.34
0.409
21.47
0.113
48.16
0.477
HC
52.10
0.485
49.75
0.378
20.12
0.095
46.20
0.465
DBSCAN
48.33
0.452
45.60
0.341
18.45
0.088
42.11
0.430
Table 2: Clustering performance comparison.
Method
Cos. ↑
MSE ↓
PSNR (dB) ↑
Cos. wins
MSE wins
K-Means
0.9695
0.0438
19.60
3
2
GMM
0.9743
0.0349
20.59
2
3
DDN
0.9844
0.0233
22.35
5
5
Table 3: Prototype quality comparison on MNIST. All prototypes are compared with the real class-wise means over 60,000 training samples resized to 32×32 . PSNR is computed with a peak value of 2 under [−1,1] normalization. Cos. wins / MSE wins report the number of digits (out of 10) on which each method achieves the best per-digit cosine similarity / MSE.
Figure 3: Comparison of cluster centers. Prototypes: Class mean, DDN, K-Means and GMM (columns = matched digit 0–9).
Figure 4: t-SNE visualization on MNIST. Colors indicate ground-truth digit classes. Our method yields more separated clusters.
Figure 5: Generated samples. Continuous results are obtained by interpolation; discrete results are randomly selected candidates.
Figure 6: Visualization of generation results with different candidate numbers K on the MNIST dataset. When K≤10 , we use a single-layer DDN with the corresponding K . When K>10 , we use logK DDN layers.
Figure 7: Time consumption of different number of cluster K on the MNIST dataset.
Config
MNIST
F-MNIST
CIFAR-10
COIL20
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
(a) Effect of network width nf (with λ=0.1 )
4
60.80
0.612
54.30
0.421
25.00
0.132
50.60
0.496
8
61.00
0.621
54.60
0.430
25.10
0.134
50.87
0.503
16
61.20
0.624
54.70
0.427
25.30
0.137
51.00
0.508
32
62.14
0.627
54.81
0.428
25.38
0.138
51.09
0.511
Table 4: Ablation study on network width nf , Sinkhorn–Knopp loss presence, and weight λ .
Figure 8: Training loss comparison of mean squared loss and Sinkhorn–Knopp loss.
Diffusion models and flow-based methods have shown impressive generative capability, especially for images, but their sampling is expensive because it requires many iterative updates. We introduce W-Flow, a framework for training a generator that transforms samples from a simple reference distribution into samples from a target data distribution in a single step. This is achieved in two steps: we first define an evolution from the reference distribution to the target distribution through a Wasserstein gradient flow that minimizes an energy functional; second, we train a static neural generator to compress this evolution into one-step generation. We instantiate the energy functional with the Sinkhorn divergence, which yields an efficient optimal-transport-based update rule that captures global distributional discrepancy and improves coverage of the target distribution. We further prove that the finite-sample training dynamics converge to the continuous-time distributional dynamics under suitable assumptions. Empirically, W-Flow sets a new state of the art for one-step ImageNet 256×256 generation, achieving 1.29 FID, with improved mode coverage and domain transfer. Compared to multi-step diffusion models with similar FID scores, our method yields approximately 100× faster sampling. These results show that Wasserstein gradient flows provide a principled and effective foundation for fast and high-fidelity generative modeling.
We introduce a new framework for one-step generative modelling on finite state spaces. To extend drifting beyond continuous domains, we use discrete Wasserstein geometry to define a target-relative KL gradient flow over the transitions of a reversible Markov kernel. We realize this probability flow at the particle level through Markov jumps and amortize the resulting transport updates into a latent-conditioned generator, so that the iterative dynamics are required only during training while inference remains one-step. In a controlled setting where the underlying distributions and transport dynamics can be computed exactly, we verify KL dissipation, consistency between the particle dynamics and the probability flow, and the predicted numerical scaling. We further show that a finite-capacity neural generator can track these exact transport targets while retaining one-step generation. These results validate the basic construction and provide a foundation for scaling Discrete Drifting to structured discrete data.
Alessandro Micheli, Andrea Zerio, Samir Bhatt
Imperial College London London, United Kingdom · Department of Computer Science, Aalborg University, Copenhagen, Denmark; Centre for Frontier AI Research (CFAR), Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore · University of Copenhagen Copenhagen, Denmark
Modern generative models based on diffusion or flow matching transport noise toward the data manifold, without representing the data manifold explicitly. We propose Distance Marching that explores an alternative design by explicitly modeling a distance field of the data manifold. A distance field offers a geometric view of generation as a descent process toward the data manifold using only the current position. The negative gradient of the distance field provides the direction, and the distance value can optionally determine the step size. However, sparse training data in high-dimensional space make it difficult to identify the closest points on the data manifold needed to train the distance field. To address this challenge, we design new loss functions that emphasize denoising targets from closer data points. Our analysis shows that, without time input, emphasizing closer targets reduces denoising ambiguity and the bias toward the data mean. This distance-field perspective naturally motivates two principled deterministic samplers, showing fast convergence in inference. As a result, our model is the only one without time input that remains competitive with time-conditioned models across datasets and architectures. Moreover, our distance prediction is also helpful for early stopping during sampling and for out-of-distribution detection. We hope distance-field modeling can open a promising direction for exploring generative modeling from a geometric perspective.