stat.MLNov 13, 2026

Lipschitz-Constrained Discriminators in Generative Modeling: Asymptotic Behavior and Support Mismatch

Authors: Eduardo Pinto, Panagiota Birmpa

Organizations: Instituto Superior Técnico, Department of Mathematics, University of Lisbon · Department of Actuarial Mathematics and Statistics, Heriot-Watt University and the Maxwell Institute for Mathematical Sciences

Abstract

We study the asymptotic behavior of optimal discriminators associated to L−L-Lipschitz regularized KL-divergences of two probability distributions QQ and PP that interpolate between the rescaled Wasserstein−1-1 metric by some parameter L>0L>0 and the KL divergence. The parameter LL plays the role of a Lipschitz constraint and is viewed as a regularization parameter: small values of LL enforce a more geometry-aware discriminator, while as LL increases the geometric regularization is progressively relaxed. It is a known result [3] that such divergences converge to the KL divergence as L→∞L\to\infty. In this paper, we characterize the asymptotic behavior of the optimal discriminators as L→∞L\to\infty in different regimes: Firstly, under Q≪PQ\ll P and the Radon-Nikodym derivative dQdP\frac{dQ}{dP} is bounded from above and bounded away from zero. Secondly, when QQ and PP are discrete distributions satisfying Q≪PQ\ll P, and finally when QQ and PP are non absolutely continuous discrete distributions. This result motivates training strategies which involve discriminators, such as adversarial generative models. In practice, the data distribution is often represented by a discrete empirical measure, while, especially at the early stages of the training, the model distribution may have little or no overlap with the empirical data distribution.

Figures & tables

Explore similar work

Mar 19, 2026

Seasoning Generative Models for a Generalization Aftertaste

The use of discriminators to train or fine-tune generative models has proven to be a rather successful framework. A notable example is Generative Adversarial Networks (GANs) that minimize a loss incurred by training discriminators along with other paradigms that boost generative models via discriminators that satisfy weak learner constraints. More recently, even diffusion models have shown advantages with some kind of discriminator guidance. In this work, we extend a strong-duality result related to ff-divergences which gives rise to a discriminator-guided recipe that allows us to refine any generative model. We then show that the refined generative models provably improve generalization, compared to its non-refined counterpart. In particular, our analysis reveals that the gap in generalization is improved based on the Rademacher complexity of the discriminator set used for refinement. Our recipe subsumes a recently introduced score-based diffusion approach (Kim et al., 2022) that has shown great empirical success, however allows us to shed light on the generalization guarantees of this method by virtue of our analysis. Thus, our work provides a theoretical validation for existing work, suggests avenues for new algorithms, and contributes to our understanding of generalization in generative models at large.
Oct 29, 2025

Revisiting GAN with Bayes-Optimal Discrimination

We approach generative adversarial network (GAN) training from its original real-versus-fake classification viewpoint, replacing the cross-entropy discriminator objective with an upper bound on the Bayes error rate (BER). Using the Bayes optimal learning threshold (BOLT) loss, we formulate an adversarial game in which the discriminator tightens this bound and the generator maximizes it. We characterize the unrestricted optimal discriminator and show that, under balanced class priors, the population game is equivalent to minimizing total variation between the data and generator distributions. Restricting the bounded discriminator to be 1-Lipschitz yields BOLT-GAN. For balanced priors and the full discriminator class, we establish upper and lower bounds relating its population discrepancy to Wasserstein-1 distance on bounded domains. Controlled experiments on four image-generation benchmarks show improved sample quality and coverage over WGAN-GP under matched architectures, optimization settings, and gradient penalties. At a common 20-epoch budget, BOLT-GAN reduces Fréchet Inception Distance (FID) from 60.0 to 44.2 on CIFAR-10 and from 102.5 to 28.4 on LSUN Bedroom-64.
Oct 24, 2026

A Distribution-Matching Perspective on LLM Training Algorithms

In this work, we propose a unified framework for training algorithms across different stages of modern large language model development, based on the perspective of distribution matching. We show that a broad class of language model training algorithms can be interpreted as minimizing a certain divergence between the current model and a target distribution, and that characterizing the properties of this target distribution provides substantial insight into the behavior of the corresponding training algorithms. By analyzing specific forms of the target distribution, we establish that On-Policy Distillation (OPD) enjoys an advantage over classical Knowledge Distillation (KD) when the current model distribution is close to the target distribution, while exhibiting a disadvantage when the two distributions are far apart. Furthermore, when the target distribution itself depends on the model being optimized, we show that the accumulation of estimation variance can induce a deterministic degradation of the target distribution. These results provide a theoretical explanation, at least in part, for the long-term behavior of existing Reinforcement Learning (RL) algorithms. With this intuition we further designed Conditional KL (CKL) regularization for mitigating the degradation of the target distribution. CKL can effectively preserve mode diversity while relaxing the constraint imposed by classical KL regularization on reward improvements. We believe this theoretical perspective can bring some new understanding of language model training algorithms and provide useful guidance for future research.