Estimating the score function of a distribution plays a central role in statistics and machine learning, with applications in distribution estimation, sampling, and score-based generative modeling. However, learning scores from sensitive data raises privacy concerns. These concerns are particularly pronounced in score-based generative modeling, where trained models may memorize individual training examples and expose sensitive information through membership inference and data extraction attacks. Although non-private score estimation has been extensively studied, the statistical limits of its private counterpart remain less understood. In this work, we study score estimation under (α,β)-differential privacy for d-dimensional sub-Gaussian distributions with locally γ-Hölder-continuous scores. We develop a sampling-based private score estimator using quadratic B-splines and the exponential mechanism, and we implement our proposal through projected Langevin Monte Carlo with runtime polynomial in the sample size n. The proposed estimator satisfies (α,β)-differential privacy, and achieves expected density-weighted squared error \mboxpolylog(n)max(n−2γ/(d+2γ+2),(nα)−2γ/(d+γ+1)). We also establish a nearly matching private minimax lower bound, thereby characterizing the privacy-accuracy trade-off for score estimation. As an application, we construct a differentially private diffusion model whose privacy guarantee is independent of the number of denoising steps. We show that, with an appropriate discretization scheme, the resulting diffusion model attains nearly minimax optimal expected total variation error among all private distribution estimators. We validate our theoretical findings through experiments on both synthetic and real-world datasets.
Figures & tables
Figure 1: Memorization in a diffusion model. A training photograph (left) and a near-identical image generated by Stable Diffusion (right). Source: Carlini et al. (2023) , Figure 1.
Figure 2: Illustration of distributions induced by the exponential mechanism.
Figure 3: Clean score estimation error for distribution ( 25 ). Each panel fixes a smoothness level γ and plots the mean density-weighted squared score error against the training sample size n on logarithmic scales. Blue curves average the risks over 100 independent experiments. Error bars give 95% confidence intervals. The dotted and dash-dot lines represent the rates n−2γ/(3+2γ) and (nα)−2γ/(2+γ) , respectively. Their log-log intercepts are chosen separately so that both lines pass through the private error point at n=1000 in each panel. These lines illustrate slopes for decay rates rather than actual lower bounds.
Figure 4: Clean-score estimation risk for distribution ( 25 ). We set γ=0.5 , and each panel fixes a privacy level α and plots the mean population-weighted squared score error against the training sample size n on logarithmic scales. All other plotting conventions are as in Figure 3 .
Figure 5: Illustration of single observation perturbations.
Figure 6: Mean TV error of the private diffusion model for γ=0.5 , with each panel fixing a privacy level α . Error bars indicate pointwise 95% confidence intervals. Dotted and dash-dot lines show the reference rates, with their intercepts chosen separately to match the first private error point in each panel.
Figure 7: Mean TV error of the private diffusion model for α=2 , with each panel fixing a smoothness level γ . Error bars indicate pointwise 95% confidence intervals. Dotted and dash-dot lines show the reference minimax rates, with their intercepts chosen separately to match the first private error point in each panel.
Figure 8: TV changes of both private and non-private DDPMs induced by replacing one observation with 3 .
Figure 9: Clean score estimation results for the NHANES real-data experiment. The left panel reports the held-out empirical score-matching risk for the private score estimator and the non-private ERM benchmark. The right panel reports the RMS difference when we perturb one training sample. Markers denote means and error bars are pointwise 95% confidence intervals.
Figure 10: Binned TV distances between the generated and target distributions, estimated using held-out datasets. The left panel reports the joint binned TV distance, and the right panel reports the marginal binned TV distances. Solid curves denote means for DP diffusion models across 100 repetitions with pointwise 95% confidence intervals, and dashed horizontal lines denote non-private ERM means.
Figure 11: Binned TV distances between the output distributions of diffusion models trained on the original and perturbed datasets. Markers denote means over 100 repetitions and error bars denote pointwise 95% confidence intervals. The dashed curve gives the non-private ERM benchmark.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Empirical convergence diagnostics for projected LMC. Each cell reports the maximum R across all spline coefficients, computed from four independent chains. The left panel fixes α=2 and varies the smoothness parameter γ , while the right panel fixes γ=0.5 and varies α .
Figure 13: Two-dimensional clean-score estimation risk with α=2 . Each panel fixes a smoothness parameter γ and plots the density-weighted mean squared score error against the training sample size n on logarithmic scales. The non-private ERM benchmark is included for comparison. The dotted and dash-dotted lines represent n−2γ/(4+2γ) and (nα)−2γ/(3+γ) , respectively, with intercepts calibrated to the first private error point in each panel. The lines illustrate theoretical slopes.
Figure 14: Two-dimensional clean-score estimation risk with γ=0.5 . Each panel fixes a privacy level α .
Figure 15: Two-dimensional diffusion-model TV error with γ=0.5 . Each panel fixes a privacy level α and shows the mean TV error against the training sample size n , together with the non-private benchmark. The dotted and dash-dotted lines display the reference rates n−(γ+1)/(4+2γ) and (nα)−(γ+1)/(3+γ) , respectively, and their intercepts are matched to the first private error point in each panel.
Figure 16: Two-dimensional diffusion-model TV error with α=2 . Each panel fixes a smoothness parameter γ . The non-private benchmark and the reference rates are plotted using the same conventions as in Figure 15 .
Figure 17: The target distribution and sampled observations for the 2-dimensional setting, illustrating the replacement of a single observation by (3,3) .
Figure 18: Empirical stability under single-observation perturbations in the two-dimensional setting with γ=0.5 . Results are shown for different privacy levels and training sample sizes, with non-private ERM as a benchmark.
Estimating the difference of two Stein's score functions is a fundamental problem in generative modeling. In particular, score differences arise naturally in transfer learning, where the score difference provides the mechanism for adapting a pre-trained model to a new target distribution, and in diffusion model-based post-training methods such as discriminator guidance. Existing estimators for score differences in these settings either lack of statistical consistency or are difficult to scale up in high-dimensions. We propose a statistically consistent and scalable estimator for score differences based on Sobolev regularization, which plays a crucial role in ensuring consistency and stablizing the training in the small-sample regime. Mathematically, we establish a convergence rate of O(n−d+2s−2s−1) where d is the dimension and s denotes the smoothness of the underlying densities, and provide a minimax lower bound of Ω~(n−d+2s2(s−1)) (in mean-squared error). Empirically, our estimator exhibits significantly improved stability in small-sample regimes compared to existing methods. We demonstrate its effectiveness on real-world tasks, including transfer learning for ECG signal generation, where it substantially outperforms non-regularized score difference estimators in downstream classification performance.
Chenghan Xie, Jose Blanchet, Renyuan Xu
Department of Management Science and Engineering, Stanford University.
Score matching is an alternative to maximum likelihood estimation when the normalizing constant is unknown or too costly to evaluate. However, vanilla score matching has shown to be inefficient relative to maximum likelihood estimation for multimodal distributions with well-separated modes, which are commonly encountered in practical applications. We compare a novel diffusion-based denoising score matching estimator (DDSME) to the vanilla score matching estimator (SME) in this scenario. In particular, we prove statistical guarantees for both estimators, showing that the error bound for the vanilla SME worsens when the separation between the modes increases, which can be avoided in case of the DDSME with suitable hyperparameter tuning. This provides a novel theoretical explanation for the superior behavior of diffusion-based score matching over the vanilla version.
Benedikt Lütke Schwienhorst, Nadja Klein, Johannes Lederer
Department of Mathematics University of Hamburg Hamburg, Germany · Scientific Computing Center Karlsruhe Institute of Technology2026 Karlsruhe, Germany · Department of MathematicsMay University of Hamburg Hamburg, Germany21
Score-based diffusion models are increasingly considered in settings where the underlying data distribution may differ from the training distribution, yet existing theoretical guarantees largely focus on the no-shift setting. In this work, we study robust score estimation under Wasserstein perturbations of a reference distribution. For the Ornstein--Uhlenbeck diffusion, we show that robust estimation decomposes into two fundamental components: the statistical cost of learning the reference distribution and the intrinsic cost of distribution shift. The latter scales quadratically with the Wasserstein radius, and this dependence is minimax optimal. We construct an explicit finite-sample estimator achieving the resulting robust minimax rate without knowing the shift radius. When the reference distribution lies on an unknown low-dimensional subspace, the statistical term adapts to the intrinsic dimension while the shift cost remains unchanged. Finally, we show that the same decomposition governs positive-time reverse sampling and obtain matching minimax guarantees in KL divergence. Together, these results characterize how finite data, intrinsic dimension, and distribution shift affect the robustness of score-based diffusion models.
Wei Luo, Neil K. Chada, Shijie Zhang +1
Department of Mathematical Sciences, Tsinghua University · Department of Mathematics, City University of Hong Kong, Hong Kong SAR · Artificial Intelligence Research Institute, Shenzhen MSU-BIT University +1