Denoising and score estimation are classically linked through Tweedie's formula, which relates the posterior mean under Gaussian noise to the Stein score of the noisy marginal. In this work, we extend this perspective beyond Gaussian noise to a broad class of Gibbs (energy-based) noise distributions, with the generalized Gaussian family as an example. We derive the Energy-Tweedie identity: when the denoising posterior is viewed through the lens of scoring rules, the path derivative of a kernel scoring rule defined by the noise potential recovers the Stein score of the noisy marginal. Thus, the familiar correspondence between Gaussian noise, posterior means, squared loss, and Tweedie's formula is lifted to a distributional correspondence between Gibbs noise distributions, full posterior laws, kernel scoring rules, and the Energy-Tweedie identity, which gives a Tweedie-style relation for each noise potential. Among its consequences, this identity gives a posterior-samples-to-score route to score estimation, yields a principled criterion for estimating unknown noise parameters, and enables diffusion-style sampling along user-chosen paths through the noise-parameter space, supplying the score-based perspective on recent generative methods trained with scoring rules.
Figures & tables
Estimator
β
λ
σ
u
v
ET posterior
0.050±0.010
0.039±0.004
0.077±0.002
0.037±0.004
0.029±0.004
ET-DSM
0.019±0.003
0.014±0.000
0.017±0.001
0.018±0.001
0.015±0.003
Hyvärinen
0.027±0.002
0.022±0.003
0.023±0.001
0.025±0.002
0.023±0.003
GT disagr.
0.0003
0.0003
0.0092
0.0003
0.0003
Table 1: Noisy-score accuracy across five parameter sweeps. Each cell is the mean over two models, sample std below. Range of parameters: β∈[1.3,2] , λ∈[1,2.5] , σ∈[0.1,0.75] , and u,v∈[−0.6,0.6] . GT disagreement is the NMSE between ground-truth scores computed on separate halves of the clean particles.
Fitted family
Score/ Posterior
β
λ
u
v
NMSE
generalized
oracle/ oracle
1.4094±0.0062
1.7814±0.0041
0.2537±0.0043
0.3505±0.0037
0.0061±0.0004
generalized
learned/ oracle
1.4118±0.0075
1.7733±0.0145
0.2493±0.0075
0.3512±0.0059
0.0061±0.0005
generalized
oracle/ learned
1.4379±0.0111
1.7952±0.0098
0.2557±0.0036
0.3463±0.0056
0.0145±0.0018
generalized
learned/ learned
1.4398±0.0120
1.7875±0.0150
0.2512±0.0061
0.3472±0.0066
0.0146±0.0017
forced Gaussian anisotropic
oracle/ oracle
≡2
1.4697±0.0210
0.2588±0.0087
0.3466±0.0043
0.0342±0.0016
forced isotropic generalized
oracle/ oracle
1.1206±0.0240
1.8078±0.0154
≡0
≡0
0.1324±0.0025
Table 2: Full-SPD noise-parameter estimation. Ground truth: (β∗,λ∗,u∗,v∗)=(1.4,1.8,0.25,0.35) . Estimated parameters and held-out score NMSE for the generalized family and three misspecified alternatives. NMSE is computed against the oracle noisy score, pooled over the three σ levels. Standard deviations are over the five data replicates for the oracle posterior, and over the fifteen fits (3 trained posterior models × 5 replicates) when it is learned.
Figure 1: Energy-Tweedie score fields from the five-parameter Eight-Gaussians posterior model as each noise parameter is varied. The colorbar represents the score magnitude (log scale).
Figure 2: Unselected MNIST samples from the Gaussian/isotropic, Gaussian/non-diagonal, generalized/isotropic, generalized/non-diagonal, and joint paths (left to right).
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
d
Ambient data dimension, with X,ϵ,Y∈Rd .
X,Y,ϵ
Clean data, noisy observation, and additive noise, with Y=X+ϵ .
Z,W,…
Generic random variables used in distributional statements such as Z∼P and W∼Q .
p(x),q(u),m(y)
Clean-data density, noise density, and noisy marginal density.
P(X∣Y=y)
Denoising posterior distribution of X given noisy observation y .
sm(y)
Stein score of the noisy marginal: sm(y)=∇ylogm(y) .
Appendix
Table 3: Core notation used throughout the paper.
Figure 3: Score NMSE (top, logarithmic scale) and cosine similarity (bottom) of the three estimators at every parameter combination. Error bars are sample SDs across two training seeds. The dotted gray curves show the half-bank ground-truth NMSE for the posterior evaluation.
Sweep
ET posterior
ET-DSM
Hyvärinen
β
0.970±0.011
0.989±0.002
0.985±0.013
λ
0.989±0.003
1.003±0.001
1.006±0.005
σ
1.036±0.013
0.990±0.003
0.982±0.001
u
0.987±0.003
0.991±0.002
0.991±0.006
v
0.998±0.003
0.995±0.003
0.995±0.003
Appendix
Table 4: Ratio of mean estimated-score norm to mean ground-truth score norm. Values are means ± sample SDs across training seeds after equal averaging within each sweep.
Sweep
Mean MSE
Trace ratio
Mode TV
ED
ED SD
ED excess
β
0.0057±0.0012
1.112±0.033
0.044±0.004
0.0127±0.0008
0.0256±0.0018
0.0129±0.0008
λ
0.0042±0.0003
1.042±0.001
0.043±0.002
0.0087±0.0006
0.0170±0.0011
0.0088±0.0006
σ
0.0037±0.0004
1.046±0.007
0.029±0.002
0.0066±0.0009
0.0164±0.0053
0.0069±0.0009
u
0.0042±0.0004
1.051±0.008
0.034±0.001
0.0092±0.0005
0.0274±0.0000
0.0094±0.0005
v
0.0035±0.0004
1.045±0.009
0.032±0.002
0.0067±0.0009
0.0168±0.0025
0.0068±0.0009
Appendix
Table 5: Posterior diagnostics of the learned posterior against the importance posterior, means ± sample SDs across training seeds. Mean MSE is the squared error of the posterior mean; trace ratio is the ratio of mean posterior covariance traces; mode TV is the total variation distance between nearest-mixture-center assignments. ED is the unbiased energy distance from the learned to the importance posterior, evaluated at 48 observations per parameter combination; ED SD is its standard deviation across those observations. ED excess subtracts the ED between two ground-truth resamples of Table 6 .
Split ground-truth NMSE
Importance ESS
Sweep
Posterior
Direct
Median
Minimum
Ground-truth ED
β
0.0003
0.0003
14,713
4,581
−0.0002±0.0029
λ
0.0003
0.0003
14,491
5,906
−0.0002±0.0029
σ
0.0092
0.0111
10,965
4,291
−0.0003±0.0020
u
0.0003
0.0003
14,523
5,277
−0.0002±0.0027
v
0.0003
0.0002
14,307
5,475
−0.0001±0.0027
Appendix
Table 6: Ground-truth diagnostics averaged over parameter combinations within each sweep. Split ground-truth NMSE is the NMSE between the two independent half-bank score estimates, for the posterior and the direct evaluations. ESS columns average the per-combination median and minimum importance effective sample sizes of the posterior evaluation. Ground-truth ED is the energy distance between independent resamples of the importance posterior; the value after ± is the average within-combination SD over observations.
Figure 4: Posterior-route score cosine (left two panels) and score NMSE (right two panels) against noise magnitude for isotropic Gaussian and generalized Gaussian noise. Cosine is shown over all observations and separately split into cases where the corresponding ground truth is above- or below-median in norm. Error bars are standard deviations across two independently trained posterior models. Gray shading marks magnitudes at which the half-bank ground-truth NMSE exceeds 0.01 .
Noise
Posterior
NMSE
Cosine (all)
Cosine (high norm)
Cosine (low norm)
Gaussian, σ=1.260
learned
0.0054
0.867
0.999
0.732
Gaussian, σ=1.260
exact
0.0055
0.875
0.999
0.752
Generalized, σ=2.277
learned
0.0046
0.919
0.999
0.836
Generalized, σ=2.277
importance-sampled
0.0036
0.904
1.000
0.809
Appendix
Table 7: Cosine-dip control at the dip location of Fig. 4 , with N=192 posterior draws per observation. “Learned” is the five-parameter posterior model; “exact” and “importance-sampled” draw from the true posterior and apply the same Monte Carlo score estimator. “High/low norm” correspond to the same two cases as in Fig. 4 .
Figure 5: rη2 -weighted training (top) and fixed-validation (bottom) objectives for the two direct estimators. Faint curves are logged minibatch losses; solid training curves are trailing means over 20 logged batches. Validation uses eight fixed parameter combinations per seed, with 1,024 observations per combination. Dashed lines are the estimated population floors.
Estimator
Seed
Final loss
Estimated floor
Relative gap
ET-DSM
42
0.7871
0.7731
2.0%
ET-DSM
43
0.8711
0.8652
1.0%
Hyvärinen
42
−0.6439
−0.6790
5.2%
Hyvärinen
43
−0.5965
−0.6098
2.2%
Appendix
Table 8: Final weighted validation losses and estimated population floors.
Posterior samples N
Score NMSE ↓
Score cosine ↑
1
1.446±0.110
0.574±0.039
2
0.698±0.061
0.681±0.034
4
0.368±0.052
0.780±0.041
8
0.196±0.018
0.856±0.025
16
0.112±0.012
0.909±0.016
32
0.071±0.010
0.938±0.013
Appendix
Table 9: Score accuracy versus posterior draw budget. Left: score NMSE and cosine similarity versus the number of posterior draws per observation; the standard deviations are over 20 model–repetition units from two independently trained models: ten repetitions per model. Right: per-model score NMSE over the same budgets with the pooled a+b/N fit (dashed); error bars are per-model.
Figure 6: Clean data score fields for the Eight-Gaussians dataset. Top: Gaussian, bottom: generalized Gaussian noise. In each row, the left panel shows the analytic ground-truth clean field, and the right panel shows the field obtained via Richardson extrapolation.
Figure 7: Residual magnitudes for the clean score-field comparison in Fig. 6 . Left: Gaussian, and right: generalized Gaussian noise. The color encodes log(1+∥sRE(y)−∇ylogp(y)∥) .
Table 10: Estimated noise parameters, shape error, and held-out score error for four fitted families and four score/posterior source combinations. Setup and errors are the same as in Tab. 2 . S=S(u,v) and S⋆=S(u⋆,v⋆) are the fitted and true trace-normalized shape matrices; held-out MSE is the unnormalized numerator of the NMSE.
Figure 8: Four reverse paths through the Eight-Gaussians noise-parameter space. The panels show actual RMS rt , reference RMS ρt , tail shape βt , radial concentration λt , covariance coordinates ut,vt , and the path in the (u,v) plane, with t∈[0,1] denoting path progress. Each distinct trajectory is drawn once; panel legends identify the paths it represents. In the coordinate panel, solid and dashed lines denote ut and vt , while the dotted line groups shared zero coordinates. Fixed covariance paths are shown as points.
Figure 9: One-parameter Energy-Tweedie score-field sweeps of the five-parameter conditional model.
Path
Signed ED
Mode-weight TV
Min. mode mass
Mean radial error
Isotropic, fixed tail
0.0058±0.0009
0.017±0.002
0.119±0.001
0.233±0.001
Fixed non-diagonal, fixed tail
0.0081±0.0053
0.039±0.006
0.111±0.002
0.237±0.002
Rotating covariance, fixed tail
0.0072±0.0027
0.058±0.003
0.104±0.003
0.227±0.003
Joint five-parameter path
0.0072±0.0030
0.055±0.005
0.105±0.003
0.220±0.005
Independent clean draws
0.0007±0.0017
0.009±0.000
0.121±0.000
0.160±0.001
Appendix
Table 11: Endpoint generation metrics for the four Eight-Gaussians noise-parameter paths.
Figure 10: Signed energy distance between the reverse-chain state and forward-noised clean data at the matching noise level, along the four paths. Curves show mean ± sample standard deviation across the two trained models. The vertical axis uses a symmetric logarithmic scale.
Parameter
Fixed value
Signed ED ↓
Mode-weight TV ↓
Min. mode mass ↑
Max. tracking ED ↓
Clean-draw floor
Independent draws
0.00071±0.00171
0.0090±0.0001
0.1208±0.0002
—
Ref. fixed path
β=1.4,λ=1.8(u,v)=(0,0.6)
0.00746±0.00560
0.0461±0.0044
0.1056±0.0009
0.0267±0.0010
β
1.3
0.00787±0.00621
0.0470±0.0061
0.1053±0.0001
0.0246±0.0056
β
1.6
0.00666±0.00506
0.0432±0.0030
0.1065±0.0030
0.0276±0.0046
β
1.8
0.00622±0.00554
0.0391±0.0037
0.1078±0.0030
0.0206±0.0001
β
2.0
0.00627±0.00674
0.0337±0.0089
0.1101±0.0018
0.0355±0.0030
Appendix
Table 12: Endpoint and post-burn-in marginal-tracking metrics for the fixed-parameter path isolation study.
Figure 11: Endpoint samples for each of the four paths (both trained models pooled), with the clean Eight-Gaussians reference.
Figure 12: Reverse-chain states along the shared RMS schedule rt for the four paths (rows), from the initial noise level to the endpoint (columns).
Figure 13: Noise-magnitude, radial-law, and covariance trajectories for the five MNIST sampling paths. The panels show actual per-pixel RMS rt , reference RMS ρt , tail shape βt , radial concentration λt , covariance coordinates ut,vt , and the path in the (u,v) plane, with t∈[0,1] denoting path progress. Each distinct trajectory is drawn once; panel legends identify the paths it represents. In the coordinate panel, solid and dashed lines denote ut and vt , while the dotted line groups shared zero coordinates. Fixed covariance paths are shown as points; the dashed circle bounds the training region.
Path
Feature ED ↓
Mean confidence ↑
Class TV ↓
Real MNIST vs. real reference
0.015
0.993[0.989,0.996]
0.074[0.058,0.117]
Gaussian / isotropic
0.574[0.506,0.711]
0.922[0.910,0.934]
0.162[0.133,0.198]
Gaussian / non-diagonal
0.247[0.215,0.360]
0.964[0.956,0.971]
0.111[0.089,0.150]
Generalized / isotropic
0.637[0.548,0.808]
0.908[0.893,0.921]
0.113[0.089,0.152]
Generalized / non-diagonal
0.218[0.186,0.324]
0.966[0.959,0.973]
0.085[0.066,0.124]
Joint five-parameter path
0.303[0.259,0.424]
0.960[0.951,0.968]
0.107[0.081,0.142]
Appendix
Table 13: Feature energy distance, classifier confidence, and class-distribution total variation for real MNIST and the five sampling paths.
Figure 14: Generated samples (upper rows) and their nearest MNIST training images in pixel RMSE (lower rows) for each of the five sampling paths.
Figure 15: Reverse-chain states for the five MNIST paths (columns) at matched per-pixel RMS levels rt (rows), followed by the final posterior draw. Each path–level cell contains the same nine chains in a 3×3 grid.
Figure 16: Unselected endpoint samples for each path (rows) and sampling seed (columns).
Posterior
(L,H,Z)
FiLM width
Batch size (k)
Learning rate
Steps (k)
Gaussian, isotropic
(6,200,100)
16
40
2⋅10−5→2⋅10−6
240
Generalized, isotropic
(6,100,100)
linear
80
3⋅10−5→2⋅10−6
200
Five-parameter
(6,250,100)
64
40
1.5⋅10−5→10−6
280
Parameter estimation
(6,250,100)
64
40
1.5⋅10−5→10−6
280
MNIST
(16,2560,128)
256
1.536
1.5⋅10−5→1.5⋅10−6
100
Appendix
Table 14: Posterior architectures and training budgets. The first four rows use Eight Gaussians. L counts affine layers in the residual branches (two per block), H is the hidden width, and Z is the Gaussian noise dimension per layer. FiLM width is the hidden width of the conditioning MLP; “linear” denotes a single affine map. Batch sizes and optimizer steps are reported in thousands.
Score-based generative modeling learns data-distribution gradients to generate samples through reverse diffusion, flows, and related stochastic dynamics.
Diffusion models have achieved remarkable success in generating samples from unknown data distributions. Most popular stochastic differential equation-based diffusion models perturb the target distribution by adding Gaussian noise, transforming it into a simple prior, and then use denoising score matching, a consequence of Tweedie's formula, to learn the score function and generate clean samples from noise. However, non-Gaussian diffusion models with state-dependent diffusion coefficient have been largely underexplored, as have the corresponding Tweedie's formulae. In this work, we extend Tweedie's formula to important non-Gaussian processes, including geometric Brownian motion (GBM), squared Bessel (BESQ) processes, and Cox-Ingersoll-Ross (CIR) processes, thereby yielding the corresponding denoising score-matching objectives. We then apply the derived formulae to image and financial time series generation using GBM- and CIR-based diffusion models, and to empirical Bayes estimation under the BESQ setting. The reported experimental results demonstrate the potential of non-Gaussian models.
Score matching is an alternative to maximum likelihood estimation when the normalizing constant is unknown or too costly to evaluate. However, vanilla score matching has shown to be inefficient relative to maximum likelihood estimation for multimodal distributions with well-separated modes, which are commonly encountered in practical applications. We compare a novel diffusion-based denoising score matching estimator (DDSME) to the vanilla score matching estimator (SME) in this scenario. In particular, we prove statistical guarantees for both estimators, showing that the error bound for the vanilla SME worsens when the separation between the modes increases, which can be avoided in case of the DDSME with suitable hyperparameter tuning. This provides a novel theoretical explanation for the superior behavior of diffusion-based score matching over the vanilla version.
Benedikt Lütke Schwienhorst, Nadja Klein, Johannes Lederer
Department of Mathematics University of Hamburg Hamburg, Germany · Scientific Computing Center Karlsruhe Institute of Technology2026 Karlsruhe, Germany · Department of MathematicsMay University of Hamburg Hamburg, Germany21
Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. While positivity guarantees nonnegative reverse jump rates, it does not ensure Bayes realizability: ratios at a noisy state need not be jointly induced by any clean-token posterior under the forward kernel. The score-entropy loss has the correct population optimum but does not enforce this constraint away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half lie inside it yet remain materially incompatible with any valid posterior. Such violations can produce negative pre-normalization weights in finite-step sampling. Projecting raw scores onto the bridge polytope removes all observed negative weights and improves external generative PPL from 203.6 to 175.1 without changing the sampler. We introduce mean-to-score (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The construction applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from 3.173 to 3.129 and FID-50k from \CifarSEDDFID to \CifarMtwoSFID. A 170M-parameter M2S model trained on about 262B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching generative PPL 143.3 at 128 steps versus 183.6 for the strongest pure-uniform baseline.
Jingyuan Li, Xiaoyi Jiang, Yixuan Jiang +4
Tsinghua University · Beijing Institute of Mathematical Sciences and Applications · Wuhan University +1