Neural samplers are trained against an unnormalised target π~=e−E with no samples from π, which leaves the practitioner with no way to tell whether an expensive training run has silently dropped part of the target. The diagnostics in common use are computed from the model's own draws and are therefore confined to the model's support: we exhibit a sampler whose self-normalised effective sample size is 0.99 while it misses 87% of the target mass. We argue that detecting missing mass is a strictly easier problem than sampling it: detection needs one point per missed basin plus a local curvature estimate, whereas correction needs the sampler retrained. We turn this into a pre-flight check that consumes a few percent of the sampler's own training budget and uses only E, ∇E and ∇2E. On Gaussian-mixture, Many-Well and rotated anisotropic Many-Well targets with exactly computable ground truth, the check estimates the missing mass to within 10−3 at 2.7% of training cost, where a tuned annealed SMC reference needs 70--280% of training cost to do worse. It also applies unchanged to a controlled-SDE sampler that has no tractable density, where ESS and the ELBO cannot be formed at all. The estimator carries a self-diagnostic that, without ground truth, is conservative in the safe direction: across 60 configurations it clears 16, of which 15 are accurate to 10−2 or better. We are explicit about what this does and does not license: the check cheaply produces evidence of missing mass, and sometimes evidence that the search has stabilised, but it cannot certify a run, and its thresholds are heuristic. We then map the boundary of the method on a real physical landscape, LJ-13, and report where it fails and why.
Figures & tables
Figure 1: The check in two dimensions, on the configuration of Proposition 1 . (a) A six-component target (contours) and a sampler that reproduces two of the components exactly and places no mass elsewhere (dots). Its self-normalised ESS is 1.00 – the largest value the diagnostic can take – while 46% of the target mass is absent. (b) Multi-start second-order search from a broad initialisation recovers all six modes, including the four the sampler never visits. (c) Laplace ellipses at the recovered modes, filled where the sampler’s draws populate them and open where they do not, giving M=0.46 against a true 0.46 . Nothing in the pipeline uses samples from π .
Figure 2: The self-diagnostic against ground truth it never sees, over all 60 configurations. (a) Error of the estimate, grouped by verdict; the dotted line marks 10−2 . Of 16 cleared configurations, 15 are accurate. (b) The same points against the search budget; filled markers are declared reliable. Both the error and the verdict improve with R , which is what makes “increase the budget” the right response to a warning.
target
restarts R
mean abs. error
cleared
Gaussian mixture
32
0.1955
0/5
Gaussian mixture
64
0.1330
1/5
Gaussian mixture
256
0.0183
1/5
Gaussian mixture
1024
0.0001
3/5
Many-Well
32
0.1434
0/5
Many-Well
64
0.0403
0/5
Table 1: Self-diagnostic by target and search budget, five seeds each. The verdict column counts how many of the five seeds were cleared.
pre-flight check
best SMC reference
target
cost
error
cost
error
Gaussian mixture
2.7%
0.0183
70.5%
0.0154
Gaussian mixture
10.4%
0.0001
70.5%
0.0154
Many-Well
2.7%
0.0014
281.7%
0.0080
Rotated Many-Well
2.7%
0.0008
281.7%
0.1108
Table 2: Mean absolute error of the estimated missing mass over five seeds, against the best configuration of a tuned SMC reference. Cost is a fraction of the sampler’s own training budget.
Figure 3: Error–cost frontier, five seeds. The check (blue) sits below and to the left of the tuned SMC reference (orange) on every target, and the gap widens as the target gets harder: on the rotated anisotropic target the reference has not become reliable even at 2.8× the sampler’s training cost.
d
target evals (% of training)
wall-clock (% of training)
abs. error
16
2.69%
2.7%
0.0012
32
3.96%
4.4%
0.0017
64
6.49%
9.2%
0.0017
128
11.57%
18.7%
0.0017
Table 3: Cost of the check as the dimension grows, Many-Well with m=5 ( 32 modes), R=256 , three seeds. Accuracy is flat; cost grows about linearly in d because a full Hessian needs O(d) backward passes.
target
family
Msupp
Mwt
M
ESS
norm. gap
SMC
ours
GMM
tilt τ=1
0
0.000
0.000
1.000
−0.000
0.000
0.006
tilt τ=0.5
0
0.083
0.083
0.959
+0.001
0.000
0.086
tilt τ=0.25
0
0.128
0.128
0.909
+0.002
0.000
0.131
tilt τ=0
0
0.180
0.180
0.843
+0.003
0.000
0.179
tilt τ=2
0
0.137
0.137
0.843
−0.002
0.000
0.138
tilt τ=3
0
0.235
0.235
0.498
−0.004
0.000
0.236
Table 4: Constructed collapse with exactly known decomposition. In the tilt family Msupp=0 , so by Corollary 1 the normalizer gap must be zero whatever Mwt is: it is, and the SMC estimate is exactly 0.0000 in all twelve tilt rows. The mixed rows vary Mwt at fixed Msupp and are the cleaner demonstration – on the GMM the gap reads 0.104 three times while Mwt grows from 0 to 0.059 . ESS, blind to Msupp , does respond here. Search budget R=1024 on the GMM and R=256 on Many-Well, three repetitions.
Figure 4: The blind spot of the normalizer comparison. Within each panel every sampler reaches every basin ( Msupp=0 ) and only the proportions are wrong. The SMC estimate built on comparing normalising constants stays pinned at zero while more than half the target mass is misallocated, as Corollary 1 requires. ESS responds; the check tracks the truth.
basins reached
Msupp
Mwt
M
check: abs. error
support collapse
1/32
0.569
0.000
0.569
0.0025
weight collapse
32/32
0.000
0.559
0.559
0.0057
Table 5: Two training runs of the same controlled-SDE architecture on the same target, with nearly equal missing mass and opposite decomposition. The Msupp row is the audit of § 6 (five seeds); the Mwt row differs only in the gradient clipping applied inside the control (two seeds). ESS, the ELBO and Zq are undefined for both.
Table 6: A non-normalizer competitor on Many-Well ( K=32 , true M=0.9854 ). Parallel tempering is not structurally blind the way the normalizer gap is: it reaches the right answer given enough steps. It costs 15.7× more to get there. Costs are relative to this sampler’s deliberately short training budget of 1.5×106 evaluations and are not comparable to the 2.7% of Table 2 , which is measured against the longer runs used there; what is comparable is the ratio within this table.
β
heaviest mode
#modes 90%
#modes 99%
captured by 500
by 2000
1
57.5±24.3%
6.8±3.3
15.2±7.3
3.0±2.9%
23.4±26.1%
2
52.4±25.2%
7.2±3.5
14.4±6.9
3.5±2.9%
18.8±18.6%
3
64.5±22.9%
5.0±2.9
12.0±5.9
3.7±4.0%
13.1±10.9%
5
81.9±15.7%
2.0±0.6
3.2±1.6
63.3±33.3%
64.0±33.7%
8
100.0±0.0%
1.0±0.0
1.0±0.0
100.0±0.0%
100.0±0.0%
Table 7: LJ-13. Concentration of the harmonic weight and the fraction of it captured by a cheap search, as a function of inverse temperature. Mean ± s.d. over five independent enumerations of 6000 restarts each. Note the standard deviations in the last two columns: below β=8 the captured fraction is not merely small but erratic.
Monte Carlo sampling algorithms aim to produce samples from a target distribution with a given unnormalised probability density function. The high cost of running Monte Carlo algorithms, with many simulation steps or particles needed to obtain a good empirical approximation to the target, can be reduced by introducing learnable components into the sampler -- a form of amortisation. Once these components are trained, amortised samplers can produce high-quality samples with less computation at sampling time than training-free algorithms would require. We propose an amortised sampler built upon the framework of adaptive importance sampling -- a hierarchical importance sampling procedure where the proposal is iteratively adapted. Our amortised adaptive importance sampling algorithm fits a lightweight neural network that adapts the proposal using information obtained over the course of sampling. By training our sampler over many different targets taken from the same family of distributions, we obtain a sampler that can adapt to a specific target at sampling time without a priori access to the target's parameters. We show that our procedure can efficiently sample from targets within the family the sampler was trained on, giving improvements over target-specific amortised samplers and training-free adaptive importance sampling algorithms. We also explore the potential for generalisation, where the trained sampler is used to sample from targets outwith its training family.
Arran Carter, Víctor Elvira, Esmeralda S. Whitammer
We present a drifting-based framework for amortized sampling of Boltzmann distributions defined by energy functions. The method trains a one-step neural generator by projecting samples along a Gaussian-smoothed score field from the current model distribution toward the target Boltzmann distribution. For targets specified only up to an unknown normalization constant, we derive a practical target-side drift from a smoothed energy and use two estimators: a local importance-sampling mean-shift estimator and a second-order curvature-corrected approximation. Combined with a mini-batch Gaussian mean-shift estimate of the sampler-side smoothed score, this yields a simple stop-gradient objective for stable one-step training. On a four-mode Gaussian-mixture Boltzmann target, our sampler achieves mean error 0.0754, covariance error 0.0425, and RBF MMD 0.0020. Additional double-well and banana targets show that the same formulation also handles nonconvex and curved low-energy geometries. Overall, the results support drifting as an effective way to amortize iterative sampling from Boltzmann distributions into a single forward pass at test time.
Wenhan Cao, Keyu Yan, Lin Zhao
Department of Electrical and Computer Engineering, National University of Singapore
Sampling from discrete, unnormalized distributions without access to data is a challenging problem. Neural samplers offer a promising approach by training generative models from density evaluations directly. Despite recent progress, existing discrete neural samplers are prone to mode collapse, come without convergence guarantees when trained via fixed-point iterations, and are often tied to a specific reference process such as masked or uniform diffusion. In this work, we introduce Discrete Gibbs Iterative Neural Sampler, a fixed-point neural sampler that addresses these limitations, enabling efficient, scalable learning, substantially reducing mode collapse in practice. Our framework builds on masked diffusion and also extends to transport between pairs of distributions. We demonstrate that the resulting method scales effectively to high-dimensional systems, supports amortized sampling across different conditions, and enables accurate estimation of alloy phase diagrams.