Mode Collapse Is Cheap to Detect: A Ground-Truth-Free Pre-Flight Check for Neural Samplers

Jian Xu

cs.LGAug 20, 2026

Mode Collapse Is Cheap to Detect: A Ground-Truth-Free Pre-Flight Check for Neural Samplers

Authors: Jian Xu

Organizations: RIKEN iTHEMS, Wako, Japan · RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan

Abstract

Neural samplers are trained against an unnormalised target $\tildeπ=e^{-E}$ with no samples from $π$ , which leaves the practitioner with no way to tell whether an expensive training run has silently dropped part of the target. The diagnostics in common use are computed from the model's own draws and are therefore confined to the model's support: we exhibit a sampler whose self-normalised effective sample size is $0.99$ while it misses $87\%$ of the target mass. We argue that detecting missing mass is a strictly easier problem than sampling it: detection needs one point per missed basin plus a local curvature estimate, whereas correction needs the sampler retrained. We turn this into a pre-flight check that consumes a few percent of the sampler's own training budget and uses only $E$ , $\nabla E$ and $\nabla^2 E$ . On Gaussian-mixture, Many-Well and rotated anisotropic Many-Well targets with exactly computable ground truth, the check estimates the missing mass to within $10^{-3}$ at $2.7\%$ of training cost, where a tuned annealed SMC reference needs $70$ -- $280\%$ of training cost to do worse. It also applies unchanged to a controlled-SDE sampler that has no tractable density, where ESS and the ELBO cannot be formed at all. The estimator carries a self-diagnostic that, without ground truth, is conservative in the safe direction: across $60$ configurations it clears $16$ , of which $15$ are accurate to $10^{-2}$ or better. We are explicit about what this does and does not license: the check cheaply produces evidence of missing mass, and sometimes evidence that the search has stabilised, but it cannot certify a run, and its thresholds are heuristic. We then map the boundary of the method on a real physical landscape, LJ-13, and report where it fails and why.

Figures & tables

Figure 1: The check in two dimensions, on the configuration of Proposition 1 . (a) A six-component target (contours) and a sampler that reproduces two of the components exactly and places no mass elsewhere (dots). Its self-normalised ESS is

1.00

– the largest value the diagnostic can take – while

46\%

of the target mass is absent. (b) Multi-start second-order search from a broad initialisation recovers all six modes, including the four the sampler never visits. (c) Laplace ellipses at the recovered modes, filled where the sampler’s draws populate them and open where they do not, giving

\widehat{M}=0.46

against a true

0.46

. Nothing in the pipeline uses samples from

\pi

Figure 2: The self-diagnostic against ground truth it never sees, over all

60

configurations. (a) Error of the estimate, grouped by verdict; the dotted line marks

10^{-2}

. Of

16

cleared configurations,

15

are accurate. (b) The same points against the search budget; filled markers are declared reliable. Both the error and the verdict improve with

R

, which is what makes “increase the budget” the right response to a warning.

target	restarts $R$	mean abs. error	cleared
Gaussian mixture	32	0.1955	0/5
Gaussian mixture	64	0.1330	1/5
Gaussian mixture	256	0.0183	1/5
Gaussian mixture	1024	0.0001	3/5
Many-Well	32	0.1434	0/5
Many-Well	64	0.0403	0/5

Table 1: Self-diagnostic by target and search budget, five seeds each. The verdict column counts how many of the five seeds were cleared.

	pre-flight check		best SMC reference
target	cost	error	cost	error
Gaussian mixture	2.7%	0.0183	70.5%	0.0154
Gaussian mixture	10.4%	0.0001	70.5%	0.0154
Many-Well	2.7%	0.0014	281.7%	0.0080
Rotated Many-Well	2.7%	0.0008	281.7%	0.1108

Table 2: Mean absolute error of the estimated missing mass over five seeds, against the best configuration of a tuned SMC reference. Cost is a fraction of the sampler’s own training budget.

Figure 3: Error–cost frontier, five seeds. The check (blue) sits below and to the left of the tuned SMC reference (orange) on every target, and the gap widens as the target gets harder: on the rotated anisotropic target the reference has not become reliable even at

2.8\times

the sampler’s training cost.

$d$	target evals (% of training)	wall-clock (% of training)	abs. error
16	2.69%	2.7%	0.0012
32	3.96%	4.4%	0.0017
64	6.49%	9.2%	0.0017
128	11.57%	18.7%	0.0017

Table 3: Cost of the check as the dimension grows, Many-Well with

m=5

(

32

modes),

R=256

, three seeds. Accuracy is flat; cost grows about linearly in

d

because a full Hessian needs

O(d)

backward passes.

target	family	$M_{\mathrm{wt}}$	$M$	ESS	norm. gap	ours
GMM	tilt $\tau{=}1$	0.000	0.000	1.000	$-0.000$	0.006
	tilt $\tau{=}0.5$	0.083	0.083	0.959	$+0.001$	0.086
	tilt $\tau{=}0.25$	0.128	0.128	0.909	$+0.002$	0.131
	tilt $\tau{=}0$	0.180	0.180	0.843	$+0.003$	0.179
	tilt $\tau{=}2$	0.137	0.137	0.843	$-0.002$	0.138
	tilt $\tau{=}3$	0.235	0.235	0.498	$-0.004$	0.236

Table 4: Constructed collapse with exactly known decomposition. In the tilt family

M_{\mathrm{supp}}=0

, so by Corollary 1 the normalizer gap must be zero whatever

M_{\mathrm{wt}}

is: it is, and the SMC estimate is exactly

0.0000

in all twelve tilt rows. The mixed rows vary

M_{\mathrm{wt}}

at fixed

M_{\mathrm{supp}}

and are the cleaner demonstration – on the GMM the gap reads

0.104

three times while

M_{\mathrm{wt}}

grows from

0

0.059

. ESS, blind to

M_{\mathrm{supp}}

, does respond here. Search budget

R{=}1024

on the GMM and

R{=}256

on Many-Well, three repetitions.

Figure 4: The blind spot of the normalizer comparison. Within each panel every sampler reaches every basin (

M_{\mathrm{supp}}=0

) and only the proportions are wrong. The SMC estimate built on comparing normalising constants stays pinned at zero while more than half the target mass is misallocated, as Corollary 1 requires. ESS responds; the check tracks the truth.

	basins reached	$M_{\mathrm{supp}}$	$M_{\mathrm{wt}}$	$M$	check: abs. error
support collapse	1/32	0.569	0.000	0.569	0.0025
weight collapse	32/32	0.000	0.559	0.559	0.0057

Table 5: Two training runs of the same controlled-SDE architecture on the same target, with nearly equal missing mass and opposite decomposition. The

M_{\mathrm{supp}}

row is the audit of § 6 (five seeds); the

M_{\mathrm{wt}}

row differs only in the gradient clipping applied inside the control (two seeds). ESS, the ELBO and

\widehat{Z}_{q}

are undefined for both.

method	estimate	abs. error	cost
ours	0.9855	0.0001	$55.7\%$
parallel tempering, weights read directly ( $10^{3}$ steps)	0.9732	0.0122	$291.7\%$
parallel tempering, weights read directly ( $3\times 10^{3}$ steps)	0.9854	0.0000	$875.0\%$

Table 6: A non-normalizer competitor on Many-Well (

K=32

, true

M=0.9854

). Parallel tempering is not structurally blind the way the normalizer gap is: it reaches the right answer given enough steps. It costs

15.7\times

more to get there. Costs are relative to this sampler’s deliberately short training budget of

1.5\times 10^{6}

evaluations and are not comparable to the

2.7\%

of Table 2 , which is measured against the longer runs used there; what is comparable is the ratio within this table.

$\beta$	heaviest mode	#modes 90%	#modes 99%	captured by 500	by 2000
1	$57.5\pm 24.3\%$	$6.8\pm 3.3$	$15.2\pm 7.3$	$3.0\pm 2.9\%$	$23.4\pm 26.1\%$
2	$52.4\pm 25.2\%$	$7.2\pm 3.5$	$14.4\pm 6.9$	$3.5\pm 2.9\%$	$18.8\pm 18.6\%$
3	$64.5\pm 22.9\%$	$5.0\pm 2.9$	$12.0\pm 5.9$	$3.7\pm 4.0\%$	$13.1\pm 10.9\%$
5	$81.9\pm 15.7\%$	$2.0\pm 0.6$	$3.2\pm 1.6$	$63.3\pm 33.3\%$	$64.0\pm 33.7\%$
8	$100.0\pm 0.0\%$	$1.0\pm 0.0$	$1.0\pm 0.0$	$100.0\pm 0.0\%$	$100.0\pm 0.0\%$

Table 7: LJ-13. Concentration of the harmonic weight and the fraction of it captured by a cheap search, as a function of inverse temperature. Mean

\pm

s.d. over five independent enumerations of

6000

restarts each. Note the standard deviations in the last two columns: below

\beta=8

the captured fraction is not merely small but erratic.

Mode Collapse

Mode collapse is the failure of generative or inference systems to represent distinct valid outcomes in a target distribution.

See all 31 papers

Amortised adaptive importance sampling
Oct 12, 2026Arran Carter, Víctor Elvira, Esmeralda S. WhitammerImportance Sampling Adaptive Sampling
University of Edinburgh
One-Step Sampler for Boltzmann Distributions via Drifting
Mar 18, 2026Wenhan Cao, Keyu Yan, Lin ZhaoBoltzmann Sampling Sampling from Unnormalized Distributions
Department of Electrical and Computer Engineering, National University of Singapore
Fixed-point neural samplers on discrete spaces
Oct 1, 2026Jiajun He, Denis Blessing, Mouyang Cheng +2Diffusion Models
MIT · Microsoft Research New England

Oct 12, 2026J/K move · Enter open · S save

Amortised adaptive importance sampling

Arran Carter, Víctor Elvira, Esmeralda S. Whitammer

University of Edinburgh

Importance Sampling Adaptive Sampling Amortized Inference

Open paper AskPDF arXiv

Monte Carlo sampling algorithms aim to produce samples from a target distribution with a given unnormalised probability density function. The high cost of running Monte Carlo algorithms, with many simulation steps or particles needed to obtain a good empirical approximation to the target, can be reduced by introducing learnable components into the sampler -- a form of amortisation. Once these components are trained, amortised samplers can produce high-quality samples with less computation at sampling time than training-free algorithms would require. We propose an amortised sampler built upon the framework of adaptive importance sampling -- a hierarchical importance sampling procedure where the proposal is iteratively adapted. Our amortised adaptive importance sampling algorithm fits a lightweight neural network that adapts the proposal using information obtained over the course of sampling. By training our sampler over many different targets taken from the same family of distributions, we obtain a sampler that can adapt to a specific target at sampling time without a priori access to the target's parameters. We show that our procedure can efficiently sample from targets within the family the sampler was trained on, giving improvements over target-specific amortised samplers and training-free adaptive importance sampling algorithms. We also explore the potential for generalisation, where the trained sampler is used to sample from targets outwith its training family.

Abstract

Mode Collapse

Explore similar work

Amortised adaptive importance sampling

One-Step Sampler for Boltzmann Distributions via Drifting

Fixed-point neural samplers on discrete spaces