SCOPE Before You Sample: Conformal Scores Guide Posterior Exploration for Fast and Calibrated Bayesian Predictive Inference

Zahra Moslemi; Ziyi Liang; Babak Shahbaba

stat.MLOct 12, 2026

SCOPE Before You Sample: Conformal Scores Guide Posterior Exploration for Fast and Calibrated Bayesian Predictive Inference

Authors: Zahra Moslemi, Ziyi Liang, Babak Shahbaba

Organizations: University of California, Irvine · University of California, Santa Barbara

Abstract

Bayesian predictive regions account for parameter uncertainty but can be poorly calibrated under model misspecification, even with expensive full MCMC. Cheaper approximations such as variational inference reduce the cost but often yield less accurate predictive distributions. We propose SCOPE (Score-guided Conformal Posterior Exploration), which retains the predictive quality of full MCMC while achieving calibrated coverage at a fraction of its cost. The key observation is that a prediction region can depend on the posterior mainly through a few parameter directions that move the region's boundary, and these are generally not the directions of largest posterior variance. SCOPE gives nonconformity scores a new role in guiding posterior computation: score gradients on a held-out guide set, weighted toward the target boundary, identify the relevant directions. MCMC explores this subspace, while an inexpensive projected variational approximation carries the remaining uncertainty. The same guide set calibrates the region's final density level. We establish asymptotic coverage for SCOPE and provide a split variant with finite-sample validity. Controlled simulations show that score-guided exploration yields more accurate calibrated regions than non-targeted direction choices at low active rank. Across simulated and real benchmarks, SCOPE has a median test-NLL gap of only 0.002 nats relative to full Hamiltonian Monte Carlo (HMC), and on the real benchmarks it runs 12-15 times faster.

Figures & tables

Figure 1: SCOPE overview. For each candidate reference point, guide observations determine an active parameter subspace. Slice MCMC and projected variational completion produce a candidate predictive distribution. The candidate with the lower negative log-likelihood (NLL) on the guide set is selected, and a quantile of the guide scores determines its prediction region. SCOPE/Split uses an independent calibration set for the final threshold.

Figure 2: Density-level geometry in Simulation 2 (one seed). (a,b) SCOPE and CQR regions versus $x_{1}$ ; dots mark population global-level oracle boundaries. (c) At a disconnected input, SCOPE follows both modes whereas CQR spans the low-density gap.

	Guided	Uniform	Guided $-$ uniform [95% CI]
Simulation 1	13.0	16.7	$-3.70\;[-5.6,-1.8]$
Simulation 2	37.0	47.4	$-10.4\;[-15.6,-5.1]$

Table 1: Matched rank-

4

score-guidance comparison. Normalized symmetric-difference error (% of oracle-region length) to the population global-level oracle. Guided and uniform differ only in score weighting; VI/MAP anchor errors are averaged within seed. Difference is guided

-

uniform with multiplicity-adjusted paired 95% bootstrap CI over

20

seeds.

Figure 3: Main empirical summary. (a) Test-NLL excess relative to the HMC baseline across eight benchmarks (labels: median UCI runtime). (b) Predictive ESS/s relative to the HMC baseline on six UCI datasets. (c) Independently calibrated region size relative to HMC+Conf across eight benchmarks; SCOPE/Split is shown so panel (c) has standard split-conformal marginal guarantee. Points/bars are medians and whiskers are ranges.

Lower-cost methods
	Wall-clock / seed		NLL $-$ HMC ( $\downarrow$ )		Predictive ESS/s vs HMC ( $\uparrow$ )
Method	s	vs HMC	median	worst	median point	slowest point
Laplace	71	17 $\times$	+0.062	+0.938	iid	iid
Trajectory Gaussian	91	13 $\times$	+0.087	+0.986	iid	iid
MFVI	94	12 $\times$	+0.176	+0.585	iid	iid
Trajectory PCA	92	12 $\times$	+0.210	+4.152	1.7 $\times$	2.9 $\times$

Table 2: Bayesian predictive comparison (20 matched seeds). Predictive summaries use the eight main benchmarks; runtime and predictive ESS/s use the six main UCI datasets. SCOPE includes both candidate fits and converged-MAP construction; baselines charge a single selected fit. NLL excess is test NLL minus HMC. ESS/s is for log predictive density relative to HMC, reported at the median and slowest-mixing test point and then summarised across the six UCI datasets; parentheses give the active MCMC chain of SCOPE without completion. Lower-cost methods take at most a tenth of HMC’s median wall-clock. Bold marks the best lower-cost summary, not statistical significance. Full per-benchmark values and region comparisons are in the appendix.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$\mathcal{D}_{\mathrm{train}},\mathcal{D}_{\mathrm{guide}},\mathcal{D}_{\mathrm{conf}},\mathcal{D}_{\mathrm{test}}$	training, guide, independent calibration, test sets
$n,\ m_{g},\ m_{c}$	sizes of $\mathcal{D}_{\mathrm{train}}$ , $\mathcal{D}_{\mathrm{guide}}$ , $\mathcal{D}_{\mathrm{conf}}$
$\rho_{\theta}(x,y)$	density nonconformity score $-\log p_{\theta}(y\mid x)$
$\theta_{\mathrm{VI}},\theta_{\mathrm{MAP}}$	candidate local references (VI mean, converged MAP)
$\theta_{\star}$	guide-selected reference, eq. ( 9 )
$\Sigma_{\mathrm{VI}}$	diagonal VI covariance; residual scales

Appendix

Table 3: Core notation.

Dataset	Role	Raw $n$	Analysis $n$	$d$	Train	Guide	Conf	Subgroup indicator
Concrete	main	1030	1030	8	515	206	154	age $>90$ days (predictor)
Energy	main	768	768	8	384	154	115	top-decile heating load (response)
Yacht	main	308	308	6	154	62	46	Froude number $>0.40$ (predictor)
Wine	main	6497	6497	11	3248	1299	975	quality $\geq 7$ (response)
Abalone	main	4177	4177	10	2088	835	627	rings $\geq 15$ (response)
Superconductivity	main	21263	2000	81	1000	400	300	$T_{c}>77$ K (response)

Appendix

Table 4: Real-data benchmarks. Splits are

50/20/15/15

of the analysis sample per seed, standardised on

\mathcal{D}_{\mathrm{train}}

only. The subgroup column gives the registry’s “critical” indicator; only predictor-defined indicators are used as prediction-time subgroups in the main text.

Component	Hyperparameter	Default
Candidate references	anchors	VI mean, converged MAP
Anchor choice	criterion	untempered guide NLL, eq. ( 9 )
Converged MAP	optimiser	full-batch Adam, $\leq 4000$ steps, then L-BFGS, 300 steps
VI	family / iterations / lr	diagonal Gaussian / $6000$ / $2\times 10^{-3}$
Boundary localisation	$c_{h}$ , $h_{\min}$	$0.70$ , $10^{-3}$
Fisher/GN metric	training-row budget	$512$ ( $2{,}048$ on Sim. 2)

Appendix

Table 5: Default SCOPE settings used for every benchmark and seed.

Benchmark	SCOPE	HMC	Size ratio	$\Delta$ NLL	SCOPE (s)	HMC/SCOPE	Coord.	Task-mESS
	Coverage				Wall-clock		ESS/s ratio
Sim. 1	0.902	0.894	1.03	-0.008	–	–	–	–
Sim. 2	0.904	0.906	0.99	-0.034	–	–	–	–
Concrete	0.904	0.890	1.18	+0.193	83	12 $\times$	279 $\times$	139 $\times$
Energy	0.918	0.895	1.17	-0.071	77	15 $\times$	416 $\times$	188 $\times$
Yacht	0.907	0.898	1.00	-0.211	71	14 $\times$	52 $\times$	52 $\times$

Appendix

Table 6: SCOPE versus full HMC: two simulations and six main UCI datasets (20 matched seeds). SCOPE region: guide-reused level on the untempered predictive; HMC

+

Conf: the same density-score construction on the HMC draws, calibrated on

\mathcal{D}_{\mathrm{conf}}

. Size ratio is SCOPE/HMC

+

Conf;

\Delta

NLL is SCOPE minus HMC (nats). Runtime and ESS/s comparisons use only the six main UCI datasets and the inference-time accounting of Table 2 , including both SCOPE candidate fits after shared initialisation. The time ratio is HMC/SCOPE; ESS/s ratios are SCOPE/HMC. Ratios use unrounded dataset means. The recorded task-mESS estimates use an own-basis fallback when the common-basis estimate is non-finite; the matched H1 table instead uses common-basis summary fields (Appendix C.7 ). Simulation timing cells are omitted because final selected-anchor runs and frozen baselines had different execution loads.

Method	Sim. 1	Sim. 2	Concrete	Energy	Yacht	Wine	Abalone	Supercond.	Airfoil	Naval	Protein
MFVI	$0.375$	$1.453$	$3.135$	$1.152$	$2.278$	$1.053$	$2.040$	$3.843$	$2.165$	$-4.805$	$3.008$
Laplace	$0.488$	$1.376$	$3.257$	$0.887$	$1.495$	$1.069$	$2.047$	$4.491$	$2.156$	$-5.976$	$3.000$
Trajectory Gaussian (Adam; SWAG-style)	$0.510$	$1.920$	$3.136$	$0.839$	$1.587$	$1.067$	$2.032$	$4.539$	$2.210$	$-5.866$	$2.990$
Trajectory-PCA subspace MCMC	$0.632$	$2.363$	$3.451$	$0.845$	$1.517$	$1.192$	$2.047$	$7.705$	$2.209$	$-6.063$	$3.140$
Deep ensemble	$0.473$	$1.346$	$3.055$	$\mathbf{0.807}$	$1.486$	$1.035$	$2.036$	$4.847$	$2.049$	$-5.997$	$3.009$
SGLD	$0.507$	$1.718$	$3.280$	$1.832$	$2.434$	$1.114$	$2.056$	$5.387$	$2.458$	$-4.379$	$3.253$

Appendix

Table 7: Test NLL: two simulations and nine UCI datasets (11 benchmarks; means over 20 seeds; lower is better). Bold: best in the column, full HMC included (the secondary SCOPE/MHDR output is excluded from the comparison); the shaded row is the SCOPE predictive behind the primary region. Simulation results use the anchor protocol specified in Appendix C .

Method	Sim. 1	Sim. 2	Concrete	Energy	Yacht	Wine	Abalone	Supercond.	Airfoil	Naval	Protein
CQR	$0.900$	$0.904$	$0.893$	$0.895$	$0.913$	$0.897$	$0.898$	$0.900$	$0.904$	$0.895$	$0.894$
Abs.-residual conformal	$0.899$	$0.904$	$0.910$	$0.903$	$0.918$	$0.895$	$0.900$	$0.903$	$0.892$	$0.910$	$0.895$
Norm. conformal	$0.897$	$0.901$	$0.895$	$0.891$	$0.924$	$0.891$	$0.899$	$0.895$	$0.898$	$0.898$	$0.892$
GBM-CQR	$0.896$	$0.907$	$0.892$	$0.900$	$0.900$	$0.906$	$0.906$	$0.906$	$0.897$	$0.893$	$0.904$
MFVI + Conf	$0.891$	$0.902$	$0.898$	$0.886$	$0.904$	$0.894$	$0.899$	$0.902$	$0.896$	$0.899$	$0.894$
Laplace + Conf	$0.895$	$0.904$	$0.894$	$0.902$	$0.911$	$0.890$	$0.898$	$0.899$	$0.892$	$0.900$	$0.891$

Appendix

Table 8: Nominal-90% coverage: two simulations and nine UCI datasets (means over 20 seeds). SCOPE uses the guide-reused region; baselines use independent split calibration. Coverage is assessed relative to 0.90.

Method	Sim. 1	Sim. 2	Concrete	Energy	Yacht	Wine	Abalone	Supercond.	Airfoil	Naval	Protein
CQR	$1.59\,(0.900)$	$4.96\,(0.904)$	$18.42\,(0.893)$	$1.75\,(0.895)$	$3.59\,(0.913)$	$2.25\,(0.897)$	$6.61\,(0.898)$	$43.33\,(0.900)$	$6.07\,(0.904)$	$0.0032\,(0.895)$	$14.85\,(0.894)$
Abs.-residual conformal	$1.67\,(0.899)$	$7.72\,(0.904)$	$21.31\,(0.910)$	$1.97\,(0.903)$	$5.75\,(0.918)$	$2.34\,(0.895)$	$7.42\,(0.900)$	$57.09\,(0.903)$	$6.28\,(0.892)$	$0.0024\,(0.910)$	$17.66\,(0.895)$
Norm. conformal	$1.31\,(0.897)$	$6.87\,(0.901)$	$20.22\,(0.895)$	$5.98\,(0.891)$	$11.32\,(0.924)$	$2.21\,(0.891)$	$6.03\,(0.899)$	$51.39\,(0.895)$	$9.45\,(0.898)$	$0.0108\,(0.898)$	$15.17\,(0.892)$
GBM-CQR	$1.26\,(0.896)$	$4.88\,(0.907)$	$26.09\,(0.892)$	$4.14\,(0.900)$	$7.49\,(0.900)$	$\mathbf{2.06\,(0.906)}$	$6.85\,(0.906)$	$49.17\,(0.906)$	$11.62\,(0.897)$	$0.0412\,(0.893)$	$16.15\,(0.904)$
MFVI + Conf	$\mathbf{1.17\,(0.891)}$	$3.21\,(0.902)$	$18.11\,(0.898)$	$2.18\,(0.886)$	$11.02\,(0.904)$	$2.19\,(0.894)$	$6.00\,(0.899)$	$43.99\,(0.902)$	$7.03\,(0.896)$	$0.0061\,(0.899)$	$15.28\,(0.894)$
Laplace + Conf	$1.26\,(0.895)$	$3.31\,(0.904)$	$19.73\,(0.894)$	$1.77\,(0.902)$	$3.65\,(0.911)$	$2.19\,(0.890)$	$5.98\,(0.898)$	$40.06\,(0.899)$	$6.44\,(0.892)$	$0.0015\,(0.900)$	$14.99\,(0.891)$

Appendix

Table 9: Nominal-90% region size: two simulations and nine UCI datasets (coverage in parentheses). Baselines calibrate their scores on

\mathcal{D}_{\mathrm{conf}}

; SCOPE reuses

\mathcal{D}_{\mathrm{guide}}

. Bold denotes the smallest mean size, including HMC

+

Conf; size should be interpreted together with coverage.

Method	Coverage max $\|$ dev. $\|$	Size / HMC+Conf	Sim. 2 $\triangle$ oracle
CQR	0.013	1.14 [0.81, 1.72]	3.01
Abs.-residual conformal	0.018	1.32 [1.09, 2.68]	5.92
Norm.-residual conformal	0.024	1.42 [1.02, 3.68]	5.06
GBM-CQR	0.008	1.57 [0.96, 2.55]	2.89
MALA+Conf	0.014	1.04 [1.00, 1.14]	1.23
SCOPE/Split	0.011	1.02 [1.00, 1.28]	0.83

Appendix

Table 10: Finite-sample prediction-set comparison on the eight main benchmarks. Every method uses an independent calibration split and therefore has the standard finite-sample marginal guarantee under exchangeability. Coverage reports the maximum absolute deviation from 0.90; size is relative to HMC+Conf (median [range]); the last column is mean symmetric-difference length from the population global-level oracle region on Simulation 2. GBM-CQR is gradient-boosted CQR.

	Anchor			Guide NLL			Test NLL					Size / HMC $+$ Conf
Benchmark	RMSE short	RMSE conv.	conv. time (s)	VI	MAP	MAP chosen	VI	MAP	selected	envelope	penalty	before	after
Concrete	6.647	5.806	6.0	3.145	3.467	5%	3.138	3.327	3.137	3.138	-0.002	1.23	1.18
Energy	2.057	0.539	5.5	1.185	0.978	100%	1.204	0.810	0.810	0.810	0.000	1.89	1.17
Yacht	3.332	1.176	5.2	2.261	1.606	100%	2.253	1.482	1.482	1.482	0.000	2.45	1.00
Wine	0.700	0.701	6.2	1.046	1.065	5%	1.051	1.077	1.051	1.051	0.000	1.04	1.04
Abalone	2.112	2.108	7.7	2.009	2.002	70%	2.040	2.038	2.037	2.038	-0.001	1.02	1.01

Appendix

Table 11: Anchor selection on all nine UCI datasets (20 seeds). Short is the shared staged initialisation, including its 80-epoch MAP stage; converged continues it with full-batch Adam and L-BFGS on

\mathcal{D}_{\mathrm{train}}

only. Guide NLL evaluates each candidate’s untempered completed predictive on

\mathcal{D}_{\mathrm{guide}}

; MAP chosen is the fraction of seeds selecting the converged-MAP candidate. The selected test NLL is the primary SCOPE result in Table 7 . The envelope is the better of the two fixed-anchor strategies in mean test NLL, a retrospective comparison, and penalty is selected minus envelope. The last two columns give guide-reused region size relative to HMC

+

Conf before and after selection.

Simulation	Rank	Uniform	Guided	Guided $-$ uniform [95% CI]
Simulation 1	2	21.4	15.0	$-6.4\;[-9.2,-3.5]$
	4	16.7	13.0	$-3.7\;[-5.6,-1.8]$
	8	13.7	11.5	$-2.2\;[-3.4,-1.0]$
	16	11.3	10.3	$-1.0\;[-1.8,-0.2]$
	32	10.2	9.9	$-0.3\;[-0.9,+0.3]$
Simulation 2	2	57.1	40.0	$-17.1\;[-24.0,-10.2]$

Appendix

Table 12: Score-guidance benefit across active rank. Normalized symmetric-difference error (%) relative to the population global-level oracle region for guided and uniform weighting under the fixed-anchor protocol. Within each seed, the VI- and MAP-anchor errors are averaged before comparison. Differences are guided minus uniform, with multiplicity-adjusted paired 95% bootstrap intervals over 20 matched seeds. Lower is better.

	Coverage			Size			NLL			TV			Model-region cov.
Benchmark	active	unguarded	completed	active	unguarded	completed	active	unguarded	completed	active	unguarded	completed	active	unguarded	completed
Sim. 1	0.900	0.902	0.902	1.20	1.20	1.21	0.379	0.378	0.368	0.069	0.070	0.070	0.903	0.913	0.913
Concrete	0.905	0.901	0.904	18.67	18.89	18.83	3.138	3.154	3.137	0.178	0.205	0.193	0.892	0.925	0.918
Energy	0.917	0.902	0.918	1.84	4.26	1.90	0.836	1.984	0.810	0.132	0.537	0.109	0.956	1.000	0.983
Yacht	0.913	0.903	0.907	3.91	13.84	4.44	1.523	2.831	1.482	0.213	0.524	0.109	0.941	1.000	0.984
Wine	0.897	0.899	0.898	2.22	2.23	2.23	1.057	1.053	1.051	0.060	0.064	0.061	0.898	0.909	0.904

Appendix

Table 13: Completion ablation (H2): Simulation 1 and six main UCI datasets; matched active states and seeds. Active-only uses the retained active states without a complement (

K=1

); unguarded adds the projected VI complement without the energy guard; completed is the guarded SCOPE predictive. The completed NLLs match the primary SCOPE results in Table 7 . Coverage and size use the guide-reused region. NLL, TV (to the oracle or HMC reference), and model-region coverage use the untempered predictive.

	Coverage					Size
Benchmark	SCOPE	SCOPE/Split	gap	Split (tempered)	HMC $+$ Conf	SCOPE	SCOPE/Split	ratio	HMC $+$ Conf
Sim. 1	0.902	0.893	+0.009	0.893	0.894	1.21	1.17	1.026	1.17
Sim. 2	0.904	0.905	-0.001	0.909	0.906	2.86	2.89	0.989	2.88
Concrete	0.904	0.898	+0.006	0.899	0.890	18.83	18.33	1.027	15.90
Energy	0.918	0.895	+0.023	0.898	0.895	1.90	1.80	1.054	1.63
Yacht	0.907	0.911	-0.004	0.913	0.898	4.44	4.42	1.004	4.42

Appendix

Table 14: Guide reuse versus an independent calibration split (20 seeds). SCOPE (level from

\mathcal{D}_{\mathrm{guide}}

) and SCOPE/Split (level from

\mathcal{D}_{\mathrm{conf}}

) on the same untempered predictive; ‘gap’ is the empirical paired coverage difference SCOPE

-

SCOPE/Split; it is not an estimate of the theoretical coverage penalty in Theorem 1 . SCOPE/Split (tempered) calibrates and evaluates the same guide-tempered predictive. HMC

+

Conf is shown for reference.

	Concrete ( $m_{g}^{\mathrm{full}}=206$ )				Energy ( $m_{g}^{\mathrm{full}}=154$ )				Supercond. ( $m_{g}^{\mathrm{full}}=400$ )				Yacht ( $m_{g}^{\mathrm{full}}=62$ )
$m_{g}$	SCOPE	Split	size	overlap	SCOPE	Split	size	overlap	SCOPE	Split	size	overlap	SCOPE	Split	size	overlap
25	0.912	0.893	1.13	0.10	0.918	0.894	1.14	0.14	0.931	0.902	1.17	0.08	0.917	0.911	1.07	0.29
50	0.909	0.895	1.06	0.25	0.909	0.892	1.05	0.25	0.908	0.905	1.06	0.10	0.907	0.912	1.00	0.60
62	–	–	–	–	–	–	–	–	–	–	–	–	0.907	0.911	1.00	–
100	0.896	0.900	1.01	0.47	0.904	0.893	1.03	0.50	0.904	0.904	1.00	0.21	–	–	–	–
154	–	–	–	–	0.918	0.895	1.05	–	–	–	–	–	–	–	–	–

Appendix

Table 15: Guide-size study (H3): the whole SCOPE pipeline rerun on random subsamples of

\mathcal{D}_{\mathrm{guide}}

of size

m_{g}

(20 seeds). For each size: coverage of the guide-reused region (SCOPE), coverage of the split-conformal region on the untouched

\mathcal{D}_{\mathrm{conf}}

(Split) from the same run, the size ratio SCOPE/Split, and the mean squared projection of the active basis at

m_{g}

onto the basis obtained with the full guide set at the same anchor (overlap). Anchor selection, geometry, pilot refinement and level all use only the

m_{g}

points.

	Boundary-functional ESS/s						Predictive-probe ESS/s
Benchmark	SCOPE	PCA	MALA	HMC	vs MALA	vs HMC	SCOPE	PCA	MALA	HMC	vs MALA	vs HMC	$n_{\mathrm{HMC}}$
Sim. 1	19.70	9.21	–	–	–	–	23.11	17.95	–	–	–	–	–
Sim. 2	4.72	–	–	–	–	–	–	–	–	–	–	–	–
Concrete	34.23	25.28	0.35	0.25	96 $\times$	135 $\times$	43.80	44.14	0.77	0.62	57 $\times$	71 $\times$	20
Energy	36.33	34.51	0.36	0.56	101 $\times$	65 $\times$	39.27	40.93	1.06	0.93	37 $\times$	42 $\times$	20
Yacht	38.84	31.78	0.43	1.16	89 $\times$	33 $\times$	40.35	42.15	1.11	1.51	36 $\times$	27 $\times$	20

Appendix

Table 16: Chain efficiency for common predictive functionals (per fit second; means over seeds). Boundary: ESS of the common boundary functional defined in Appendix C.7 (mean smoothed score membership on guide observations at a fixed reference threshold). Probe: 90th percentile over 32 fixed test probes of the ESS of

\log p_{s}(y^{\star}_{j}\mid x^{\star}_{j})

along retained draws. SCOPE is the guide-selected run; PCA the matched control at the converged-MAP anchor; MALA is rerun at its guide-selected anchor with the published seeds; HMC is the reference chain rerun on

n_{\mathrm{HMC}}

seeds (matched seeds and sampler configuration). MALA-relative ratios cover all nine UCI datasets; HMC-relative ratios cover only the six main UCI datasets. All estimates use the initial-positive-sequence estimator.

	min over coordinates				10th percentile				median				90th percentile
Benchmark	SCOPE	HMC	/HMC	/MALA	SCOPE	HMC	/HMC	/MALA	SCOPE	HMC	/HMC	/MALA	SCOPE	HMC	/HMC	/MALA
Concrete	10.2	0.010	994 $\times$	142 $\times$	10.9	0.015	736 $\times$	141 $\times$	13.6	0.030	453 $\times$	118 $\times$	15.8	0.057	278 $\times$	73 $\times$
Energy	11.7	0.009	1303 $\times$	161 $\times$	12.7	0.012	1024 $\times$	158 $\times$	16.2	0.025	647 $\times$	142 $\times$	20.0	0.048	412 $\times$	112 $\times$
Yacht	21.4	0.358	60 $\times$	154 $\times$	21.8	0.372	59 $\times$	151 $\times$	23.5	0.426	55 $\times$	141 $\times$	25.5	0.488	52 $\times$	118 $\times$
Wine	4.2	0.015	270 $\times$	75 $\times$	4.9	0.022	222 $\times$	75 $\times$	6.1	0.040	150 $\times$	58 $\times$	7.3	0.065	112 $\times$	40 $\times$
Abalone	5.3	0.016	340 $\times$	106 $\times$	6.2	0.023	272 $\times$	111 $\times$	7.9	0.050	159 $\times$	91 $\times$	9.7	0.091	107 $\times$	64 $\times$

Appendix

Table 17: Sensitivity of coordinate ESS/s to the coordinate summary. For each seed, ESS is summarised at the stated percentile over the 20 common-basis coordinates; the mean of these counts is divided by mean wall-clock. SCOPE uses the two-candidate inference time of Table 2 . The table reports these rates and their SCOPE/HMC and SCOPE/MALA ratios. Only UCI benchmarks with comparable timing cohorts are included. The main comparison uses the same 90th coordinate percentile, but averages seedwise ESS/s rates before taking method ratios, so its values need not equal this table’s ratios of means (Appendix C.7 ). These summaries assess autocorrelation in the common basis, not accuracy of the posterior approximation.

(a) Median over test points
	Effective draws of $2{,}000$						ESS/s relative to HMC
	SCOPE						SCOPE
Benchmark	Full	MCMC only	HMC	MALA	SGLD	PCA	Full	MCMC only	MALA	SGLD	PCA
Sim. 1	1,622	1,323	1,676	32	84	18	16 $\times$	13 $\times$	0.43 $\times$	1.2 $\times$	0.39 $\times$
Sim. 2	1,867	1,184	123	76	99	43	47 $\times$	30 $\times$	2.3 $\times$	36 $\times$	4.7 $\times$
Concrete	1,757	1,254	382	17	94	22	56 $\times$	40 $\times$	0.98 $\times$	5.4 $\times$	1.8 $\times$

Appendix

Table 18: Predictive ESS for every chain on the eight main benchmarks (20 seeds). For each seed and method, the initial-positive-sequence ESS of

\log p(y_{j}\mid x_{j},\theta_{s})

is computed along the

2{,}000

retained draws at every test point

j

and summarised by its median over the test set in (a) and by its minimum, the slowest-mixing test point, in (b). Effective draws are means over seeds. ESS/s divides the mean count by the method’s mean wall-clock, and the ratio is taken to the same quantity for HMC, so (b) reports the relative min-ESS/s. UCI rates charge both SCOPE candidate fits and converged-MAP construction, and one selected baseline fit, excluding shared initialisation and test evaluation. Simulation ratios use reconstructed SCOPE times: recorded single-anchor time multiplied by the measured full-to-single-fit cost ratio (

1.9

and

2.3

). They are descriptive estimates under differing machine loads and are excluded from headline efficiency summaries. Each SCOPE draw combines a state of its MCMC chain in the score-guided subspace with an independent draw of the remaining coordinates from the variational completion. Full : SCOPE’s draws as reported; MCMC only : the chain states alone, with the remaining coordinates fixed at the anchor and no completion draws. Laplace, MFVI and the trajectory Gaussian draw independently, so their ESS equals the number of draws by construction. Dashes denote unavailable slowest-point records.

Benchmark	Size ratio	$\Delta$ coverage	$\Delta$ NLL	Task-mESS/s ratio	$n_{\rm H1}$
Sim. 1	1.03 [1.00, 1.05]	+0.008 [-0.001, +0.017]	-0.008 [-0.021, +0.005]	7.49 [6.19, 8.95]	18
Sim. 2	0.99 [0.98, 1.00]	-0.002 [-0.009, +0.005]	-0.034 [-0.051, -0.017]	–	–
Concrete	1.18 [1.14, 1.23]	+0.014 [+0.003, +0.025]	+0.193 [+0.173, +0.213]	3.81 [2.87, 4.93]	12
Energy	1.16 [1.10, 1.22]	+0.023 [+0.006, +0.040]	-0.071 [-0.085, -0.057]	3.95 [2.77, 5.79]	18
Yacht	1.00 [0.94, 1.05]	+0.009 [-0.014, +0.029]	-0.211 [-0.236, -0.186]	3.51 [2.02, 6.29]	18
Wine	1.04 [1.02, 1.05]	+0.005 [+0.001, +0.010]	+0.023 [+0.019, +0.027]	5.86 [4.75, 7.24]	17

Appendix

Table 19: Seed-level paired comparisons: mean paired difference or geometric-mean ratio with 95% bootstrap confidence intervals. Size and coverage compare SCOPE with HMC

+

Conf; NLL compares SCOPE with HMC. Differences are SCOPE minus the reference, and size is SCOPE/reference. The first three comparisons use 20 paired seeds. The task-mESS/s ratio compares SCOPE geometry with matched trajectory PCA (H1), using only the

n_{\rm H1}

finite paired log-ratios shown; non-finite contrasts are excluded. Each contrast pairs methods on the same seed, split and initialisation. The small available sample counts limit the precision of some efficiency comparisons.

Figure 4: Sensitivity of SCOPE on the six main UCI datasets (

20

seeds each). (a) Realised minus nominal coverage after rerunning the complete procedure at nominal levels

0.80

0.90

, and

0.95

; filled markers denote the guide-reused SCOPE region and hollow markers denote SCOPE/Split. (b) Test-NLL change under boundary-bandwidth and kernel-shape perturbations. (c,d) Sensitivity to fixed active rank, normalised by each dataset’s deployed spectral-mass construction: TV distance to the finite-budget HMC predictive (c; lower indicates closer agreement with the HMC baseline) and coordinate ESS/s (d; higher is better; log scale). Dashed lines mark the corresponding default. The rank panels vary rank within the guided construction and are distinct from the matched-rank guided-versus-uniform experiment in Appendix D.3 .

Benchmark	$n$	Main	Alt.	Main	Alt.	Main	Alt.	s
		NLL		Coverage		Region size		Alt. chain
Concrete	10	2.944	2.953	0.890	0.901	15.90	15.90	1434
Energy	10	0.881	0.883	0.895	0.879	1.63	1.57	1294
Yacht	10	1.693	1.715	0.898	0.893	4.42	4.64	1266
Supercond.	10	3.553	3.548	0.896	0.898	32.70	32.59	1559

Appendix

Table 20: Sensitivity to the HMC baseline configuration. The main HMC columns are the 20-seed summaries used in the primary tables. The alternative configuration uses diagonal VI-based preconditioning and three jittered chains with the same per-chain trajectory budget;

n

gives the number of alternative-configuration runs. Comparisons are descriptive because the two summaries use different run cohorts. Chain time is per alternative HMC chain.

	Simulation 1						Simulation 2
Method	TV	KL	RMSE $\mu$	RMSE $\sigma$	sym. diff.	cov.	TV	KL	RMSE $\mu$	RMSE $\sigma$	sym. diff.	cov.
MFVI	0.065	0.020	0.062	0.042	0.112	0.891	0.276	0.400	0.647	0.322	1.204	0.902
Laplace	0.155	0.142	0.164	0.066	0.264	0.895	0.276	0.315	0.622	0.308	1.353	0.904
Trajectory Gaussian (Adam; SWAG-style)	0.166	0.161	0.177	0.063	0.293	0.894	0.354	0.858	0.717	0.374	1.486	0.903
Trajectory-PCA subspace MCMC	0.207	0.287	0.196	0.085	0.351	0.895	0.404	1.279	0.796	0.424	1.713	0.902
Deep ensemble	0.145	0.118	0.140	0.059	0.247	0.892	0.236	0.289	0.479	0.246	1.012	0.907

Appendix

Table 21: Oracle comparisons on the simulations (20 seeds). TV and KL of the test predictive to the true conditional law, RMSE of the predictive mean and standard deviation to their oracle values, and the symmetric difference between each method’s calibrated region and the population global-level oracle region (SCOPE: guide-reused region; others: split-conformal on

\mathcal{D}_{\mathrm{conf}}

), with realised coverage.

Benchmark	rank $r$	accept.	VI	pilot	main	compl.	fit total	end-to-end	energy ratio	guard $\gamma$
Sim. 1	23.4	0.59	–	–	–	–	–	–	0.87	1.00
Sim. 2	80.0	0.58	–	–	–	–	–	–	0.63	1.00
Concrete	16.4	0.59	9.4	2.5	29.3	0.5	43.1	83.3	1.86	0.96
Energy	7.0	0.58	8.7	2.3	28.2	0.6	40.9	77.2	25.10	0.21
Yacht	3.6	0.58	8.2	2.2	26.1	0.4	37.3	71.2	30.04	0.21
Wine	40.4	0.59	11.2	3.0	34.7	2.7	58.0	114.8	1.55	0.96

Appendix

Table 22: SCOPE run diagnostics (means over 20 seeds). Final active rank

r

(spectral-mass rule, capped at 80), main-chain acceptance, and wall-clock seconds for one fit. VI, pilot, main and completion show selected components; the fit total also includes active-geometry construction and other fit overhead. End-to-end inference time adds converged-MAP construction and the second candidate fit, charging shared VI once; common MAP initialisation and downstream evaluation are excluded. Timing is reported only for UCI tasks; the simulation candidate fits and frozen baselines were measured under different machine load. Energy ratio and guard scale are means over seeds of their within-fit medians:

D_{s}(1)/(d_{\perp}/2)

before the shrink-only guard and

\gamma_{s}

, respectively.

Construction	Mean error $\bar{E}_{m}$	Paired difference interval	Relative error reduction (%)
A. Integrated posterior reference; 40 datasets; absolute error
Boundary-weighted SCOPE	0.00579	–	–
Uniform weights	0.01779	$[-0.01517,-0.00869]$	$67.4\%$
Whitened random span	0.01990	$[-0.01736,-0.01074]$	$70.9\%$
Permuted boundary weights	0.01819	$[-0.01552,-0.00912]$	$68.1\%$
Same-span oracle Gaussian †	0.01316	$[-0.00938,-0.00525]$	$56.0\%$

Appendix

Table 23: Boundary weighting reduces final calibrated-region error. Mean error

\bar{E}_{m}

is the across-dataset mean of the per-dataset error defined above: symmetric-difference length in response units for A and within-dataset symmetric-difference length as a percentage of the reference region length for B. Paired differences are guided minus comparator, so negative values favor guided inference. A uses Bonferroni-adjusted paired bootstrap intervals targeting 95% familywise coverage for its four primary contrasts (

\dagger

: two secondary contrasts); B uses pointwise 95% paired bootstrap intervals. The final column reports

\mathrm{Red}_{c}=100(1-\bar{E}_{\mathrm{guided}}/\bar{E}_{c})

, the percentage reduction in mean region-approximation error of the guided construction relative to the row’s comparator; positive values favor guided inference. All reductions use unrounded means. A printed

-0.00

does not establish exclusion of zero.

Explore similar work

Oct 18, 2024

Predictive variational inference: Learn the predictively optimal posterior distribution

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

Jinlin Lai, Antonio Linero, Yuling Yao

College of Information and Computer Sciences, University of Massachusetts Amherst · Department of Statistics and Data Sciences, University of Texas, Austin

Model Misspecification Variational Inference

Ask PDF

Oct 8, 2026

Product Monte Carlo: Expectation Under a Product of Densities

Aggregating independently computed sub-sample inferences is a well-known challenge in large-scale Bayesian computation, federated learning, and opinion pooling. The core problem is to approximate a distribution whose density is proportional to a product of individual densities. We propose Product Monte Carlo (PMC), an algorithm for computing arbitrary expectations under a product of densities using only Monte Carlo draws from individual densities. We turn the intractable product expectation into a regression problem by expressing it through a family of computable conditional expectations. Besides the point estimate, our fully Bayesian regression formulation provides coherent uncertainty quantification and principled diagnostics. We establish theoretical guarantees on consistency and characterize convergence rates for the PMC estimator. Across simulated and real-data experiments varying the number, dimensionality, and heterogeneity of the component distributions, PMC achieves higher estimation accuracy, better-calibrated uncertainty coverage, and improved scalability at a lower computational cost than existing methods. In addition, PMC requires no density evaluation, making it applicable to simulation-based inference, as we demonstrate in an application with protein images.

Pietro Carlotti, Sifan Liu, Yuling Yao

University of Texas at Austin, Department of Statistics and Data Sciences · Duke University, Department of Statistical Science

Bayesian Inference

Ask PDF

May 5, 2026

Amortized Variational Inference for Joint Posterior and Predictive Distributions in Bayesian Uncertainty Quantification

Bayesian predictive inference propagates parameter uncertainty to quantities of interest through the posterior-predictive distribution. In practice, this is typically performed using a two-stage procedure: first approximating the posterior distribution of model parameters, and then propagating posterior samples through the predictive model via Monte Carlo simulation. This sequential workflow can be computationally demanding, particularly for high-fidelity models such as those governed by partial differential equations. We propose a variational Bayesian framework that directly targets the posterior-predictive distribution and jointly learns variational approximations of both the posterior and the corresponding predictive distribution. The formulation introduces a variational upper bound on the Kullback--Leibler divergence together with moment-based regularization terms. The variational distributions are trained in an amortized manner, shifting computational effort to an offline stage and enabling efficient online inference. Numerical experiments ranging from analytical benchmarks to a finite-element solid mechanics problem demonstrate that the proposed method achieves more accurate predictive distributions than conventional two-stage variational inference, while substantially reducing the cost of online predictive inference.

Nan Feng, Xun Huan

Department of Mechanical Engineering, University of Michigan, Ann Arbor, Michigan, United States

Uncertainty Quantification Amortized Inference Bayesian Inference Variational Inference

Ask PDF