Evaluating Bayesian deep learning for robust biomedical time-series classification under distribution shift: A sleep scoring case study

Mads Birch Sørensen; Ulrike Richter; Francois Gastambide; Michael Riis Andersen

cs.LGOct 26, 2026

Evaluating Bayesian deep learning for robust biomedical time-series classification under distribution shift: A sleep scoring case study

Authors: Mads Birch Sørensen, Ulrike Richter, Francois Gastambide, Michael Riis Andersen

Organizations: Technical University of Denmark, Denmark · H. Lundbeck A/S, Denmark

Abstract

In biomedical time-series classification, machine learning models often suffer from deteriorating accuracy and calibration due to distribution shift. We investigate to what degree Bayesian-inspired deep learning improves robustness through Bayesian model averaging (BMA). Using rodent sleep scoring as a case study, we evaluate BMA under distribution shifts in genotype, laboratory, and species. We systematically compare nine approximate inference methods spanning single- and multi-basin approaches across two neural network backbones on in-distribution (ID) data and five out-of-distribution (OOD) datasets. In our results, multi-basin methods perform best, outperforming baselines and single-basin methods. Compared with point estimates, BMA generally improves OOD accuracy and calibration across both architectures, while yielding only modest ID gains. It also often improves selective prediction and OOD and artifact detection, but, interestingly, not inter-rater disagreement detection.

Figures & tables

Method	$q_{m}(\boldsymbol{\theta})$
MAP	$\displaystyle\delta\!\left(\boldsymbol{\theta}-\boldsymbol{\theta}_{\mathrm{MAP}}\right)$
LLLA	$\displaystyle\delta\!\left(\boldsymbol{\theta}^{-\mathrm{LL}}-\boldsymbol{\theta}_{\mathrm{MAP}}^{-\mathrm{LL}}\right)\mathcal{N}\!\left(\boldsymbol{\theta}^{\mathrm{LL}}\mid\boldsymbol{\theta}_{\mathrm{MAP}}^{\mathrm{LL}},\boldsymbol{\Sigma}^{\mathrm{LL}}\right)$
SWA	$\displaystyle\delta\!\left(\boldsymbol{\theta}-\boldsymbol{\theta}_{\mathrm{SWA}}\right)$
SWAG	$\displaystyle\mathcal{N}\!\left(\boldsymbol{\theta}\mid\boldsymbol{\theta}_{\mathrm{SWA}},\boldsymbol{\Sigma}_{\mathrm{SWAG}}\right)$
VGD	$\displaystyle\mathcal{N}\!\left(\boldsymbol{\theta}\mid\boldsymbol{\mu},\sigma^{2}\mathbf{I}\right)$
MFVI	$\displaystyle\mathcal{N}\!\left(\boldsymbol{\theta}\mid\boldsymbol{\mu},\operatorname{diag}\!\left(\boldsymbol{\sigma}^{2}\right)\right)$

Table 1: Method-specific parameter distributions

q_{m}(\boldsymbol{\theta})

used in Eq. ( 3 ).

\theta^{LL}

and

\theta^{-\mathrm{LL}}

denote the last-layer and non-last-layer parameters, respectively. Point estimates are represented as Dirac measures

\delta

Cohort	Use	Species	$N_{\mathrm{rec}}$	Primary shift
WT	Train	Mouse	169	—
WT	Val.	Mouse	36	—
WT	ID test	Mouse	36	None
NT1	OOD test	Mouse	28	Genotype
spindle -A	OOD test	Mouse	4	Lab
spindle -D	OOD test	Mouse	6	Lab

Table 2: Datasets and evaluation domains.

N_{\mathrm{rec}}

denotes the number of 24 h recordings. WT is the source domain; NT1 and the spindle cohorts are held-out OOD cohorts. The final column reports the primary shift relative to WT; cohorts may also differ in acquisition system, signal montage, and annotation protocol.

U-Sleep
Model	MF1 $\uparrow$	$\kappa$ $\uparrow$	LPD $\uparrow$	Brier $\downarrow$
MAP	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.186\,{\scriptstyle\pm.048}$	$9.4\,{\scriptstyle\pm 2.6}$
Temp.	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.185\,{\scriptstyle\pm.046}$	$9.4\,{\scriptstyle\pm 2.5}$
SWA	$91.5\,{\scriptstyle\pm 0.3}$	$91.4\,{\scriptstyle\pm 0.7}$	${\scriptstyle-}.141\,{\scriptstyle\pm.009}$	$7.1\,{\scriptstyle\pm 0.5}$
SWAG	$91.7\,{\scriptstyle\pm 0.2}$	$91.6\,{\scriptstyle\pm 0.5}$	${\scriptstyle-}.138\,{\scriptstyle\pm.005}$	$6.9\,{\scriptstyle\pm 0.3}$
Laplace	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.186\,{\scriptstyle\pm.048}$	$9.4\,{\scriptstyle\pm 2.6}$

Table 3: In-distribution accuracy and calibration on the WT test set. Accuracy-type metrics in %, LPD in nats; mean

\pm

95% CI over training seeds (

t

-based); for DE, MultiSWA and MultiSWAG, mean

\pm

half the 2.5–97.5% interval over 100 bootstrap draws of ten members; bold = best in the column over all models, underline = best within its section (U-Sleep, CNN-GRU, Other baselines).

U-Sleep
Model	MF1 $\uparrow$	$\kappa$ $\uparrow$	LPD $\uparrow$	Brier $\downarrow$
MAP	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.481\,{\scriptstyle\pm.149}$	$24.2\,{\scriptstyle\pm 6.7}$
Temp.	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.460\,{\scriptstyle\pm.122}$	$23.9\,{\scriptstyle\pm 6.3}$
SWA	$78.0\,{\scriptstyle\pm 3.2}$	$74.3\,{\scriptstyle\pm 5.9}$	${\scriptstyle-}.435\,{\scriptstyle\pm.105}$	$21.7\,{\scriptstyle\pm 4.8}$
SWAG	$79.2\,{\scriptstyle\pm 2.8}$	$75.7\,{\scriptstyle\pm 5.3}$	${\scriptstyle-}.376\,{\scriptstyle\pm.060}$	$19.8\,{\scriptstyle\pm 3.6}$
Laplace	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.481\,{\scriptstyle\pm.149}$	$24.2\,{\scriptstyle\pm 6.7}$

Table 4: Out-of-distribution performance , each metric averaged over the five OOD cohorts (NT1 + SleepyRat A–D), computed per training seed; mean

\pm

95% CI over training seeds (

t

-based); for DE, MultiSWA and MultiSWAG, mean

\pm

half the 2.5–97.5% interval over 100 bootstrap draws of ten members of that average. bold = best in the column over all models, underline = best within its section.

Figure 1: ID versus mean OOD LPD for U-Sleep and CNN-GRU (higher is better) across seeds. Black crosses denote the corresponding MAP models, and each dot represents a seed.

	Single solution/trajectory								Multi-basin			Other baselines
Scope	MAP	Temp.	SWA	SWAG	Laplace	VGD	MFVI	SGLD	DE	MultiSWA	MultiSWAG	Somnotate	Spindle
ID	17.5	16.5	6.5	6.5	18.5	15.0	19.5	12.0	9.0	2.5	3.0	23.0	24.0
OOD	18.1	16.0	13.0	9.1	18.5	12.6	13.5	9.9	6.2	6.8	4.7	23.6	19.6
ID + OOD	18.0	16.1	11.9	8.7	18.5	13.0	14.5	10.2	6.7	6.1	4.4	23.5	20.3

Table 5: Average LPD rank. Models were ranked jointly within each cohort, and ranks were averaged over cohorts and backbones (

1=

best; lower is better). Other baselines are ranked individually. Bold and underlining denote the best overall and within each model group, respectively.

U-Sleep
	AUROC $\uparrow$			AURC $\downarrow$
Model	OOD	Artifact	Disagree	Reject
MAP	$.72\,{\scriptstyle\pm.06}$	$.44\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.02}$	$.054\,{\scriptstyle\pm.015}$
Temp.	$.71\,{\scriptstyle\pm.06}$	$.44\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.02}$	$.054\,{\scriptstyle\pm.015}$
SWA	$.80\,{\scriptstyle\pm.03}$	$.49\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.01}$	$.049\,{\scriptstyle\pm.019}$
SWAG	$.82\,{\scriptstyle\pm.02}$	$.51\,{\scriptstyle\pm.03}$	$.75\,{\scriptstyle\pm.01}$	$.041\,{\scriptstyle\pm.014}$

Table 6: Predictive entropy for detection and selective prediction. AUROC (higher is better) for detecting OOD recordings (per-recording mean entropy, WT vs. the five held-out OOD cohorts), artifact epochs, and inter-rater disagreement (epoch-level entropy, spindle cohorts); AURC (lower is better) for rejecting the most uncertain epochs on the OOD cohorts. Mean

\pm

95% interval over training seeds; for DE, MultiSWA, and MultiSWAG, the 2.5–97.5% interval over 100 resampled ensembles. Bold : best overall; underline : best within the model group.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Species/genotype	Source	Channels	$N_{\mathrm{rec}}$	Role	Shift w.r.t. training
Training and validation
WT	Mouse/wild-type	Lundbeck	ECoG, LFP, EMG	169	Train	—
WT	Mouse/wild-type	Lundbeck	ECoG, LFP, EMG	36	Validation	None
Test
WT	Mouse/wild-type	Lundbeck	ECoG, LFP, EMG	36	Test (ID)	None
NT1	Mouse/NT1	Lundbeck	ECoG, LFP, EMG	28	Test (OOD)	Genotype

Appendix

Table 7: Datasets used for training, model selection, and evaluation, grouped by training/validation and test sets.

N_{\mathrm{rec}}

denotes the number of 24 h recordings. ID denotes in-distribution, whereas OOD denotes out-of-distribution.

Model	Architecture	Input	Context
Somnotate	LDA+HMM	Spectrogram	Full recording
Spindle	CNN+HMM	Spectrogram	Full recording
U-Sleep	Fully convolutional	Raw signals	35 windows
CNN-GRU	CNN+GRU	Raw signals	35 windows

Appendix

Table 8: Overview of the baseline models and neural backbones.

Figure 2: Validation MF1 as a function of model capacity during architecture selection. (a) U-Sleep complexity factor

\alpha

, evaluated across 10 random seeds. (b) CNN encoder complexity factor

\alpha

, evaluated across five random seeds. (c) GRU hidden dimension with the CNN encoder fixed at

\alpha=1.0

, evaluated across 10 random seeds. Points show means and error bars show one standard deviation across seeds. Validation performance varies only weakly once sufficient model capacity is reached. The selected configurations are

\alpha=1.0

for U-Sleep,

\alpha=1.0

for the CNN encoder, and

H=128

for the GRU.

Figure 3: MAP optimization trajectories across prior strengths for CNN-GRU and U-Sleep. Solid lines show the training MAP objective and dashed lines show validation cross-entropy, averaged across three random seeds. All configurations were optimized for 2000 blocks (approximately 57 passes through the training set). The trajectories were used to verify convergence. The backbone-specific prior strengths were selected using mean validation MF1.

Model	Architecture	Parameters	$\lambda$	$\sigma_{p}^{2}$
U-Sleep	9 blocks, $\alpha=1.0$	$\sim 0.87$ M	$3\times 10^{-3}$	$9.42\times 10^{-5}$
CNN-GRU	Encoder $\alpha=1.0$ , GRU $H=128$	$\sim 1.27$ M	$10^{-3}$	$2.83\times 10^{-4}$

Appendix

Table 9: Final MAP model hyperparameters.

Method	CNN-GRU	U-Sleep	Training/collection budget	Shared settings
MAP	$\sigma_{p}^{2}=2.83\times 10^{-4}$	$\sigma_{p}^{2}=9.42\times 10^{-5}$	2000 blocks	Final checkpoint; deterministic prediction
SWA / SWAG	$\eta_{\mathrm{collect}}=10^{-3}$	$\eta_{\mathrm{collect}}=3\times 10^{-4}$	5 transition $+$ 200 collection blocks	Same trajectory for SWA and SWAG; one iterate per collection block; $R=100$ ; SWAG $S=30$ ; BN statistics recomputed using $10\%$ of the training data
LLLA	Full covariance	Full covariance	–	MAP prior variance; GLM predictive with $S=30$ Monte Carlo samples
VGD	$\sigma^{2}=10^{-4}$	$\sigma^{2}=10^{-5}$	3000 blocks	Random initialization; MAP prior retained; $S_{\mathrm{train}}=1$ , $S_{\mathrm{val}}=10$ , final $S=30$ ; learning rate $=10^{-4}$
MFVI	$\sigma_{q,0}^{2}=10^{-4}$	$\sigma_{q,0}^{2}=10^{-4}$	3000 blocks	MAP initialization of posterior means; MAP prior retained; $S_{\mathrm{train}}=1$ , $S_{\mathrm{val}}=10$ , final $S=30$ ; learning rate $=10^{-4}$
SGLD	$\eta_{\mathrm{batch}}=10^{-1}$	$\eta_{\mathrm{batch}}=10^{-1}$	3000 blocks	MAP initialization and prior; burn-in 1000 blocks; states stored every 10 blocks; $S=30$ selected uniformly from 200 post-burn-in states

Appendix

Table 10: Selected hyperparameters, optimization budgets, and predictive-integration settings for the controlled inference comparison. All tunable settings were selected using mean validation MF1.

Figure 4: Sensitivity of validation MF1 to the primary method-specific hyperparameters. For sampling-based posterior approximations, predictive probabilities were estimated using

S=30

samples. (a) MAP prior strength

\lambda

(equivalently, prior variance

\sigma_{p}^{2}

). (b) Shared SWA/SWAG collection learning rate. (c) VGD posterior variance. (d) MFVI initial posterior variance

\sigma_{q,0}^{2}

. (e) SGLD batch-scaled step size

\eta_{\mathrm{batch}}

. (f) DE size. Error bars show one standard deviation across seeds in (a–e) and the central

95\%

empirical interval across 100 random ensemble subsets in (f) .

Figure 5: Mean-scale MAP-objective trajectories for SGLD chains initialized from the final MAP checkpoints. Colors indicate candidate values of $\eta_{\mathrm{batch}}$ , dashed horizontal lines show the objective values at initialization, and the shaded region denotes the 1000-block burn-in period. The selected value $\eta_{\mathrm{batch}}=10^{-1}$ was chosen using validation MF1; the trajectories were used only to identify and remove the clearest initialization transient, not to claim formal convergence.

U-Sleep
Model	Acc. $\uparrow$	MF1 $\uparrow$	$\kappa$ $\uparrow$	LPD $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Entropy
MAP	$93.8\,{\scriptstyle\pm 1.7}$	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.186\,{\scriptstyle\pm.048}$	$9.4\,{\scriptstyle\pm 2.6}$	$1.7\,{\scriptstyle\pm 0.7}$	$.168\,{\scriptstyle\pm.041}$
Temp.	$93.8\,{\scriptstyle\pm 1.7}$	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.185\,{\scriptstyle\pm.046}$	$9.4\,{\scriptstyle\pm 2.5}$	$1.6\,{\scriptstyle\pm 0.4}$	$.189\,{\scriptstyle\pm.025}$
SWA	$95.3\,{\scriptstyle\pm 0.4}$	$91.5\,{\scriptstyle\pm 0.3}$	$91.4\,{\scriptstyle\pm 0.7}$	${\scriptstyle-}.141\,{\scriptstyle\pm.009}$	$7.1\,{\scriptstyle\pm 0.5}$	$\textbf{\lx@text@underline{1.4}}\,{\scriptstyle\pm 0.1}$	$.149\,{\scriptstyle\pm.010}$
SWAG	$95.4\,{\scriptstyle\pm 0.3}$	$91.7\,{\scriptstyle\pm 0.2}$	$91.6\,{\scriptstyle\pm 0.5}$	${\scriptstyle-}.138\,{\scriptstyle\pm.005}$	$6.9\,{\scriptstyle\pm 0.3}$	$1.6\,{\scriptstyle\pm 0.1}$	$.162\,{\scriptstyle\pm.009}$
Laplace	$93.8\,{\scriptstyle\pm 1.7}$	$89.7\,{\scriptstyle\pm 1.5}$	$88.6\,{\scriptstyle\pm 3.1}$	${\scriptstyle-}.186\,{\scriptstyle\pm.048}$	$9.4\,{\scriptstyle\pm 2.6}$	$1.7\,{\scriptstyle\pm 0.7}$	$.168\,{\scriptstyle\pm.041}$

Appendix

Table 11: In-distribution performance on the WT test set. Values are mean

\pm

95% CIs across seeds; for DE, MultiSWA, and MultiSWAG,

\pm

denotes half the empirical 2.5–97.5% interval across 100 resampled 10-member ensembles. Accuracy, MF1,

\kappa

, Brier score, and ECE are reported in percent; LPD and entropy are in nats. Bold and underlining denote the best overall and within each model group, respectively; entropy is descriptive and not ranked.

U-Sleep
Model	Acc. $\uparrow$	MF1 $\uparrow$	$\kappa$ $\uparrow$	LPD $\uparrow$	Brier $\downarrow$	ECE $\downarrow$	Entropy
MAP	$84.6\,{\scriptstyle\pm 3.5}$	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.481\,{\scriptstyle\pm.149}$	$24.2\,{\scriptstyle\pm 6.7}$	$7.7\,{\scriptstyle\pm 4.5}$	$.224\,{\scriptstyle\pm.040}$
Temp.	$84.6\,{\scriptstyle\pm 3.5}$	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.460\,{\scriptstyle\pm.122}$	$23.9\,{\scriptstyle\pm 6.3}$	$7.2\,{\scriptstyle\pm 3.9}$	$.246\,{\scriptstyle\pm.015}$
SWA	$85.8\,{\scriptstyle\pm 3.1}$	$78.0\,{\scriptstyle\pm 3.2}$	$74.3\,{\scriptstyle\pm 5.9}$	${\scriptstyle-}.435\,{\scriptstyle\pm.105}$	$21.7\,{\scriptstyle\pm 4.8}$	$6.3\,{\scriptstyle\pm 2.6}$	$.243\,{\scriptstyle\pm.031}$
SWAG	$86.7\,{\scriptstyle\pm 2.7}$	$79.2\,{\scriptstyle\pm 2.8}$	$75.7\,{\scriptstyle\pm 5.3}$	${\scriptstyle-}.376\,{\scriptstyle\pm.060}$	$19.8\,{\scriptstyle\pm 3.6}$	$5.5\,{\scriptstyle\pm 1.9}$	$.282\,{\scriptstyle\pm.027}$
Laplace	$84.6\,{\scriptstyle\pm 3.5}$	$76.0\,{\scriptstyle\pm 4.4}$	$72.1\,{\scriptstyle\pm 6.3}$	${\scriptstyle-}.481\,{\scriptstyle\pm.149}$	$24.2\,{\scriptstyle\pm 6.7}$	$7.7\,{\scriptstyle\pm 4.5}$	$.224\,{\scriptstyle\pm.040}$

Appendix

Table 12: Out-of-distribution performance. Metrics are averaged over the five OOD cohorts within each training seed. Values are mean

\pm

95% CIs across seeds; for DE, MultiSWA, and MultiSWAG,

\pm

denotes half the empirical 2.5–97.5% interval across 100 resampled 10-member ensembles. Accuracy, MF1,

\kappa

	Single solution/trajectory								Multi-basin			Other baselines
Scope	MAP	Temp.	SWA	SWAG	Laplace	VGD	MFVI	SGLD	DE	MultiSWA	MultiSWAG	Somnotate	Spindle
ID	19.5	19.5	8.0	7.0	21.0	13.0	15.0	10.5	9.5	3.5	1.5	20.0	24.0
OOD	16.6	16.6	12.9	10.1	17.2	13.6	13.6	10.9	7.4	8.4	7.0	16.8	14.6
ID + OOD	17.1	17.1	12.1	9.6	17.8	13.5	13.8	10.8	7.8	7.6	6.1	17.3	16.2

Appendix

Table 13: Average MF1 rank. Models were ranked jointly within each cohort, and ranks were averaged over cohorts and backbones (

1=

best; lower is better). Other baselines are ranked individually. Bold and underlining denote the best overall and within each model group, respectively.

U-Sleep
	AUROC $\uparrow$									AURC $\downarrow$
	OOD			Artifact			Disagree			Reject
Model	TU	EU	AU	TU	EU	AU	TU	EU	AU	TU	EU	AU
MAP	$.72\,{\scriptstyle\pm.06}$	$.50\,{\scriptstyle\pm.00}$	$.72\,{\scriptstyle\pm.06}$	$.44\,{\scriptstyle\pm.03}$	$.50\,{\scriptstyle\pm.00}$	$.44\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.02}$	$.50\,{\scriptstyle\pm.00}$	$.76\,{\scriptstyle\pm.02}$	$.054\,{\scriptstyle\pm.015}$	$.152\,{\scriptstyle\pm.035}$	$.054\,{\scriptstyle\pm.015}$
Temp.	$.71\,{\scriptstyle\pm.06}$	$.50\,{\scriptstyle\pm.00}$	$.71\,{\scriptstyle\pm.06}$	$.44\,{\scriptstyle\pm.03}$	$.50\,{\scriptstyle\pm.00}$	$.44\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.02}$	$.50\,{\scriptstyle\pm.00}$	$.76\,{\scriptstyle\pm.02}$	$.054\,{\scriptstyle\pm.015}$	$.152\,{\scriptstyle\pm.035}$	$.054\,{\scriptstyle\pm.015}$
SWA	$.80\,{\scriptstyle\pm.03}$	$.50\,{\scriptstyle\pm.00}$	$.80\,{\scriptstyle\pm.03}$	$.49\,{\scriptstyle\pm.03}$	$.50\,{\scriptstyle\pm.00}$	$.49\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.01}$	$.50\,{\scriptstyle\pm.00}$	$.76\,{\scriptstyle\pm.01}$	$.049\,{\scriptstyle\pm.019}$	$.139\,{\scriptstyle\pm.031}$	$.049\,{\scriptstyle\pm.019}$

Appendix

Table 14: Detection and rejection by total, epistemic, and aleatoric uncertainty. Total uncertainty (TU) is the entropy of the predictive distribution, aleatoric uncertainty (AU) the mean entropy of the posterior samples or ensemble members it averages, and epistemic uncertainty (EU = TU

-

AU) their mutual information; the TU columns are those of Table 6 . AUROC measures detection of OOD recordings, artifact-containing epochs, and inter-rater disagreement; higher is better. OOD detection uses the per-recording mean of the epoch-level score to distinguish WT from the five held-out OOD cohorts, whereas artifact and disagreement detection use the epoch-level score on the spindle cohorts. AURC measures selective prediction across the OOD cohorts with the score as the rejection criterion; lower is better. Methods with a single predictive component (MAP, Temp., SWA, Somnotate, Spindle) have EU = 0 and AU = TU: their EU columns are the chance-level AUROC (.50) and the AURC of rejecting in random order. Results are mean

\pm

95% intervals across training seeds; for DE, MultiSWA, and MultiSWAG, intervals are the 2.5–97.5% empirical interval across 100 resampled 10-member ensembles. Bold and underlining denote the best overall and within each model group, respectively.

U-Sleep
	Entropy ( $H$ )						Detection AUROC $\uparrow$			AURC $\downarrow$
Model	ID	OOD	Agree	Disagree	No artifact	Artifact	OOD	Disagree	Artifact	OOD
MAP	$.17\,{\scriptstyle\pm.04}$	$.22\,{\scriptstyle\pm.04}$	$.22\,{\scriptstyle\pm.04}$	$.48\,{\scriptstyle\pm.08}$	$.24\,{\scriptstyle\pm.04}$	$.20\,{\scriptstyle\pm.03}$	$.72\,{\scriptstyle\pm.06}$	$.76\,{\scriptstyle\pm.02}$	$.44\,{\scriptstyle\pm.03}$	$.054\,{\scriptstyle\pm.015}$
Temp.	$.19\,{\scriptstyle\pm.02}$	$.25\,{\scriptstyle\pm.02}$	$.24\,{\scriptstyle\pm.02}$	$.51\,{\scriptstyle\pm.05}$	$.26\,{\scriptstyle\pm.02}$	$.22\,{\scriptstyle\pm.01}$	$.71\,{\scriptstyle\pm.06}$	$.76\,{\scriptstyle\pm.02}$	$.44\,{\scriptstyle\pm.03}$	$.054\,{\scriptstyle\pm.015}$
SWA	$.15\,{\scriptstyle\pm.01}$	$.24\,{\scriptstyle\pm.03}$	$.24\,{\scriptstyle\pm.03}$	$.51\,{\scriptstyle\pm.05}$	$.26\,{\scriptstyle\pm.03}$	$.26\,{\scriptstyle\pm.02}$	$.80\,{\scriptstyle\pm.03}$	$.76\,{\scriptstyle\pm.01}$	$.49\,{\scriptstyle\pm.03}$	$.049\,{\scriptstyle\pm.019}$
SWAG	$.16\,{\scriptstyle\pm.01}$	$.28\,{\scriptstyle\pm.03}$	$.28\,{\scriptstyle\pm.03}$	$.56\,{\scriptstyle\pm.05}$	$.30\,{\scriptstyle\pm.03}$	$.31\,{\scriptstyle\pm.01}$	$.82\,{\scriptstyle\pm.02}$	$.75\,{\scriptstyle\pm.01}$	$.51\,{\scriptstyle\pm.03}$	$.041\,{\scriptstyle\pm.014}$

Appendix

Table 15: Predictive entropy and uncertainty utility. Mean predictive entropy

H

(nats) is shown for WT (ID), the mean over the five OOD cohorts, and spindle windows grouped by scorer agreement and artifact flags. Agree/Disagree excludes artifact-flagged windows; Artifact denotes a flag by either scorer. AUROC evaluates OOD-recording detection using per-recording mean entropy and disagreement/artifact detection using window-level entropy. AURC evaluates entropy-based selective prediction on the OOD cohorts; lower is better. Values are mean

\pm

95% CIs across seeds; for DE, MultiSWA, and MultiSWAG,

\pm

denotes half the empirical 2.5–97.5% interval across 100 resampled 10-member ensembles. Bold and underlining denote the best overall and within each model group, respectively.

Figure 6: Reliability curves per OOD cohort. Columns are the five held-out OOD cohorts (NT1, spindle -A–D) and rows the two backbones. Each point is a confidence bin, plotted at its mean confidence against the accuracy of its epochs; the dotted diagonal is perfect calibration, and points below it indicate overconfidence. Curves are averaged across training seeds; for DE, MultiSWA, and MultiSWAG, across 100 resampled 10-member ensembles. MAP is shown as the dashed black reference.

Figure 7: Selective prediction on the OOD cohorts. Within each held-out OOD cohort, epochs are rejected in order of decreasing predictive entropy; curves show the accuracy of the retained epochs against the retained fraction (coverage), averaged over the five cohorts. The legend reports the AURC, the area under the corresponding risk–coverage curve with risk

=1-

accuracy, as in Table 6 ; lower is better. Curves and AURC are averaged across training seeds; for DE, MultiSWA, and MultiSWAG, across 100 resampled 10-member ensembles. MAP is shown as the dashed black reference. Somnotate and Spindle do not depend on the backbone and appear in both panels (dash-dotted).

Figure 8: Detection by predictive entropy. ROC curves for OOD, disagreement, and artifact detection (rows) with predictive entropy as the score, for the two backbones (columns). OOD detection uses the per-recording mean entropy to distinguish WT test recordings from those of the five held-out OOD cohorts; disagreement and artifact detection use the epoch-level entropy on the spindle cohorts to identify epochs with inter-rater disagreement and artifact-containing epochs, respectively. Curves pool all training seeds (for DE, MultiSWA, and MultiSWAG, the 100 resampled 10-member ensembles); the legend reports the AUROC as in Table 6 , mean

\pm

95% CI across seeds, where for the ensembles

\pm

denotes half the empirical 2.5–97.5% interval. MAP is shown as the dashed black reference, and the dotted diagonal is chance. Somnotate and Spindle do not depend on the backbone and appear in both columns (dash-dotted); Somnotate is a single model and has no interval.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 285 papers

Sep 25, 2026

Model-Agnostic Online Certificate-Driven Calibration for Time Series Forecasting Under Distribution Shift

Time series out-of-distribution generalization requires forecasters to remain reliable when deployment dynamics differ from training conditions due to covariate shift, concept shift, and temporal dependence. Probably Approximately Correct Bayesian domain adaptation provides computable certificates by decomposing target risk into a source risk term, a source-to-target mismatch term, and a complexity term, but standard analyses rely on independent sampling and distributional stability, assumptions that are violated in time series by serial dependence and nonstationary shift. We propose a model-agnostic online martingale Probably Approximately Correct Bayesian framework that yields finite-sample certificates under temporal dependence and distribution shift. The certificate replaces independent-sample concentration with martingale concentration that adapts to loss scale and predictable variation. We use the certificate as a surrogate regularizer for online calibration by training a gated residual Bayesian head on top of a fixed forecasting backbone, producing a corrective update that reverts to the backbone prediction when the gate is closed. Online calibration combines a source risk anchor, a posterior-shift penalty, and a time-adaptive mismatch term computed from target windows observed before forecasting. It follows a predict-then-update protocol in which outcomes become available only after forecasting and are used to update subsequent predictions. Experiments across convolutional, attention-based, and large language model-based forecasters show improved stability and accuracy under covariate and concept shift.

Chenfeng Huang, Zixuan Ma, George Michailidis

Department of Statistics and Data Science, University of California, Los Angeles, Los Angeles, California, USA

Online Calibration Time Series Forecasting OOD Generalization Model Calibration Temporal Distribution Shift PAC-Bayesian Generalization Bounds

Ask PDF

Apr 25, 2026

Unstable Rankings in Bayesian Deep Learning Evaluation

Standard evaluations of Bayesian deep learning methods assume that metric estimates are reliable, but we show this assumption fails under data scarcity. Method rankings are not only unreliable at small

n

, but also dataset-dependent in ways that point estimates cannot reveal: the same method comparison yields

P(\mathrm{MCD} \prec \mathrm{Ensemble}) = 1.000

n = 50

on one dataset and remains below

0.95

even at

n = 500

on another. Across the datasets we consider, no universal sample size threshold exists, which is precisely why dataset-specific posterior inference is necessary. To address this, we use a Bayesian hierarchical model with method-specific variances to treat evaluation metrics as random variables across data realizations, and we use a predictive Minimum Detectable Difference curve to assess whether an observed gap would be detectable at a given training size. Across six Bayesian deep learning methods and five regression datasets, our results show that uncertainty-aware evaluation is necessary in low-data settings, because current evidence for method superiority and predictive detectability at the same training size can diverge substantially. Our framework provides practitioners with principled tools to determine whether their evaluation data is sufficient before drawing conclusions about method superiority.

Qishi Zhan, Minxuan Hu, Guansu Wang +2

Department of Mathematical and Statistical Sciences Marquette University, USA · Cornell University, USA · The University of Melbourne, Australia +2

Bayesian Neural Networks Statistical Power Analysis Bayesian Inference

Ask PDF

Sep 11, 2026

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores

Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the Label-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.

Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2

MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea

Label Shift Conformal Prediction Regression Prediction Intervals

Ask PDF

Abstract

Distribution Shift Robustness

Explore similar work

Model-Agnostic Online Certificate-Driven Calibration for Time Series Forecasting Under Distribution Shift

Unstable Rankings in Bayesian Deep Learning Evaluation

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores