Modular Bayesian Foundation Models: Bayesian time series encoders for OOD detection, routing and forecasting

Mame Diarra Toure; Quang Hung Pham; Mohamed Ryad Zemouri; David Stephens; Luc Vouligny; Martin Gagnon

stat.MLOct 28, 2026

Modular Bayesian Foundation Models: Bayesian time series encoders for OOD detection, routing and forecasting

Authors: Mame Diarra Toure, Quang Hung Pham, Mohamed Ryad Zemouri, David Stephens, Luc Vouligny, Martin Gagnon

Organizations: Department of Mathematics and Statistics McGill University · Hydro-Quebec

Abstract

Industrial systems continuously generate time series that vary in sampling frequency, scale, dynamics, and operating regime. Consequently a single time-series foundation model may not be able to represent all regimes effectively and retraining a large model for every new dataset or regime shift is computationally expensive. Modular foundation models address this issue by reusing specialised encoders trained on diverse datasets and by allowing the expansion of capabilities through the addition of new experts. This approach requires compact experts to minimize storage cost and a proper selection mechanism of experts for unseen data. We propose a modular foundation model framework that combines end-to-end Bayesian encoder training with uncertainty based routing. Using Singular Bayesian Neural Networks (SBNNs), we build a collection of compact Bayesian encoders whose epistemic uncertainty guides label-free selection. At the selected ranks, our Bayesian collection uses 63.8% less storage than its deterministic counterpart. We evaluate this framework on time-series data through distribution shift, encoder selection, and probabilistic forecasting. Across seven incoming datasets, uncertainty-based routing is competitive with distance-based routing in terms of forecast MSE, while reducing average ECE from 0.086 to 0.072 and average NLL from 0.907 to 0.836. It also increases PICP90 from 82.28% to 89.03%. In a separate model-family comparison, we evaluate the SBNN pipeline against full-rank Bayes by Backprop, deep ensembles, and MC Dropout. The SBNN pipeline achieves an average MSE of 0.390, comparable with 0.396 for deep ensembles, 0.389 for MC dropout, and 0.443 for the full-rank Bayesian model. It uses 63.9% fewer parameters than MC dropout, 81.9% fewer than the full-rank model, and 92.8% fewer than the deep ensemble.

Figures & tables

Figure 1: The modular Bayesian pipeline. Padlocks mark frozen posteriors whose weights are sampled. The dashed boundary marks

N_{\mathrm{MC}}

joint encoder–head draws; selection, projections and aggregation weights remain fixed.

Figure 2: Training-free uncertainty routing. Posterior draws estimate uncertainty for each of the

M

encoders. The

K

lowest source-normalized ratios receive inverse-ratio weights; others receive zero weight. Subscript

(j)

denotes rank

j

; grids show array dimensions. Routing uses no target labels, decoder or head.

Table 3

Policy	Selection rule	Weights
Top- $K$ uncertainty	$K$ lowest source-normalized uncertainty ratios	Inverse-ratio
Top- $K$ distance	$K$ lowest source-distance scores	Inverse-distance
Source-diverse top- $K$	Lowest-ratio encoder per source, then $K$ lowest ratios	Inverse-ratio
Random- $K$ , full pool	Sample $K$ encoders without replacement	Inverse-ratio
Random- $K$ , shortlist	Sample $K$ encoders without replacement	Uniform
Bottom- $K$ , shortlist	$K$ highest uncertainty ratios within the shortlist	Uniform

Table 3: Selection policies and weights. Each policy selects

K

encoders. Top and bottom denote the lowest and highest scores. The shortlist contains ratios

r_{m}\leq\tau_{\mathrm{route}}

. Weights are normalized over selected encoders.

Family	MSE $\downarrow$	ECE $\downarrow$	NLL $\downarrow$	PICP90 (%) $\rightarrow 90\%$	MPIW90	Params (M) $\downarrow$
SBNN	$0.3896\pm 0.0031$	$\mathbf{0.0687}\pm 0.0025$	$\mathbf{0.814}\pm 0.021$	$\mathbf{90.47}\pm 0.09$	$1.771\pm 0.018$	1.759
Deterministic	$0.4056\pm 0.0008$	$0.0816\pm 0.0026$	$1.353\pm 0.422$	$82.60\pm 0.36$	$1.476\pm 0.007$	4.868
MC Dropout	$\mathbf{0.3894}\pm 0.0041$	$0.0767\pm 0.0008$	$0.817\pm 0.004$	$87.78\pm 0.07$	$1.647\pm 0.007$	4.868
Full-rank BBB	$0.4426\pm 0.0016$	$0.0877\pm 0.0018$	$0.899\pm 0.002$	$92.29\pm 0.06$	$2.100\pm 0.002$	9.731
Deep ensemble (5)	$0.3965\pm 0.0028$	$0.0734\pm 0.0039$	$0.881\pm 0.008$	$84.55\pm 0.44$	$1.517\pm 0.007$	24.342

Table 4: Forecasting results and stored parameters. Mean

\pm

sample SD over three seeds, each averaged over seven datasets. Counts include the full encoder collection and forecasting head. Bold marks the best mean or smallest count.

Figure 5: Forecasting quality and compactness. Top-4 uncertainty routing; seven datasets, three seeds. Left: raw metrics and encoder–head counts; error bars show sample SD of dataset-averaged seed results. Dashed line: 90% coverage. Right: higher is better. MSE, ECE and parameter scores use the lowest family value divided by each family’s value; coverage uses

1-|\mathrm{PICP90}-0.90|/0.90

Figure 6: Calibration under distance and uncertainty routing. Bars average three seeds per dataset; the dashed line marks nominal 90% coverage.

Table 9

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Measured signal	Origin
Source datasets
Acoustic	Generator acoustic signal	Industrial partner
HAR	Human-activity signals	Public ( Reyes-Ortiz et al., 2013 )
Torque45	Shaft torque signal at 45% of maximum unit power	Industrial partner
ETT (ETTh1)	Transformer oil temperature	Public ( Zhou et al., 2021 )
Incoming datasets

Appendix

Table 7: Source and incoming datasets. Source datasets support encoder pretraining. Incoming datasets support routing, reconstruction evaluation and downstream forecasting.

SBNN configuration	$\alpha$	KL coefficient SBNN	KL coefficient Full-rank BBB
Acoustic balanced	0.1	$10^{-4}$	$5\times 10^{-6}$
Acoustic lean	0.1	$7.5\times 10^{-5}$	$5\times 10^{-6}$
Acoustic counterpart	0.1	$1.5\times 10^{-4}$	$5\times 10^{-6}$
Acoustic strong variants (3 encoders)	0.1	$5\times 10^{-6}$	$5\times 10^{-6}$
Acoustic CNN	0	$3\times 10^{-4}$	$5\times 10^{-6}$
HAR variants (3 encoders)	0.5	$0.03$	$0.03$

Appendix

Table 8: Source-pretraining coefficients.

\alpha

controls the reconstruction–contrastive mixture. KL columns give the final regularization coefficients. Full-rank BBB coefficients are specified per source family.

Encoder configuration	Configured masking
Acoustic Transformer	Point fraction 0.05; span fraction 0.25
HAR	Mask ratio 0.50
ETT	Mask ratio 0.25
Torque45	Mask ratio 0
Acoustic CNN	Mask ratio 0

Appendix

Table 9: Source-pretraining masking settings. Point and span masking are separate operations. A zero ratio means that no reconstruction mask is applied.

Figure 7: Rank–KL tradeoffs in ETT source reconstruction. Each circle represents one tested configuration. Colours identify the encoder and decoder ranks; point labels give the KL scale. Encoder ranks follow the order convolution 1, convolution 2, intermediate projection, and latent layer. Decoder ranks follow the order fully connected layer 1, fully connected layer 2, and output layer. MSE is evaluated over the whole window; ECE and PICP90 use the masked-point calibration evaluation. Dark outlines mark non-dominated configurations for MSE and ECE (left), or MSE and absolute coverage error from 90% (right).

Figure 8: Stored encoder parameters by source and family. Percentages give SBNN reductions relative to deterministic encoders. Panels use separate scales. Decoders and forecasting heads are excluded.

Family	Encoders	Head	Total
SBNN	1,743,722	14,784	1,758,506
Deterministic	4,818,737	49,600	4,868,337
MC Dropout	4,818,737	49,600	4,868,337
Full-rank BBB	9,632,608	98,752	9,731,360
Deep ensemble (5)	24,093,685	248,000	24,341,685

Appendix

Table 10: Encoder and forecasting-head parameter counts. Encoder totals cover the full 17-encoder collection in each family. The total includes the collection and forecasting head. Bayesian counts include posterior means and scales; ensemble counts include all five members. Bold indicates the smallest count in each column.

Method	Stored components
SBNN ( Toure and Stephens, 2026 )	Means and scales of the low-rank weight factors; no retained dense weight matrix for the factorized layer.
Rank-1 BNNs ( Dusenberry et al., 2020 )	Shared dense weight matrix and posterior parameters for multiplicative vectors.
SWAG ( Maddox et al., 2019 )	Dense weight mean, diagonal variance statistics and retained low-rank covariance deviation vectors.
Low-rank Gaussian VI ( Tomczak et al., 2020 )	Dense weight mean and low-rank covariance factors, with a fixed diagonal constant or a learned diagonal variance vector.
BLoB ( Wang et al., 2024 )	Frozen dense backbone, one deterministic adapter factor, and the mean and scale of the other adapter factor.

Appendix

Table 11: Stored components of Bayesian parameterizations. The comparison describes the published constructions, including frozen backbone weights.

Acoustic
Model	MSE $\downarrow$	ECE $\downarrow$	PICP90 (%) $\rightarrow 90\%$	MPIW90
Deterministic	0.07613	—	—	—
MC Dropout	0.07613	0.3620	29.48	0.2880
Deep ensemble (5)	0.05515	0.3391	30.86	0.2865
Full-rank BBB	0.07492	0.1251	71.09	0.8453
SBNN	0.10601	0.0453	80.72	1.3125

Appendix

Table 12: Source reconstruction and calibration. MSE uses validation windows; uncertainty metrics follow each source’s protocol described below. Bold marks the lowest MSE or ECE, or coverage closest to 90%. Dashes denote unreported values.

Figure 9: Source reconstructions and uncertainty bands. Selected validation windows from the four source domains. The first row shows the original signal. Subsequent rows show reconstructions (solid), the original signal (dashed) and nominal 90% uncertainty bands. Models share the same window and axes within each column. The deterministic model has no uncertainty band.

Acoustic
Model	MSE $\downarrow$	ECE $\downarrow$	PICP90 (%) $\rightarrow 90\%$	MPIW90
Deterministic	0.07613	—	—	—
MC Dropout	0.07613	0.3620	29.48	0.2880
Deep ensemble (5)	0.05515	0.3391	30.86	0.2865
Full-rank BBB	0.07492	0.1251	71.09	0.8453
SBNN (reconstruction)	0.09097	0.0898	74.79	1.0428

Appendix

Table 13: Source reconstruction results across model families and SBNN configurations. Bold indicates the lowest MSE or ECE, or coverage closest to 90%, within each source. Dashes indicate unreported uncertainty metrics.

Figure 10: Acoustic source reconstruction. Each column shows the same window across all model families. The first row displays the original signal. Subsequent rows show reconstructions with nominal 90% uncertainty bands; dashed curves indicate the original signal. The deterministic model has no uncertainty band.

Figure 11: HAR source reconstruction. Each column shows the same window across all model families. The first row displays the original signal. Subsequent rows show reconstructions with nominal 90% uncertainty bands; dashed curves indicate the original signal. The deterministic model has no uncertainty band.

	Torque45 encoder		Acoustic encoder		ETT encoder
Dataset	$D$	$r$	$D$	$r$	$D$	$r$
Torque45	0.000	1.000	1.055	0.751	1.739	1.538
Acoustic	1.055	3.177	0.000	1.000	1.939	2.000
HAR	1.059	0.962	1.874	0.780	1.863	1.437
ETT	1.739	7.759	1.939	1.982	0.000	1.000
WTH	1.130	3.913	1.063	1.289	1.547	1.627

Appendix

Table 14: Statistical-feature distance and encoder uncertainty across eight domains.

D

is the distance from the encoder’s source dataset;

r

is its source-normalized uncertainty ratio.

Family	MSE $\downarrow$	ECE $\downarrow$	PICP90 (%) $\rightarrow 90\%$	MPIW90
SBNN	0.456	0.088	80.60	1.874
Deterministic	0.407	—	—	—
MC Dropout	0.392	0.341	32.77	0.363
Deep ensemble (5)	0.394	0.385	24.61	0.283

Appendix

Table 15: Initial forecasting with epistemic-only intervals. Results are averaged across seven incoming datasets. Heads were trained with MSE. A dash indicates an unavailable interval metric.

Figure 12: Forecasts and uncertainty bands across all seven incoming datasets. Columns show incoming datasets and rows show model families. Each column uses the same saved forecast window across all five families. Dashed curves show the true signal, solid curves the forecast, and shaded regions nominal 90% prediction bands. Panels retain their original vertical scales.

Figure 13: Forecasting coverage at two nominal levels. Points show observed coverage at nominal 70% and 90%, averaged over seven incoming datasets and three seeds. The dashed diagonal marks agreement between nominal and observed coverage.

Metric	Top-3 distance	Top-3 uncertainty
MSE $\downarrow$	$0.4062\pm 0.0059$	$\mathbf{0.3976}\pm 0.0070$
ECE $\downarrow$	$0.0857\pm 0.0013$	$\mathbf{0.0721}\pm 0.0038$
NLL $\downarrow$	$0.907\pm 0.039$	$\mathbf{0.836}\pm 0.041$
PICP90 (%) $\rightarrow 90\%$	$82.28\pm 0.58$	$\mathbf{89.03}\pm 0.35$
MPIW90	$1.795\pm 0.007$	$1.715\pm 0.032$

Appendix

Table 16: Forecasting with top-3 distance and uncertainty routing. Mean

\pm

sample SD over three seeds, each averaged over seven datasets. Bold marks the better mean.

Torque10
Policy	MSE $\downarrow$	ECE $\downarrow$	NLL $\downarrow$	PICP90 (%) $\rightarrow 90\%$	MPIW90
Top-3 distance	$0.5531\pm 0.0098$	$0.0624\pm 0.0086$	$1.108\pm 0.014$	$82.62\pm 1.81$	$1.997\pm 0.013$
Top-3 uncertainty	$0.5004\pm 0.0159$	$0.0419\pm 0.0071$	$1.067\pm 0.020$	$85.45\pm 0.56$	$2.038\pm 0.010$
Torque90
Top-3 distance	$0.3164\pm 0.0041$	$0.0436\pm 0.0036$	$0.889\pm 0.005$	$83.52\pm 0.22$	$1.999\pm 0.024$
Top-3 uncertainty	$0.3319\pm 0.0029$	$0.0195\pm 0.0023$	$0.808\pm 0.001$	$91.46\pm 0.21$	$1.914\pm 0.010$

Appendix

Table 17: Forecasting under distance and uncertainty routing. PICP90 is expressed as a percentage.

Figure 14: Forecasting under distance and uncertainty routing. Both policies select three SBNN encoders. Bars show averages across seven incoming datasets. Whiskers show sample standard deviations across three seeds, after averaging each seed across datasets. The dashed line marks nominal 90% coverage.

Figure 15: Reconstruction and forecasting under the selection policies. The upper row reports reconstruction on incoming datasets; the lower row reports downstream forecasting. Columns show MSE, ECE and PICP90. Lower MSE and ECE are preferred; dashed lines mark nominal 90% coverage. Policies retain the same colours and ordering.

	Reconstruction MSE $\downarrow$
Policy	Torque10	Torque90	WTH	ETTh2	ECL	Vibration	EMG
Top-3 uncertainty	0.0015	$\mathbf{2.43\times 10^{-6}}$	0.0268	0.0134	0.0104	0.0184	0.0090
Top-4 uncertainty	0.2069	0.1260	0.0832	0.1899	0.2602	0.0193	0.0098
Source-diverse top-3	0.5247	0.3461	0.2405	0.5437	0.7143	0.0478	0.1360
Source-diverse top-4	0.4926	0.3424	0.2060	0.4592	0.7136	0.1686	0.1226
Random-4 of top-8	0.5989	0.4088	0.2092	0.6303	0.8713	0.0219	0.0079

Appendix

Table 18: Reconstruction MSE on the incoming datasets. Bold marks the lowest error within each dataset.

	Reconstruction ECE $\downarrow$
Policy	Torque10	Torque90	WTH	ETTh2	ECL	Vibration	EMG
Top-3 uncertainty	0.3752	0.2316	0.1669	0.1936	0.3404	0.2912	0.3451
Top-4 uncertainty	0.4044	0.2933	0.2173	0.2596	0.3752	0.2931	0.3432
Source-diverse top-3	0.4660	0.4250	0.3649	0.4045	0.4531	0.3129	0.2775
Source-diverse top-4	0.4568	0.4237	0.3001	0.3459	0.4529	0.3129	0.2566
Random-4 of top-8	0.4688	0.4179	0.3065	0.4082	0.4506	0.2996	0.2759

Appendix

Table 19: Calibration of incoming-data reconstructions. Bold marks the lowest ECE within each dataset.

	Forecasting ECE $\downarrow$
Policy	Torque10	Torque90	WTH	ETTh2	ECL	Vibration	EMG
Top-3 uncertainty	0.0419	0.0195	0.0055	0.0874	0.0067	0.1378	0.2061
Top-4 uncertainty	0.0166	0.0327	0.0101	0.0643	0.0068	0.1345	0.2158
Source-diverse top-3	0.0095	0.0326	0.0161	0.0485	0.0135	0.1273	0.2296
Source-diverse top-4	0.0129	0.0401	0.0108	0.0514	0.0098	0.1240	0.2285
Random-4 of top-8	0.0118	0.0345	0.0052	0.0593	0.0186	0.1171	0.2263

Appendix

Table 20: Forecasting calibration by incoming dataset and selection policy. Lower ECE is better. Bold values indicate the lowest ECE within each dataset.

Policy	MSE $\downarrow$	ECE $\downarrow$	PICP90 (%) $\to 90\%$	NLL $\downarrow$	MPIW90
Top- $3$	$0.3976\pm 0.0070$	$0.0721\pm 0.0038$	$89.03\pm 0.35$	$0.8357\pm 0.0413$	$1.715\pm 0.032$
Top- $4$	$0.3896\pm 0.0031$	$0.0687\pm 0.0025$	$90.47\pm 0.09$	$0.8135\pm 0.0213$	$1.771\pm 0.018$
Cutoff $1.10$	$0.3972\pm 0.0070$	$0.0719\pm 0.0045$	$89.03\pm 0.23$	$0.8240\pm 0.0125$	$1.716\pm 0.020$

Appendix

Table 21: Fixed-

K

and fixed-cutoff uncertainty routing. The cutoff policy selects all encoders with

r_{m}\leq 1.10

. Entries are means and sample standard deviations across the three seed-specific averages over seven incoming datasets.

Incoming dataset	Selected encoders
Torque10	3
Torque90	3
WTH	6
ETTh2	3
ECL	3
Vibration	10

Appendix

Table 22: Expert selection with a fixed uncertainty cutoff. The eligible pool contains 17 encoders. The policy retains encoders with

r_{m}\leq 1.10

Time-Series Foundation Models

Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.

See all 303 papers

Jul 22, 2026

Expert-Guided Forecast Editing for Time-Series Foundation Models

Time-series foundation models can forecast across heterogeneous domains without task-specific training, but their forecasts are fixed once produced and cannot directly incorporate task-specific expert feedback. We study expert-guided forecast editing: a frozen foundation model generates candidate future trajectories, and an expensive expert evaluator scores them to guide forecast revision. Under a tight query budget, two natural strategies sit at opposite ends: best-of-

N

purely exploits the foundation model's predictive distribution, while optimization approaches mostly explore the forecast horizon as an unstructured high-dimensional vector. Each extreme is individually sub-optimal. We introduce DEFT, an expert-guided forecast editing framework that balances the two by first exploiting the foundation model's predictive samples in a decomposed trend--seasonal space, then exploring around them via component-wise refinement. DEFT queries the expert only on complete trajectories, then reuses scores for the trend and seasonal components that appeared in the queried recombinations. This lets each expert query provide structured component-level feedback while keeping the foundation model frozen. We compare DEFT against direct search approaches, including best-of-

N

, cross-entropy methods, and Bayesian optimization, under matched expert-query budgets. Across two forecasting benchmarks consisting of 78 datasets, three time-series foundation models, four feedback types, and seven query budgets, DEFT consistently improves the effectiveness of expert guidance. A molecular-dynamics case study further suggests that the same principle extends to more physically grounded feedback, supporting the hypothesis that sparse test-time guidance should be spent balancing prior exploitation with structured exploration.

Hung Le, Minh Hoang Nguyen, Manh Nguyen +2

Deakin Applied Artifical Intelligence Initiative Deakin University, Australia

Black-Box Optimization Time Series Forecasting Time-Series Foundation Models Inference-Time Guidance

Ask PDF

Mar 10, 2026

Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers

Foundation models are increasingly being deployed in contexts where understanding the uncertainty of their outputs is critical to ensuring responsible deployment. While Bayesian methods offer a principled approach to uncertainty quantification, their computational overhead renders their use impractical for training or inference at foundation model scale. State-of-the-art models achieve parameter counts in the trillions through carefully engineered sparsity including Mixture-of-Experts (MoE) layers. In this work, we demonstrate calibrated uncertainty at scale by introducing Variational Mixture-of-Experts Routing (VMoER), a structured Bayesian approach for modelling uncertainty in MoE layers. VMoER confines Bayesian inference to the expert-selection stage which is typically done by a deterministic routing network. We instantiate VMoER using two inference strategies: amortised variational inference over routing logits and inferring a temperature parameter for stochastic expert selection. Across fine-tuning tested foundation models, VMoER improves routing stability under noise by 38%, reduces calibration error by 94%, and increases out-of-distribution AUROC by 12%, while incurring less than 1% additional FLOPs. These results suggest VMoER offers a scalable path toward robust and uncertainty-aware foundation models.

Albus Yizhuo Li, Matthew Wicker

Department of Computing, Imperial College London.

Uncertainty Quantification Mixture-of-Experts Models Uncertainty Calibration Variational Inference

Ask PDF

Jun 10, 2026

TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models

Time-series foundation models (TSFMs) are increasingly explored as predictive experts within emerging agentic time-series systems. However, TSFMs exhibit heterogeneous inductive biases, and no single model consistently dominates across forecasting regimes, making expert selection a critical challenge. Existing systems often delegate this decision to LLM-based controllers, incurring substantial inference overhead. We present TimeRouter, an efficient routing framework that leverages empirical complementarity across a pool of pretrained TSFMs through lightweight discriminative routing, selective gating, and ensemble fallback. Concretely, TimeRouter combines a learned routing head, a selective gate, and an ensemble fallback, enabling adaptive expert selection without invoking an LLM at inference time. TimeRouter achieves state-of-the-art performance on the GIFT-EVAL leaderboard, with an LB MASE of 0.6765. Beyond benchmark performance, our ablation studies provide empirical insights into TSFM routing design, highlighting the importance of pool composition and selective gating. Taken together, these results position TimeRouter as a modular and lightweight routing layer for future agentic time-series systems built upon foundation-model pools. Our code is available at https://github.com/UConn-DSIS/TimeRouter.

Kanghui Ning, Yushan Jiang, Kashif Rasul +3

School of Computing, University of Connecticut, Storrs, USA · Department of Machine Learning Research, Morgan Stanley, New York, USA.

Adaptive Model Routing Time Series Forecasting Time-Series Foundation Models

Ask PDF

Abstract

Time-Series Foundation Models

Explore similar work

Expert-Guided Forecast Editing for Time-Series Foundation Models

Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers

TimeRouter: Efficient and Adaptive Routing of Time-Series Foundation Models