Evaluating Uncertainty Calibration in Probabilistic Time Series Foundation Models
Organizations: Siemens AG · LMU Munich · Munich Center for Machine Learning (MCML)
Abstract
Time series foundation models (TSFMs) are increasingly used for probabilistic forecasting across diverse domains, yet the reliability of their uncertainty estimates remains poorly understood. In this work, we study uncertainty calibration in probabilistic TSFMs from a broader methodological and empirical perspective. We introduce a taxonomy of how TSFMs represent predictive uncertainty and develop a fine-grained evaluation framework for calibration in multi-horizon forecasting that distinguishes pooled, horizon-wise, and stronger conditional notions of calibration. Using this framework, we conduct a comprehensive empirical evaluation of popular TSFMs across diverse datasets and show that current models exhibit distinct and systematic patterns of miscalibration that standard metrics often obscure. We further analyze common post-hoc recalibration methods and characterize which forms of calibration they can improve and which tend to persist. Our results provide practical guidance for evaluating and improving uncertainty calibration in TSFMs and highlight important limitations of current practice.
Figures & tables
| # | Category | How is obtained | Training loss | Representative models |
|---|---|---|---|---|
| 1 | Parametric Density | The model predicts parameters of a conditional density family ; the predictive CDF may be obtained analytically or via sampling in practice. | Negative log-likelihood | Lag-Llama Rasul et al. (2023) , MOIRAI 1.x Woo et al. (2024) , MOIRAI-MoE Liu et al. (2024) , Toto 1.0 Cohen et al. (2025) |
| 2 | Discretized Distribution | The model predicts discrete probabilities over predefined value intervals; the predictive CDF is obtained by cumulative mass over the discretized support, interpolated within bins for bar distributions. | Cross-entropy / discretized log-likelihood | Chronos-T5 Ansari et al. (2024a) , TabPFN-TS Hoo et al. (2025) |
| 3 | Direct Quantile | The model predicts at finitely many levels ; uncertainty is therefore typically available only at the resolution of the chosen quantile grid. | Pinball loss / weighted quantile loss | Chronos-Bolt Ansari et al. (2024b) , Chronos-2 Ansari et al. (2025) , MOIRAI 2.0 Liu et al. (2025a) , TiRex Auer et al. (2025) , TimesFM Das et al. (2024) , KAIROS Feng et al. (2025) , FlowState Graf et al. (2026) , YINGLONG Wang et al. (2025) |
| 4 | Implicit Generative Process | The model generates forecast trajectories from an implicit joint predictive law; marginal predictive CDFs are estimated empirically from Monte Carlo samples. | Flow matching / diffusion denoising | Sundial Liu et al. (2025b) , TimeDiT Cao et al. (2024) |
| % significant | |||||||||||
| Cat. | Model | raw | corr. | ||||||||
| 1 | Moirai-Base | 0.0536 | 0.0829 | 0.1579 | 0.1215 | 0.1862 | 0.0293 | 0.0386 | 0.0199 | 76% | 76% |
| 1 | Moirai-Large | 0.0484 | 0.0776 | 0.1581 | 0.1225 | 0.1906 | 0.0292 | 0.0448 | 0.0249 | 80% | 82% |
| 1 | Moirai-MoE-Base | 0.0533 | 0.0826 | 0.1537 | 0.1204 | 0.1878 | 0.0293 | 0.0379 | 0.0194 | 84% | 76% |
| 1 | Moirai-MoE-Small | 0.0521 | 0.0837 | 0.1578 | 0.1234 | 0.1908 | 0.0317 | 0.0397 | 0.0215 | 88% | 76% |
| 1 | Moirai-Small | 0.0530 | 0.0855 | 0.1607 | 0.1283 | 0.1982 | 0.0326 | 0.0428 | 0.0238 | 86% | 73% |
| Model | IR | KDE | PS | IR | KDE | PS | IR | KDE | PS |
|---|---|---|---|---|---|---|---|---|---|
| Moirai-Base | 13.36% | 12.93% | 12.16% | 4.77% | 4.58% | 4.08% | 5.79% | 5.66% | 4.95% |
| Moirai-Large | 12.44% | 11.78% | 9.08% | 0.52% | 0.29% | 0.95% | 1.57% | 1.46% | 0.49% |
| Moirai-MoE-Base | 19.72% | 19.62% | 13.91% | 5.83% | 5.77% | 4.09% | 5.51% | 5.43% | 4.21% |
| Moirai-MoE-Small | 18.77% | 19.05% | 16.24% | 7.78% | 7.82% | 7.07% | 5.62% | 5.58% | 4.92% |
| Moirai-Small | 9.29% | 9.45% | 7.89% | 3.55% | 3.44% | 2.30% | 2.23% | 2.32% | 1.59% |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Regime | Predictive distribution | Quantile function | Outcome | Coverage |
|---|---|---|---|---|
| \begin{array}[]{c}2\eta\,\mathrm{U}[-\tfrac{1}{4},0]\\ +\,(1-2\eta)\,\mathrm{U}[1,2]\end{array} |
| UQ Category | Model | HuggingFace ID / Package |
| Cat. 1: Parametric | Moirai-Small | Salesforce/moirai-1.1-R-small |
| Moirai-Base | Salesforce/moirai-1.1-R-base | |
| Moirai-Large | Salesforce/moirai-1.1-R-large | |
| Moirai-MoE-Small | Salesforce/moirai-moe-1.0-R-small | |
| Moirai-MoE-Base | Salesforce/moirai-moe-1.0-R-base | |
| Cat. 2: Discretized | Chronos-T5-Small | amazon/chronos-t5-small |
| permutation test | bootstrap test | |||||
|---|---|---|---|---|---|---|
| Scenario | ||||||
| Calibrated (size) | 0.05 | 0.04 | 0.04 | 0.06 | 0.04 | 0.08 |
| Value-dependent spread | 1.00 | 0.99 | 0.48 | 0.03 | 0.06 | 0.06 |
| Horizon-dependent spread | 0.03 | 0.06 | 0.05 | 1.00 | 1.00 | 1.00 |
| % significant | ||||||
|---|---|---|---|---|---|---|
| Windows | Pairs | raw | noise | corrected | ||
| 98 | 0.063 | 0.043 | 0.020 | 37% | 61% | |
| 50–199 | 98 | 0.023 | 0.016 | 0.007 | 47% | 68% |
| 200–499 | 196 | 0.024 | 0.010 | 0.014 | 76% | 94% |
| 500–999 | 182 | 0.023 | 0.007 | 0.015 | 68% | 80% |
| 112 | 0.035 | 0.011 | 0.024 | 79% | 95% | |
| PCE | QCE | Diag. | ||||||
| Cat. | Model | pool | avg | max | avg | max | ||
| 1 | Moirai-Base | 0.0536 | 0.0829 | 0.1579 | 0.1215 | 0.1862 | 0.0293 | 0.0386 |
| 1 | Moirai-Large | 0.0484 | 0.0776 | 0.1581 | 0.1225 | 0.1906 | 0.0292 | 0.0448 |
| 1 | Moirai-MoE-Base | 0.0533 | 0.0826 | 0.1537 | 0.1204 | 0.1878 | 0.0293 | 0.0379 |
| 1 | Moirai-MoE-Small | 0.0521 | 0.0837 | 0.1578 | 0.1234 | 0.1908 | 0.0317 | 0.0397 |
| 1 | Moirai-Small | 0.0530 | 0.0855 | 0.1607 | 0.1283 | 0.1982 | 0.0326 | 0.0428 |
| below | above | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Dataset | corr. | lowest | highest | lowest | highest | point mass | |||
| Chronos-T5-Small | LOOP_SEATTLE/5T | 0.031 | 0.034 | 0.079 | 0.030 | 15% | 22% | 29% | 4% | 0% |
| Moirai-Small | kdd_cup_2018/H | 0.015 | 0.032 | 0.071 | 0.028 | 2% | 26% | 12% | 9% | 2% |
| Moirai-Base | electricity/D | 0.023 | 0.034 | 0.075 | 0.031 | 8% | 17% | 14% | 5% | 0% |
| TiRex | electricity/W | 0.009 | 0.039 | 0.070 | 0.020 | 6% | 12% | 11% | 4% | 0% |
| TabPFN-TS | m4_hourly | 0.036 | 0.050 | 0.088 | 0.030 | 6% | 15% | 17% | 10% | 1% |
| Model | IR | KDE | PS | IR | KDE | PS | IR | KDE | PS | IR | KDE | PS |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Moirai-Base | 1.67% | 1.53% | 1.98% | 1.80% | 1.75% | 2.31% | 9.86% | 9.64% | 9.69% | 8.46% | 8.48% | 7.22% |
| Moirai-Large | 0.86% | 0.73% | 0.40% | 0.76% | 0.76% | 0.02% | 17.00% | 16.62% | 15.70% | 3.71% | 3.87% | 3.44% |
| Moirai-MoE-Base | 3.43% | 3.37% | 3.06% | 3.00% | 3.06% | 2.59% | 16.67% | 16.68% | 11.82% | 4.68% | 4.54% | 4.51% |
| Moirai-MoE-Small | 0.33% | 0.31% | 0.20% | 0.63% | 0.58% | 0.58% | 7.00% | 7.30% | 5.27% | 0.19% | 0.03% | 0.50% |
| Moirai-Small | 2.64% | 2.78% | 2.13% | 1.46% | 1.44% | 1.04% | 4.69% | 5.18% | 5.72% | 0.93% | 0.38% | 0.11% |
| Model | IR | KDE | PS | IR | KDE | PS | IR | KDE | PS |
|---|---|---|---|---|---|---|---|---|---|
| Moirai-Base | 53.45% | 53.23% | 46.30% | 22.56% | 22.39% | 20.86% | 14.60% | 14.47% | 13.40% |
| Moirai-Large | 44.36% | 44.47% | 39.68% | 14.84% | 14.72% | 13.80% | 9.53% | 9.45% | 8.76% |
| Moirai-MoE-Base | 57.14% | 56.55% | 48.77% | 22.02% | 21.82% | 19.81% | 14.50% | 14.30% | 12.83% |
| Moirai-MoE-Small | 59.97% | 59.06% | 48.86% | 22.22% | 22.03% | 18.87% | 13.64% | 13.48% | 11.85% |
| Moirai-Small | 47.60% | 47.43% | 38.48% | 17.46% | 17.41% | 14.40% | 9.14% | 9.16% | 7.41% |
| A: fitted on all 13 levels | B: fitted on 0.1–0.9 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| None | IR | HB | KDE | PS | IR | HB | KDE | PS | |
| 0.01 | 0.077 | 0.019 | 0.061 | 0.029 | 0.031 | 0.103 | 0.060 | 0.066 | 0.039 |
| 0.05 | 0.098 | 0.030 | 0.030 | 0.030 | 0.034 | 0.058 | 0.058 | 0.057 | 0.041 |
| 0.1–0.9 | 0.091 | 0.047 | 0.047 | 0.047 | 0.050 | 0.047 | 0.047 | 0.047 | 0.049 |
| 0.95 | 0.078 | 0.029 | 0.029 | 0.031 | 0.032 | 0.053 | 0.053 | 0.053 | 0.034 |
| 0.99 | 0.053 | 0.017 | 0.041 | 0.021 | 0.022 | 0.106 | 0.037 | 0.064 | 0.027 |
| Moirai-Base | Chronos-T5-S | Chronos-2 | Sundial | |||||
|---|---|---|---|---|---|---|---|---|
| PCE h | QCE h | PCE h | QCE h | PCE h | QCE h | PCE h | QCE h | |
| 0.01 | 0.010 | 0.015 | 0.133 | 0.135 | 0.010 | 0.014 | 0.154 | 0.156 |
| 0.05 | 0.031 | 0.046 | 0.143 | 0.150 | 0.033 | 0.048 | 0.196 | 0.203 |
| 0.10 | 0.051 | 0.074 | 0.143 | 0.154 | 0.058 | 0.081 | 0.202 | 0.214 |
| 0.20 | 0.075 | 0.108 | 0.134 | 0.149 | 0.087 | 0.117 | 0.180 | 0.201 |
| 0.30 | 0.084 | 0.124 | 0.123 | 0.142 | 0.101 | 0.130 | 0.150 | 0.177 |
| Model | |||||
|---|---|---|---|---|---|
| Moirai-Base | 20 | 0.0499 | 0.0734 | 0.1084 | 0.0350 |
| Moirai-Base | 50 | 0.0463 | 0.0717 | 0.1038 | 0.0322 |
| Moirai-Base | 100 | 0.0461 | 0.0717 | 0.1025 | 0.0308 |
| Moirai-Base | 200 | 0.0462 | 0.0720 | 0.1021 | 0.0301 |
| Moirai-Base | 500 | 0.0463 | 0.0722 | 0.1018 | 0.0296 |
| Chronos-T5-Small | 20 | 0.1084 | 0.1249 | 0.1409 | 0.0160 |
| Task | avg. | ||||
|---|---|---|---|---|---|
| Short | 49 | 0.056 | 0.073 | 0.130 | 0.017 |
| Medium | 491 | 0.085 | 0.125 | 0.260 | 0.040 |
| Long | 737 | 0.097 | 0.144 | 0.297 | 0.047 |
Uncertainty Calibration
Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.
See all 292 papers