Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting

Pengyu Nie; Chenglang Xu; Yaoshi Chen; Chaogan Ren; Wei Hu; Chao Yang; Jiangong Zhang

cs.LGSep 30, 2026

Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting

Authors: Pengyu Nie, Chenglang Xu, Yaoshi Chen, Chaogan Ren, Wei Hu, Chao Yang, Jiangong Zhang

Organizations: JD.com, Inc.

Abstract

Time-series forecasting models achieve strong benchmark performance but exhibit severe systematic bias in industrial deployments. This train--deploy gap is conventionally attributed to temporal-structural errors or distribution shifts. We characterize a complementary source that these explanations overlook: canonical losses embed fixed statistical priors, while industrial demand mixes benign and pathological regimes---zero-inflation, skewness, high variability---in which these priors are systematically violated. The induced bias persists even under perfect temporal modeling, remains in a distributional-shape component that normalization cannot remove, and creates an aggregation trade-off invisible to aggregate metrics. We turn these observations into an evaluation toolkit centered on the Regime-wise Relative Bias Vector (RBV): a metric-agnostic, regime-decomposed diagnostic that audits how pooled training allocates systematic mismatch across pathological subpopulations. A controlled attribution analysis decomposes RBV into a model-independent intrinsic floor, set by each loss's estimand, and an excess component attributable to training, tracing observed bias to the loss rather than the model. A large-scale study---13 loss objectives, 3 seeds, 60,000+ series spanning RetailShiftBench and M5, with random-split controls---shows that regime-aware diagnosis separates optimization-type from bias-type failure, and that regime-aware training resolves the pooling-induced bias that capacity scaling cannot, for mean-type losses. A formal structural observation, that risk under evaluation-distribution contamination is affine in the pathology mixture weight, grounds these findings. Our work complements model ranking with mechanism-grounded, regime-oriented evaluation.

Figures & tables

Figure 1: Evolution of time-series forecasting evaluation paradigms.

Loss	Implicit Distributional Priors	Industrial Departure	Mismatch Mechanism
MSE	Symmetric light-tailed residuals, stable finite variance; mean as optimal fitting target	Heavy-tailed skewed demand with unstable variance	Quadratic penalty amplifies burst residuals and averages out tail signals; the mean estimand systematically overshoots sparse regimes.
MAE	Symmetric continuous distribution; median as optimal estimation target	Skewed, burst-heavy demand with frequent zero-inflation	Zeros and a long upper tail pull the median below the mean; the burst mass carrying most expected demand is discarded, widening the gap
Quantile	Stationary distribution; fixed quantile targets valid across sub-populations; non-degenerate tails	Zero-inflated baseline with skewed long-tail demand	Zero-inflation collapses lower-tail quantiles; the estimand degenerates into a step function under sparse bursts
Poisson	Variance $=$ mean; discrete counts; mean estimand under exponential curvature	Over-dispersed burst clustering with variance exceeding the mean	Burst clustering breaks equi-dispersion and mis-sets the fixed exponential curvature; the mass redistribution is invisible to linear-residual metrics
Huber	Stationary residual scale; outliers are genuine rare anomalies	Heavy-tailed baseline with frequent extreme demand bursts	A fixed threshold clips legitimate bursts as outliers; the effective estimand drifts with per-cell residual scale
Tweedie	Fixed dispersion and Tweedie-power parameters shared across all samples	Heterogeneous SKU-level dispersion under varying industrial statistical regimes	Static dispersion parameters fail to fit heterogeneous regime-wise distributions, leaving the estimand mismatched to varying statistical regimes.

Table 1: Statistical Assumptions and Deployment Mismatch Patterns

Semantics	Regime	XGB						TFT						Rule
Semantics	Regime	MSE	MAE	QL 0.60	Huber	Tweedie	Poisson	MSE	MAE	QL 0.60	Huber	Tweedie	Poisson	Rule
Bias	All	0.035	-0.088	0.045	-0.035	0.025	-0.01	-0.004	-0.076	0.05	-0.03	-0.039	-0.086	0.033
	Benign	0.022	-0.03	0.073	-0.02	0.017	-0.043	0.015	0.016	0.095	0.022	0.038	-0.001	0.031
	Skew	0.061	-0.185	-0.003	-0.051	0.044	0.048	-0.039	-0.232	-0.043	-0.116	-0.17	-0.223	0.045
	CV	0.262	-0.685	-0.421	-0.042	0.176	0.341	-0.33	-0.894	-0.726	-0.437	-0.838	-0.868	0.057
	ZIR	0.116	-0.385	-0.114	-0.110	0.085	0.156	-0.082	-0.53	-0.182	-0.277	-0.443	-0.532	0.038

Table 2: Systematic bias and cross-regime vector misalignment (RBV) under hierarchical model-loss.

	XGB				TFT				Rule
Metric	MSE	MAE	Huber	Tweedie	MSE	MAE	Huber	Tweedie	Rule
RBV	$0.243{\to}0.041$ ( $-83.1\%$ )	$0.675{\to}0.899$ ( $+33.2\%$ )	$0.077{\to}0.262$ ( $+240.3\%$ )	$0.163{\to}0.014$ ( $-91.4\%$ )	$0.335{\to}0.14$ ( $-58.2\%$ )	$0.914{\to}0.863$ ( $-5.6\%$ )	$0.459{\to}0.572$ ( $+24.6\%$ )	$0.873{\to}0.881$ ( $+0.9\%$ )	$0.033$
WMAPE	$0.4679{\to}0.462$ ( $-1.3\%$ )	$0.4270{\to}0.4255$ ( $-0.4\%$ )	$0.4670{\to}0.4765$ ( $+2.0\%$ )	$0.4593{\to}0.4539$ ( $-1.2\%$ )	$0.4675{\to}0.4730$ ( $+1.2\%$ )	$0.4475{\to}0.4487$ ( $+0.3\%$ )	$0.4603{\to}0.4595$ ( $-0.2\%$ )	$0.4514{\to}0.4549$ ( $+0.8\%$ )	$0.5182$
MSE	$11.909{\to}11.068$ ( $-7.1\%$ )	$12.057{\to}11.524$ ( $-4.4\%$ )	$13.575{\to}20.176$ ( $+48.6\%$ )	$11.992{\to}11.465$ ( $-4.4\%$ )	$15.68{\to}14.55$ ( $-7.2\%$ )	$15.182{\to}14.40$ ( $-5.2\%$ )	$14.140{\to}14.84$ ( $+5.0\%$ )	$14.714{\to}14.274$ ( $-3.0\%$ )	$16.452$

Table 3: Effect of pathology decomposition.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Conceptual schematic of the aggregation-breadth trade-off between distributional-statistical bias and estimation variance. The global-pooled, random-cell (R2, random-label control), and pathology-cell markers correspond to measured configurations (Tab. 3 ; § D.2 , Tab. 8 ); remaining markers are positional schematics without measured magnitude. Measured anchors are consistent with, but do not locate, a total-error minimum near pathology-aware subgrouping.

Variant	Base functional $\phi_{r}$	Units	Question answered
Pathology bias-RBV (default, § 4.2 )	$\frac{\sum_{i\in I_{r}}(\hat{y}_{i}-y_{i})}{\sum_{i\in I_{r}}\|y_{i}\|}$	demand units	Which regime is systematically over-/under-estimated, and by how much?
Quantile RBV	$\phi_{r}(\tau)=E_{r}(\tau)-\tau$	probability	Where – and at which $\tau$ – is calibration error concentrated?
Share-normalized RBV	$B_{r}/\hat{S}_{r}$	dimensionless (relative bias)	How is mismatch severity allocated per unit of business volume?

Appendix

Table 4: The RBV family: one template, three instantiations. All variants share the same centering and

\ell_{2}

aggregation, and differ only in the base functional

\phi_{r}

E_{r}(\tau)

is the empirical exceedance under the mid-convention

\widehat{\Pr}[\hat{q}_{\tau}>y]+\tfrac{1}{2}\,\widehat{\Pr}[\hat{q}_{\tau}=y]

(Figs. 8 , 11 );

\hat{S}_{r}=\sum_{i\in I_{r}}\hat{y}_{i}

is the regime-level predicted total, and

B_{r}=\big(\sum_{i\in I_{r}}(\hat{y}_{i}-y_{i})\big)/\big(\sum_{i\in I_{r}}|y_{i}|\big)

denotes the regime-wise pooled bias of § 4.2 .

Loss	$B$ [95% CI]	RBV [95% CI]
MAE	$-0.257$ $[-0.267,-0.248]$	$0.409$ $[0.391,0.427]$
MSE	$0.058$ $[0.048,0.068]$	$0.419$ $[0.387,0.450]$
Huber	$-0.107$ $[-0.115,-0.098]$	$0.241$ $[0.215,0.265]$
Tweedie	$0.054$ $[0.044,0.063]$	$0.378$ $[0.350,0.407]$
Poisson	$0.058$ $[0.049,0.068]$	$0.477$ $[0.445,0.507]$
Quantile $\tau{=}0.1$	$-0.859$ $[-0.866,-0.852]$	$0.229$ $[0.218,0.242]$

Appendix

Table 5: Bootstrap uncertainty (SKU-level,

B{=}1000

) for the pooled bias

B

and regime bias variation (RBV), single-origin M5 protocol. All RBV intervals exclude zero.

Figure 3: One-at-a-time threshold sensitivity of regime-decomposed RBV on M5. Each panel varies one threshold (

t_{\rm zir}

t_{\rm skew}

t_{\rm cv}

) with the other two fixed at baseline (

0.20/1.00/1.50

, dotted line); the remaining 9 losses behave similarly and are omitted for clarity. Rankings and the U-shaped RBV–

\tau

structure are invariant across all threshold settings; the dominant sensitivity is along

t_{\rm cv}

, whose spread tracks the extremity of the CV cell (89–10,746 SKUs across settings) rather than the threshold definition itself. Note: for this sensitivity analysis, regime cells are formed directly from the continuous pathology statistics (ZIR rate, skewness, CV) without applying the § 3 thresholds, so RBV magnitudes are not directly comparable to Table 2 (which uses the thresholded regime labels); only rankings and shapes are comparable.

Figure 4: Synthetic illustration and parameter sensitivity for Prop. B.1 . (a) Example trajectory under perfect temporal modeling $\mathcal{B}_{temp}=0$ . (b) Quantitative comparison of loss-implied predictors (MSE-opt, MAE-opt, and a fixed-quantile target) under pinball evaluation. (c) Ablation over quantile level $\tau$ : the ranking of predictors flips as the target functional varies, illustrating that the optimal estimator is prior-dependent. (d) Ablation over zero-inflation rate $p_{0}$ . This figure serves as an illustrative synthetic example; formal statements and proofs are given in the main text.

Figure 5: Dose-response ablation over three core pathological dimensions: zero-inflation rate (ZIR), skewness, and within-series scale variability (CV). We report squared and absolute deviation of the fitted predictor from the clean conditional trajectory as pathological intensity increases. Consistently, both deviation metrics rise monotonically with pathological severity, demonstrating that pathological marginals systematically corrupt the recovery of the underlying signal. This synthetic experiment complements the theoretical analysis in Prop. B.1 .

Figure 6: Coupled-pathology performance across varying $\alpha$ . The linear-sum baseline aggregates excess losses from isolated-ZIR and isolated-CV runs. Pink shaded regions highlight regimes where coupled loss strictly exceeds this reference, indicating emergent super-additive degradation.

Figure 7: RetailShiftBench temporal structure. Aggregate daily sales across all 31,213 series over the full evaluation panel. Sales are dominated by a stable weekly periodicity with no trend breaks or regime shifts, indicating that the benchmark’s difficulty stems from cross-sectional distribution heterogeneity rather than temporal non-stationarity.

Pathology	Retail	Energy / Grid	Manufacturing / IoT	Finance
Skew	Promo bursts; seasonal peaks	Extreme weather load spikes	Defect clustering; rework spikes	Jump‑driven return asymmetry; earnings announcement effects
ZIR	Stockouts; intermittently stocked SKUs	Turbine downtime; zero irradiance	Sensor dropout; line stoppages	Halted trading; zero‑tick intervals in illiquid assets
CV	Long‑tail SKU heterogeneity	Ramping‑driven output dispersion across seasons	Batch‑to‑batch variance	Cross‑sectional dispersion of return scales
Coupled Distributional Pathologies	Sporadic luxury bursts; Skew‑CV: high‑variance promo series	Zero‑output + sudden load spikes; weather‑driven price spikes	Stoppage + large‑batch orders; high‑variance defect batches	Illiquidity + price jumps; earnings‑driven clustered volatility
Coupling with Temporal‑Structure	Pathology + holiday / lifecycle shifts	Pathology + policy / tariff changes	Pathology + line changeovers	Pathology + macro regime transitions

Appendix

Table 6: Cross‑domain instantiation of the pathology taxonomy.

Dataset	#Series	Domain & Granularity	Low Skewness	Low CV	Low ZIR	PCS	Benchmark Grade
ETT	28	Energy (transformer temp), hourly/15-min	0.5714	0.8571	0.8571	0.2381	Unqualified; basic temporal verification only
M4-Daily	4,227	Mixed (macro $\sim$ micro), daily	0.863	0.9970	1.000	0.046	Insufficient pathological spectrum
M4-Hourly	414	Mixed (macro $\sim$ micro), hourly	0.865	0.993	1.0	0.048	Insufficient pathological spectrum
M4-Monthly	48,000	Mixed (macro $\sim$ micro), monthly	0.845	0.999	1.0	0.052	Insufficient pathological spectrum
M4-Quarterly	24,000	Mixed (macro $\sim$ micro), quarterly	0.849	0.999	1.0	0.051	Insufficient pathological spectrum
M4-Weekly	359	Mixed (macro $\sim$ micro), weekly	0.652	0.997	1.0	0.117	Insufficient pathological spectrum

Appendix

Table 7: Pathology coverage and industrial validity of mainstream forecasting benchmarks. Higher PCS represents stronger industrial pathological spectrum coverage and better deployment-aligned robustness evaluation capability.

Figure 8: Quantile calibration as an attribution test (XGBoost, single- $\tau$ pinball objectives, tree depth fixed across $\tau$ ). Coverage is computed on positive-demand days with the mid convention $P(\hat{y}>y)+\tfrac{1}{2}P(\hat{y}=y)$ ; the dashed line marks ideal calibration. Left : global pooling glues pathological regimes below the diagonal while benign stays over-covered. Middle : random-label decomposition with cells matched in count and size to the pathology partition (permutation without replacement) reproduces the glued pattern—granularity alone does not release the curves. Right : pathology-informed decomposition releases the bundle toward the diagonal, with the residual gap at $\tau{=}0.9$ marking the variance wall. Near-zero low- $\tau$ coverage on pathological regimes reflects the estimand floor of zero-inflated marginals (§ 4.2 ), not additional miscalibration.

	non-linear loss family
Metric	MSE	Tweedie
RBV	$0.243\rightarrow 0.26\rightarrow 0.041\downarrow$	$0.163\rightarrow 0.151\rightarrow 0.014\downarrow$
WMAPE	$0.4679\rightarrow 0.4681\rightarrow 0.462\downarrow$	$0.4593\rightarrow 0.4606\rightarrow 0.4539\downarrow$
MSE	$11.909\rightarrow 12.615\rightarrow 11.068\downarrow$	$11.992\rightarrow 12.678\rightarrow 11.465\downarrow$
	linear loss family
Metric	MAE	Huber

Appendix

Table 8: Loss-function trajectory grid under grouping-scheme interventions (XGB).

Figure 9: Capacity scaling (solid) vs. regime decomposition (dashed) under the MSE loss. (a) RBV (log scale): pooled depth-2 RBV improves $7\times$ ( $0.837\to 0.120$ ) with $32\times$ the trees, and the best pooled configuration overall (1600 trees, depth 4) saturates at a $0.116$ ceiling—decomposition (dashed) holds RBV $\leq 0.06$ across all but the most extreme configurations (within-cell overfitting at 1600 trees, depth 12). (b) MSE: decomposition improves aggregate accuracy as well. WMAPE varies within $0.46$ – $0.50$ across all conditions and is omitted.

Figure 10: Same setup as Fig. 9 under the MAE loss. Decomposition increases RBV (dashed curves lie at or above solid ones in (a)): routing removes the benign mass that dilutes MAE’s within-cell estimand mismatch, exposing the mean–median gap rather than repairing it—whereas aggregate error (b) still improves.

	non-linear loss family
Metric	MSE	Tweedie	Poisson
RBV	$0.422\rightarrow 0.436{\scriptscriptstyle\pm 0.0071}\rightarrow 0.036\downarrow$	$0.377\rightarrow 0.359{\scriptscriptstyle\pm 0.0019}\rightarrow 0.062\downarrow$	$0.476\rightarrow 0.487{\scriptscriptstyle\pm 0.0018}\rightarrow 0.044\downarrow$
WMAPE	$0.7597\rightarrow 0.7612{\scriptscriptstyle\pm 0.0004}\rightarrow 0.7337\downarrow$	$0.7553\rightarrow 0.7526{\scriptscriptstyle\pm 0.0005}\rightarrow 0.7259\downarrow$	$0.7612\rightarrow 0.7612{\scriptscriptstyle\pm 0.0001}\rightarrow 0.7326\downarrow$
MSE	$4.784\rightarrow 4.745{\scriptscriptstyle\pm 0.020}\rightarrow 4.607\downarrow$	$4.758\rightarrow 4.699{\scriptscriptstyle\pm 0.020}\rightarrow 4.493\downarrow$	$4.753\rightarrow 4.690{\scriptscriptstyle\pm 0.025}\rightarrow 4.505\downarrow$
	linear loss family
Metric	MAE	QL60	Huber

Appendix

Table 9: Loss-function trajectory grid under grouping-scheme interventions on M5 (XGB, fixed-origin protocol; mean over three seeds 42, 41, 39).

Figure 11: Quantile-calibration attribution on M5 (mean over three seeds 42, 41, 39; $\tau\in\{0.1,\dots,0.9\}$ ; $\tau{=}0.5$ shown via MAE, since $\mathrm{pinball}(0.5)\equiv\mathrm{MAE}$ ). Exceedance uses the mid convention $\Pr[\hat{y}>y]+\tfrac{1}{2}\Pr[\hat{y}=y]$ . Left: global pooling — the aggregate curve hugs the diagonal while regime curves fan out. Center: random decomposition (size-matched control) — reproduces the pooled panel, isolating label informativeness. Right: regime decomposition — high- $\tau$ curves are released to the diagonal while the High-CV collapse at mid $\tau$ is exposed. Low- $\tau$ elevations reflect the zero-inflation estimand floor ( $\mathrm{zir}/2$ ), not miscalibration.

	RBV			WMAPE
Loss	All	Random	Regime	All	Random	Regime
MSE	0.473	0.472	0.400 $\downarrow$	0.7526	0.7381	0.6764 $\downarrow$
MAE	0.428	0.454	0.416 $\downarrow$	0.6660	0.6918	0.6625 $\downarrow$
QL60	0.420	0.437	0.480 $\uparrow$	0.6791	0.7096	0.6910 $\uparrow$
Huber	0.397	0.390	0.369 $\downarrow$	0.6911	0.6887	0.6659 $\downarrow$
Tweedie	0.500	0.469	0.421 $\downarrow$	0.7818	0.7236	0.6810 $\downarrow$

Appendix

Table 10: Regime decomposition on the M5 deep backbone (NHiTS). Each cell reports All

\to

Random

\to

Regime-wise training. All: global pooled training; Random: random split with cells matched in count and size to the pathology partition (mean over three shuffle seeds); Regime-wise: per-cell training on the skew–CV–ZIR partition. Single seed (42) for All and Regime-wise.

Method	$B_{\mathrm{benign}}$	$B_{\mathrm{zir}}$	$B_{\mathrm{skew}}$	$B_{\mathrm{cv}}$	$B$	RBV	WMAPE
Croston	$0.098$	$0.077$	$0.086$	$0.100$	$0.087$	0.020	0.739
SBA	$0.043$	$0.023$	$0.031$	$0.045$	$0.033$	0.019	0.725
TSB	$0.095$	$0.062$	$0.076$	$0.061$	$0.075$	0.028	0.734
TSB-HB	$0.057$	$0.035$	$0.036$	$0.059$	$0.043$	0.024	0.739
Oracle mean	$0.068$	$0.026$	$0.035$	$0.023$	$0.040$	0.036	0.738
Oracle median	$0.001$	$-0.456$	$-0.390$	$-0.941$	$-0.353$	0.695	0.668

Appendix

Table 11: Intermittent-demand baselines and oracle references on the M5 single-origin protocol. Bias

B_{r}

follows the paper’s sample-pooled sum-ratio convention, and RBV is the unweighted

\ell_{2}

norm of the centered bias vector (identical to Tab. 8 ). Excess bias is relative to oracle-mean. Croston’s uniformly positive bias is its known over-prediction; SBA’s damping factor largely removes it. Oracle-median achieves the best WMAPE yet the largest regime bias variation: accuracy under MAE does not imply calibration for the pooled-mean estimand.

estimand	$B_{\rm benign}$	$B_{\rm zir}$	$B_{\rm skew}$	$B_{\rm cv}$	$B_{\rm global}$	RBV	WMAPE
mean (quadrature)	$0.114$	$-0.119$	$-0.060$	$-0.328$	$0.008$	$\mathbf{0.381}$	$0.703$
median ( $q_{0.5}$ )	$0.070$	$-0.493$	$-0.368$	$-0.853$	$-0.194$	$0.789$	$\mathbf{0.672}$

Appendix

Table 12: Chronos-Bolt zero-shot on the M5 single-origin protocol (App. E.1 ), horizons 1–7 pooled (

n{=}187{,}222

samples). Mean vs. median estimands decoded from the same frozen forward pass. Bias

=\sum(\hat{y}-y)/\sum|y|

per cell; RBV and WMAPE as in Tab. 2 .

Figure 12: Chronos-Bolt zero-shot evaluation on M5. Left: RBV and aggregate WMAPE as the decoding quantile

\tau

is swept on a frozen checkpoint; the dotted vertical line marks

\tau^{\ast}{=}0.4

, the quantile minimizing aggregate WMAPE. RBV(

\tau

) is U-shaped and attains its minimum at the quadrature-mean estimand (Tab. 12 ). Right: per-regime exceed rate,

\Pr[\hat{y}>y]+0.5\Pr[\hat{y}=y]

, of zero-shot quantile decoding against the diagonal: quantile heads over-cover throughout the mid range (exceed

0.574

at nominal

0.5

), the high-CV segment shows the largest over-coverage, and calibration recovers at high

\tau

. Unlike Fig. 8 , the low-

\tau

end shows no zero-inflation floor—the continuous quantile heads emit no atom at zero, so the ZIR curve tracks the diagonal while the floor in Fig. 11 is a construct of discrete zero-mass predictors. Negative predictions (

65\%

q_{0.1}

) mechanically depress the exceed rate at low

\tau

(App. E.3 ).

Explore similar work

Jun 1, 2026cs.LG

Regime-Arrival Uncertainty in Generalization Bounds under Distribution Shift

The standard generalization bounds assume that the training and deployment distributions are the same, or are static, and don't consider regime switching environments where the ratio of calm vs crisis states is different. This paper proposes a framework that generalizes regime-aware models by quantifying the extra risk due to regime composition mismatch, when distribution shifts are Markov-switching. We obtain an exact decomposition, separating regime mismatch from regime sensitivity; we extend the bound to beta-mixing data using the effective sample size corrected for the spectral gap; and we show a minimax lower bound for synthetic data and on 25 years of global equity indices. The proposed penalty is an ex post realized generalization gap, whereas the training-only estimator does not show significant correlation: the feature geometry of crises can be detected, but not the temporal arrival. Thus, the framework is not a forecast machine. Forecasting the composition of the future regime is an open question in the rare cases of regime change.

Prince Poudel

Independent researcher.

Distribution Shifts Generalization Bounds Exponential Family Non-Stationarity Uncertainty

Ask PDF

Jun 16, 2026cs.LG

Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.

Yingshuo Wang, Xian Sun, Lingdong Kong +4

University of California, Berkeley, CA, USA · Duke University, Durham, NC, USA · National University of Singapore, Singapore +3

Time Series Foundation Models Accurate Traffic Forecasting Traffic Standard Traffic Speed Benchmarks

Ask PDF

Sep 27, 2026cs.LG

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.

Kangjia Yan, Linfeng Wang, Tianen Shen +7

East China Normal University, Shanghai, China. · University of Electronic Science and Technology of China, Chengdu, China. · Aalborg University, Aalborg, Denmark.

Time Series Time Series Foundation Models Skewness Bits

Ask PDF