Severe but Harmless: Stress-Testing Shift Severity for Frozen Time-Series Forecasters
Organizations: School of Computing, University of Georgia, Boyd Research and Education Center, Athens, GA 30602, USA
Abstract
Zero-shot time-series foundation models (TSFMs) have become attractive alternatives to task-specific models because they handle diverse downstream tasks, including anomaly detection and forecasting, without per-series training. However, time-series data undergo distribution shifts, and the dissimilarity between two regimes is commonly scored with statistical measures such as Wasserstein-1, where a higher score is read as a more severe shift. In systematic experiments using three established severity measures on nine regime shifts from three standard traffic datasets, we show that these measures do not reliably predict degradation for instance-normalized TSFMs. The regimes rated most severe are not the regimes that degrade the models most, and degradation reaches 29% in the lowest-scoring regime. In most cases, the audit shows no positive relationship between measured severity and model degradation. We provide an equivariance-based stress-testing analysis that explains why severity readings can become decoupled from model degradation.
Figures & tables
| Task | Dataset | Wall time |
|---|---|---|
| Chronos-2 inference | All 3 datasets | 0.7 h |
| TimesFM inference | All 3 datasets | 192.8 h |
| Graph WaveNet training | LargeST-SD | 2.2 h |
| STAEformer training | LargeST-SD | 4.0 h |
| TTRA adapter | LargeST-SD S1 | 2.8 h |
| Table/figure generation | All results | s |
| S1 (Weekend) | S2 (Off-peak) | S3 (Spatial) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Dataset | C2 | TFM | C2 | TFM | C2 | TFM | |||
| METR-LA | 4.3 | 6.8 | 5.5 | ||||||
| PEMS-BAY | 4.2 | 7.4 | 3.5 | ||||||
| LargeST-SD | 55.1 | 142.2 | 92.6 | ||||||
| METR-LA (mph) | PEMS-BAY (mph) | LargeST-SD (veh/15 min) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Std | S1 | S2 | S3 | Std | S1 | S2 | S3 | Std | S1 | S2 | S3 |
| HA | 4.19 | 3.52 | 3.44 | 4.99 | 2.63 | 1.71 | 1.94 | 2.26 | 27.93 | 21.57 | 20.16 | 22.32 |
| ARIMA | 5.23 | 5.45 | 4.77 | 5.87 | 2.23 | 1.28 | 1.76 | 1.84 | 55.43 | 42.16 | 45.47 | 42.62 |
| DCRNN | 3.16 | 2.73 | 2.71 | 3.73 | 1.61 | 1.02 | 1.23 | 1.34 | 21.20 | 17.73 | 16.31 | 16.10 |
| Graph WaveNet | 3.04 | 2.52 | 2.64 | 3.59 | 1.58 | 1.00 | 1.22 | 1.31 | 18.50 | 16.21 | 14.19 | 14.64 |
| STAEformer | 2.94 | 2.47 | 2.58 | 3.45 | 1.54 | 0.98 | 1.18 | 1.24 | 18.44 | 16.17 | 13.65 | 14.34 |
| Dataset | S1 (Weekend) | S2 (Off-peak) | S3 (Spatial) |
|---|---|---|---|
| METR-LA | 0.905 | 0.905 | 1.000 |
| PEMS-BAY | 1.000 | 1.000 | 1.000 |
| LargeST-SD | 0.867 | 1.000 |
| Model | Intervention | Avg MAE |
|---|---|---|
| Chronos-2 | None | 26.14 |
| TimesFM | None | |
| Chronos-2 | TTRA adaptation | 27.80 |
| Chronos-2 | 48-hour context | 26.80 |
| Chronos-2 | 24-hour context | 29.76 |
| TimesFM | 48-hour context | 23.11 |
| ID | Prediction | Outcome | Verdict |
|---|---|---|---|
| P1 | Ranking lower under S1 than S2 | Held on LargeST-SD ( ); ties elsewhere | Partially held |
| P2 | Graph models degrade more than per-sensor models under S3 | Needs S3-retrained graph baselines; scoped out by protocol | Not testable |
| P3 | STAEformer top-2 under S2, drops ranks under S1 | Rank 1 under both shifts on every dataset evaluated | Falsified |
| P4 ′ | TTRA cuts frozen SD-S1 degradation by | Adapted MAE 27.80 frozen 26.14 at every horizon | Falsified |
| P5 | Inner-loop steps beat static registers | Not run; gated on the failed adaptation gate (Sec. IV-E ) | Not evaluated |
Time-Series Foundation Models
Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.
See all 344 papers