Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external conditions, and expert expectations. However, reliable evaluation remains difficult: existing benchmarks often have limited domain coverage, lack standardized history-future splits, or provide text that is not guaranteed to inform the future. We introduce GIFT-Eval-CTX, a benchmark that isolates whether forecasting methods can translate verified future information into accurate numerical trajectories. Built upon GIFT-Eval, GIFT-Eval-CTX contains 10,000 instances across 20 datasets. Each instance provides structured effects derived from the realized future: Macro effects describe horizon-wide behavior, while Micro effects specify localized changes. We construct these annotations with an agentic pipeline combining LLM-based discovery and deterministic code validation. We benchmark TSFMs, frontier LLMs, and tool-using agentic systems. The results show that structured effects reduce aggregate relative MASE by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8 relative to history-only prompting. The evaluated MoiraiAgent system achieves the best overall result. We further analyze the contributions of structured effects and agentic systems to performance, and identify opportunities for future research.
Figures & tables
Figure 1: Illustration of agentic annotation, an annotated instance, and paired evaluation.
Method
Input
Short
Medium
Long
Overall
Seasonal naive
History
1.000
1.000
1.000
1.000
Chronos-2
History
0.764
0.803
0.848
0.795
Toto-2
History
0.765
0.811
0.851
0.798
Moirai-2
History
0.740
0.798
0.852
0.782
TimesFM-3
History
0.750
0.795
0.823
0.780
Claude Opus 4.8
History
0.967
1.325
1.206
1.109
Table 1: MASE on GIFT-Eval-CTX. “Effects” denotes the structured context supplied by the benchmark. The best/second-best result in each column is bold/underlined.
Method
Input
Short
Medium
Long
Overall
GPT-5.6 Sol
History only
0.851
1.071
1.049
0.953
History + Macro
0.601
0.992
1.009
0.780
History + Micro
0.520
0.796
0.815
0.650
History + Macro + Micro
0.417
0.727
0.782
0.565
MoiraiAgent
History only
0.740
0.798
0.852
0.782
History + Macro
0.605
0.802
0.858
0.711
Table 2: MASE under different structured-effect inputs. MoiraiAgent’s history-only performance is the same as Moirai-2.
Method
Micro family
Short
Medium
Long
Overall
Hist.
+Micro
Hist.
+Micro
Hist.
+Micro
Hist.
+Micro
GPT-5.6 Sol
Excursion
0.826
0.540
1.056
0.780
1.031
0.739
0.930
0.642
Mean shift
0.810
0.240
0.995
0.446
0.982
0.304
0.890
0.294
Seasonal change
0.857
0.753
1.104
1.025
1.070
1.119
0.975
0.914
MoiraiAgent
Excursion
0.708
0.498
0.811
0.628
0.862
0.584
0.770
0.550
Mean shift
0.699
0.235
0.864
0.405
0.900
0.300
0.780
0.283
Table 3: Comparison of MASE between history-only forecasting and forecasting with Micro effects. Each Micro family contains samples with at least one effect from that family.
Method
Input
Inside Micro regions
Outside Micro regions
Short
Medium
Long
Overall
Short
Medium
Long
Overall
GPT-5.6 Sol
History only
0.943
1.107
1.055
1.011
0.744
1.060
1.062
0.890
History + Micro
0.411
0.650
0.542
0.496
0.638
0.918
1.101
0.802
MoiraiAgent
History only
0.838
0.893
0.939
0.876
0.610
0.729
0.820
0.687
History + Micro
0.398
0.535
0.454
0.444
0.594
0.671
0.760
0.652
Table 4: MASE inside and outside labeled Micro regions.
Figure 2: Qualitative examples of the benefits provided by Micro effects. Left: Micro effects improve the unlabeled trajectory between two labeled events. Right: Micro effects preserve meaningful periodic dynamics even when the sample-level MASE does not improve.
Method
Short (T/P)
Medium (T/P)
Long (T/P)
Follow level range
Follow intensity
Claude Opus 4.8
0.0% / 0.2%
48.4% / 49.3%
34.0% / 64.8%
59.83%
40.79%
GPT-5.6 Sol
0.0% / 0.1%
54.0% / 7.1%
51.8% / 7.7%
97.41%
81.57%
MoiraiAgent
0.0% / 0.0%
0.0% / 0.0%
0.0% / 0.0%
99.98%
92.73%
Table 5: Analysis of output-length compliance and numerical constraint adherence. T/P denotes the truncation and padding rates, respectively. The final two columns report sample-level adherence to the specified level range and event intensities using a tolerance of five history-normalized points.
Figure 3: Qualitative limitations of MoiraiAgent. Left: MoiraiAgent replaces local fluctuations with extended plateaus in the BizITObs series. Right: it approximates the recovery from a sustained downward excursion with a nearly linear trajectory in the KDD Cup series. In both examples, sample-level MASE improves despite the loss of fine-grained temporal dynamics.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Instances
10,000
Source datasets
20
Dataset–frequency identifiers
38
Dataset–frequency–term configurations
78
Sampling frequencies
6
Distinct forecast horizons
7
Appendix
Table 6: Summary statistics for GIFT-Eval-CTX.
Source dataset
IDs
Groups
Short
Medium
Long
Total
LOOP_SEATTLE
3
7
1,332
685
306
2,323
M_DENSE
2
4
61
5
4
70
SZ_TAXI
2
4
279
86
73
438
bitbrains_fast_storage
2
4
799
257
112
1,168
bitbrains_rnd
2
4
681
100
97
878
bizitobs_application
1
2
1
1
0
2
Appendix
Table 7: Source-level composition of GIFT-Eval-CTX.
Frequency
Instances
Horizon
Instances
10S
79
30
1,791
5T
2,724
48
4,721
10T
543
60
47
15T
1,597
480
2,116
H
3,266
600
19
D
1,791
720
1,293
Appendix
Table 8: Distribution of sampling frequency and forecast horizon.
Family
Subtype
Count
Excursion
Impulse
10,625
Sustained
6,750
Recurrent
1,759
Mean shift
–
1,225
Seasonal change
Amplitude
2,391
Phase
29
Appendix
Table 9: Distribution of Micro families and subtypes.
Field
Type and allowed values
Definition
trend
List of one or two segment objects
Chronological decomposition of the horizon-wide center-line movement. Segment durations are positive and sum to one.
trend.direction
up , down , or flat
Direction of the center line within the segment. Flat means there is no meaningful overall rise or fall.
trend.shape
linear , curved , accelerating_curved , decelerating_curved , step , or null
Shape of a non-flat trend segment. The value is null when direction is flat .
trend.duration
Positive real number in (0,1]
Fraction of the forecast horizon occupied by the segment.
seasonality
Object or null
Dominant oscillatory structure of the future. It is null when no meaningful oscillation is present.
seasonality.period
Integer greater than one, or null
Dominant period in time steps. A null period denotes an irregular oscillation.
Appendix
Table 10: Complete Macro schema.
Figure 4: Representative examples of the Micro-effect taxonomy. Shaded regions indicate the inclusive intervals described by the effects.
Method
Model or checkpoint
Forecasting interface
Seasonal naive
–
Repeats the last observed seasonal cycle
Chronos-2
amazon/chronos-2
Direct median forecast
Toto-2
Datadog/Toto-2.0-313m
Direct median forecast
Moirai-2
Salesforce/moirai-2.0-R-small
Direct median forecast
TimesFM-3
google/timesfm-3.0-pytorch
Direct median forecast
GPT-5.6 Sol
GPT-5.6 Sol
Direct numerical generation
Appendix
Table 11: Model configurations used in the experiments.
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.