Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts

Yuze Dong; Jinsong Wu

cs.LGMar 10, 2026

Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts

Authors: Yuze Dong, Jinsong Wu

Organizations: School of Artificial Intelligence, Guilin University of Electronic Technology 1 Jinji Road, Guilin, 541004, Guangxi, China · Department of Electrical Engineering, University of Chile Av. Tupper 2007, Santiago, 8370451, Chile

Abstract

Time-series forecasting often faces challenges from non-stationarity, particularly distributional drift, where the data distribution evolves over time. This dynamic behavior can undermine the effectiveness of adaptive optimizers, such as Adam, which are typically designed for stationary objectives. In this paper, we revisit Adam in the context of non-stationary forecasting and identify that its second-order bias correction limits responsiveness to shifting loss landscapes. To address this, we propose TS_Adam, a lightweight variant that removes the second-order correction from the learning rate computation. This simple modification improves adaptability to distributional drift while preserving the optimizer core structure and requiring no additional hyperparameters. TS_Adam integrates easily into existing models and consistently improves performance across long- and short-term forecasting tasks. On the ETT datasets with the MICN model, it achieves an average reduction of 12.8% in MSE and 5.7% in MAE compared to Adam. These results underscore the practicality and versatility of TS_Adam as an effective optimization strategy for real-world forecasting scenarios involving non-stationary data. Code is available at: https://github.com/DD-459-1/TS_Adam.

Figures & tables

Dataset	Method	Model_1 (MICN)			Model_2 (PatchTST)			Model_3 (SegRNN)
Dataset	Method	MSE	MAE	SMAPE	MSE	MAE	SMAPE	MSE	MAE	SMAPE
ETTh1	TS_Adam	0.441±0.005	0.460±0.003	84.304 ± 0.035	0.447±0.003	0.439±0.001	80.274 ± 0.02	0.440±0.002	0.443±0.001	80.637 ± 0.004
	Adam	0.569±0.023	0.528±0.01	92.79 ± 0.031	0.457±0.002	0.445±0.001	81.448 ± 0.055	0.473±0.003	0.462±0.002	83.604 ± 0.079
	SGD	0.940±0.007	0.714±0.004	127.224 ± 0.062	0.899±0.014	0.644±0.005	104.293 ± 0.001	1.330±0.004	0.796±0.07	106.171 ± 0.054
	Yogi	0.457±0.002	0.476±0.001	85.541 ± 0.066	0.511±0.002	0.479±0.001	86.056 ± 0.02	0.652±0.003	0.540±0.001	95.296 ± 0.056
	AdamW	0.568±0.023	0.528±0.01	92.785 ± 0.076	0.457±0.002	0.445±0.001	81.43 ± 0.006	0.473±0.003	0.462±0.002	83.625 ± 0.006

Table 1: Performance comparison of different optimizers on long-term forecasting using the four subsets of the ETT dataset, the Weather dataset, and the ECL dataset. Results are averaged over four prediction lengths: {96, 192, 336, 720}. Each entry reports the mean and standard deviation across three runs. The best result in each row is marked in bold red, and the second-best is underlined in blue.

	ETTh1	ETTh2	ETTm1	ETTm2	ECL	Weather
Trend	59.3%	88.9%	59.1%	88.5%	20.9%	51.0%
Seasonal	32.1%	5.0%	31.9%	5.0%	73.3%	27.4%
Residual	15.5%	6.8%	15.5%	6.8%	11.4%	28.3%

Table 2: Variance contributions from STL decomposition of the ETT, ECL and Weather datasets. Note: contributions may not sum to one due to non-orthogonality.

Comp.	ETTh1	ETTh2	ETTm1	ETTm2	ECL	Weahter	Sig.Wins
vs Adam	0.0142/0.0129	0.025/0.0342	0.0013/0.0094	0.011/0.0752	0.0039/<0.0001	0.0008/0.0011	3/3
vs SGD	<0.0001/<0.0001	0.017/0.007*	<0.0001/<0.0001	0.0253/0.0095	0.0002/<0.0001	0.0051*/0.0096	4/4
vs Yogi	0.0109/0.0027*	<0.0001/<0.0001	0.0326/0.0416	<0.0001/<0.0001	0.0047/0.0051	<0.0001/<0.0001	4/5
vs AdamW	0.0141/0.0129	0.0189/0.0346	0.0013*/0.0093	0.011/0.0761	0.004/<0.0001	0.0008/0.0011	3/2
vs LA	0.0007/0.0005	0.001/0.0011	0.0174/0.0104	0.0025*/0.0194	0.0358/0.0332	0.0001/0.001	4/3

Table 3: Pairwise comparison p-values (MSE / MAE) between TS_Adam and baselines. Asterisks indicate significance in pairwise t-tests with Bonferroni correction (

p<0.05

). The "Sig.Wins" column reports the number of datasets (out of 6) where TS_Adam shows statistically superior performance over the corresponding baseline for MSE (before the slash) and MAE (after the slash), respectively. For example, "3/3" means TS_Adam significantly outperforms the baseline on 3 datasets according to MSE and on 3 datasets according to MAE. Note: LA = Lookahead.

Method	Model_1(MICN)			Model_2(PatchTST)			model_3(SegRNN)
Method	SMAPE	MASE	OWA	SMAPE	MASE	OWA	SMAPE	MASE	OWA
TS_Adam	15.451±0.017	2.528±0.004	1.231±0.002	12.813±0.028	1.780±0.009	0.937±0.003	13.960±0.003	1.967±0.001	1.029±0.001
Adam	16.524±0.006	3.167±0.001	1.437±0.001	13.943±0.071	2.092±0.027	1.061±0.01	14.021±0.003	1.997±0.001	1.039±0.001
SGD	15.165±0.008	2.637±0.005	1.248±0.001	14.911±0.062	2.322±0.027	1.157±0.009	14.208±0.001	2.044±0.001	1.058±0.001
Yogi	16.913±0.006	3.263±0.002	1.477±0.001	14.319±0.075	2.178±0.025	1.097±0.009	14.171±0.001	2.031±0.001	1.053±0.001
AdamW	16.524±0.006	3.166±0.002	1.437±0.001	13.946±0.072	2.094±0.027	1.062±0.01	14.023±0.002	1.997±0.001	1.039±0.001

Table 4: Performance comparison of different optimizers on short-term forecasting using the M4 dataset. Results are reported on six frequency-based subsets following the official evaluation protocol of the M4 competition. Each entry presents the mean and standard deviation of forecasting errors. The best result in each row is marked in bold red, and the second-best is underlined in blue.

Comparison Pair	SMAPE	MASE	OWA
vs Adam	0.0025	0.0059	0.0041
vs SGD	0.0938	0.012	0.0267
vs Yogi	0.0011	0.0034	0.0022
vs AdamW	0.0024	0.0059	0.004
vs Lookahead	0.0029	0.0247	0.0165

Table 5: Statistical significance of TS_Adam versus baselines on the M4 dataset for short-term forecasting. P-values are from pairwise t-tests comparing forecast accuracy across all series of the M4 dataset.

Figure 2: Test loss curves during training for various optimizers on ETTh1 (T=720) and M4-Hourly, using MICN, PatchTST, and SegRNN. Each curve shows the mean test loss, with shaded areas indicating one standard deviation. For optimizers that stop early, a flat horizontal line denotes the final value without uncertainty shading.

Figure 3: Sensitivity of TS_Adam to $\beta_{1}$ and learning rate on ETTh1 (a) and ETTh2 (b). Each cell shows the mean ± standard deviation of the absolute MSE difference (TS_Adam − Adam), averaged over four prediction lengths.

Optimizer	Batch	Dataset
		ETTh1		ETTh2
		MSE	MAE	MSE	MAE
Adam	16	0.536±0.028	0.518±0.015	0.616±0.038	0.541±0.019
	32	0.569±0.024	0.528±0.011	0.581±0.028	0.526±0.016
	64	0.545±0.011	0.516±0.006	0.586±0.019	0.529±0.01
TS_Adam	16	0.498±0.014	0.497±0.009	0.537±0.015	0.502±0.009

Table 6: Evaluating the Impact of Batch Size on Optimizer Performance.

optimizer	noise	Dataset
		ETTh1		ETTh2
		MSE	MAE	MSE	MAE
Adam	Gaussian	0.54±0.007	0.513±0.003	0.583±0.005	0.527±0.001
Adam	Outlier	0.514±0.002	0.499±6.7e-4	0.569±0.015	0.519±0.007
TS_Adam	Gaussian	0.438±0.002	0.455±0.001	0.543±0.003	0.507±0.002
TS_Adam	Outlier	0.436±8.9e-4	0.455±6.2e-4	0.554±0.002	0.514±0.001

Table 7: Performance comparison under data noise and outliers. Results are reported as mean ± standard deviation.

Optimizer	lr_strategy	Dataset
		ETTh1		ETTh2
		MSE	MAE	MSE	MAE
Adam	cosine	0.554±0.005	0.522±0.001	0.587±0.007	0.53±0.003
	type1	0.545±0.008	0.516±0.004	0.586±0.005	0.529±0.001
	tpye2	0.556±0.014	0.522±0.006	0.587±0.008	0.53±0.003
TS_Adam	cosine	0.443±0.004	0.46±0.002	0.53±0.004	0.5±0.002

Table 8: Performance under different learning rate scheduling strategies. type1 denotes exponential decay, type2 denotes delayed exponential decay. Results are reported as mean ± standard deviation.

Optimizer	wd	Dataset
		ETTh1		ETTh2
		MSE	MAE	MSE	MAE
Adam	0	0.569±0.023	0.528±0.01	0.581±0.027	0.526±0.014
	1.00E-04	0.569±0.068	0.528±0.01	0.581±0.027	0.526±0.014
	1.00E-02	0.568±0.023	0.528±0.01	0.581±0.027	0.526±0.014
TS_Adam	0	0.441±0.005	0.460±0.003	0.533±0.01	0.501±0.005

Table 9: Performance under different weight - decay (wd) coefficients. Results are reported as mean ± standard deviation.

Optimizer	Dataset
	ETTh1		ETTh2
	MSE	MAE	MSE	MAE
AdamW	0.568±0.023	0.528±0.01	0.581±0.027	0.526±0.014
AdamW†	0.441±0.006	0.460±0.004	0.533±0.012	0.501±0.006
Yogi	0.457±0.002	0.476±0.001	0.572±0.002	0.522±0.001
Yogi†	0.445±0.001	0.462±0.001	0.561±0.007	0.515±0.004

Table 10: Extending TS_Adam Strategy to Other Adaptive Optimizers. † denotes the use of the TS_Adam strategy, which removes second-order bias correction.

Figure 4: Cumulative regret curves on ETTh1 and ETTh2 using the MICN model with prediction length 192. Each curve shows the mean regret over three runs, and the shaded area denotes $\pm 5$ standard deviations to enhance visibility. (a) ETTh1; (b) ETTh2.

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 157 papers

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
May 5, 2026Sharan Sahu, Abir Sarkar, Cameron J. Hogan +1Stochastic Optimization Stochastic Optimization Convergence
Department of Statistics and Data Science Cornell University
Adapt Data to Model: Adaptive Transformation Optimization for Domain-shared Time Series Foundation Models
Feb 28, 2026Yunzhong Qiu, Zhiyao Cen, Zhongyi Pei +2Time Series Forecasting Time-Series Foundation Models
School of Software, BNRist, Tsinghua University, China
STEPS: A Temporal Smooth Error Propagation Solver on the Manifolds for Test-Time Adaptation in Time Series Forecasting
May 8, 2026Jiaqi Liu, Yifan Ouyang, Zhifei Song +2Source-Free Domain Adaptation Test-Time Adaptation
School of Artificial Intelligence and Robotics, Xiamen University Malaysia · S.M.A.R.T. NEXUS Centre of Excellence, Xiamen University Malaysia Sepang, Selangor 43900, Malaysia

May 5, 2026J/K move · Enter open · S save

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Sharan Sahu, Abir Sarkar, Cameron J. Hogan, Martin T. Wells

Department of Statistics and Data Science Cornell University

Stochastic Optimization Stochastic Optimization Convergence Adaptive Gradient Methods Stochastic Gradient Descent

Open paper AskPDF arXiv

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general

L

-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by

\beta_1

, and a preconditioner perturbation governed by

\beta_2

. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit

(\beta_1,\beta_2,\epsilon)

-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.

Abstract

Adaptive Gradient Methods

Explore similar work

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

Adapt Data to Model: Adaptive Transformation Optimization for Domain-shared Time Series Foundation Models

STEPS: A Temporal Smooth Error Propagation Solver on the Manifolds for Test-Time Adaptation in Time Series Forecasting