A Lightweight Plug-in Gate for Transformer-Based Time-Series Forecasters

Hongkai Zhuang; Tao Huang; Chen Hou

cs.LGSep 17, 2026

A Lightweight Plug-in Gate for Transformer-Based Time-Series Forecasters

Authors: Hongkai Zhuang, Tao Huang, Chen Hou

Organizations: School of Computer and Big Data, Minjiang University, No. 200 Xiyuangong Road, Fuzhou, 350108, Fujian, China.

Abstract

Covariate-rich time-series forecasting requires deciding how external variables enter the target forecasting path. Existing Transformer-based forecasters usually build a covariate representation and pass it to the encoder without an explicit admission stage. This paper studies pre-encoder covariate admission as an input-side interface that regulates that representation immediately before encoder processing. We implement the interface with a lightweight representation-level pre-encoder gate that assigns sigmoid scores to representation units, and we also study a usage-regularized variant that penalizes average admission. The interface is evaluated as a plug-in module for TimeXer, Inverted Transformer (iTransformer), and Patch Time Series Transformer (PatchTST) under a zero-extra-tuning protocol, where each gated model inherits the corresponding baseline configuration. Experiments on the Electricity Transformer Temperature minute-level (ETTm1 and ETTm2) datasets, Traffic, Energy, and influenza-like illness (ILI) include paired forecasting comparisons, gate-placement ablation, initialization ablation, controlled covariate-admission analysis, and a variance inflation factor (VIF)-informed permutation feature importance (PFI) diagnostic case study. In the tested settings, the gate is competitive with the corresponding baselines, and the usage penalty reduces average admission scores while keeping forecasting errors close to the unpenalized TimeXer setting.

Figures & tables

Figure 1: Architecture of the pre-encoder covariate admission gate

Dataset	Method	Metric	Horizon
Dataset	Method	Metric	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ETTm1	TimeXer-Gate	MSE	0.028	0.043	0.056	0.079	0.052
		MAE	0.125	0.158	0.183	0.217	0.171
	TimeXer	MSE	0.028	0.043	0.057	0.079	0.052
		MAE	0.125	0.158	0.185	0.217	0.171
	iTransformer-Gate	MSE	0.028	0.044	0.058	0.080	0.053

Table 1: Long-term forecasting performance

Dataset	Method	Metric	Horizon
Dataset	Method	Metric	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ILI	TimeXer-Gate	MSE	0.674	0.689	0.718	0.748	0.707
		MAE	0.613	0.654	0.688	0.716	0.668
	TimeXer	MSE	0.686	0.702	0.717	0.754	0.715
		MAE	0.609	0.653	0.681	0.718	0.665
	iTransformer-Gate	MSE	0.673	0.710	0.700	0.729	0.703

Table 1: Long-term forecasting performance (continued)

Dataset	NoGate		PreEncoderGate		PostEncoderGate		FrozenGate
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTm1	0.052	0.171	0.052	0.171	0.052	0.172	0.052	0.171
ETTm2	0.120	0.258	0.119	0.255	0.120	0.258	0.119	0.256
Traffic	0.156	0.235	0.153	0.233	0.168	0.255	0.160	0.242
ILI	0.726	0.670	0.719	0.673	0.770	0.714	0.719	0.673
Energy	0.259	0.361	0.258	0.361	0.259	0.362	0.258	0.362

Table 2: TimeXer ablation on gate placement and learnability

Dataset	NoGate		PreEncoderGate		$p_{0}=0.500$		$p_{0}=0.700$		$p_{0}=0.900$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTm1	0.052	0.171	0.052	0.171	0.052	0.171	0.052	0.171	0.052	0.171
ETTm2	0.120	0.258	0.119	0.255	0.119	0.256	0.120	0.257	0.120	0.257
Energy	0.259	0.361	0.258	0.361	0.259	0.361	0.257	0.361	0.258	0.363
ILI	0.726	0.670	0.719	0.673	0.722	0.672	0.725	0.672	0.727	0.672
Overall	0.289	0.365	0.287	0.365	0.288	0.365	0.288	0.365	0.289	0.366

Table 3: TimeXer ablation on gate initialization

Dataset	Variant	Admission setting	Average admission score	MSE	MAE
ETTm1	FullCov	all covariates	1.000	0.052	0.171
	TargetOnly	no physical covariates	0.000	0.053	0.172
	RandomK	random subset ( $k=5$ )	0.833	0.052	0.172
	VIFLowK	low-VIF subset ( $k=5$ )	0.833	0.052	0.172
	UsageGate	$\lambda=0$	0.427	0.053	0.173
	UsageGate	penalty ( $\lambda=10^{-1}$ )	0.003	0.053	0.172

Table 4: Controlled covariate admission analysis with TimeXer

Figure 2: Learned gate weight versus VIF on ETTm2 . The plot summarizes the relation between linear covariate redundancy and average gate weight in the analyzed setting.

Covariate/Group	VIF	Gate Weight	Model	PFI
Covariate/Group	VIF	Gate Weight	Model	MSE	MAE	DTW
HUFL	2.637	0.655	TimeXer-Gate	0.000	0.000	0.012
HUFL	2.637	–	TimeXer	0.001	0.001	0.023
HULL	16.956	0.448	TimeXer-Gate	0.000	0.000	0.004
HULL	16.956	–	TimeXer	0.000	0.000	0.004
MUFL	2.038	0.665	TimeXer-Gate	0.001	0.001	0.026

Table 5: PFI perturbation analysis on ETTm2

Dataset	Method	Metric	Horizon
Dataset	Method	Metric	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ETTm1	TimeXer-Gate	MSE	0.028	0.043	0.056	0.079	0.052
		MAE	0.125	0.158	0.183	0.216	0.171
	TimeXer	MSE	0.028	0.043	0.058	0.079	0.052
		MAE	0.125	0.158	0.185	0.217	0.171
	iTransformer	MSE	0.029	0.045	0.060	0.079	0.053

Table 6: TimeXer-Gate comparison with forecasting baselines

Dataset	Method	Metric	Horizon
Dataset	Method	Metric	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
Traffic	TimeXer-Gate	MSE	0.149	0.150	0.148	0.165	0.153
		MAE	0.224	0.229	0.230	0.247	0.232
	TimeXer	MSE	0.151	0.152	0.150	0.172	0.156
		MAE	0.224	0.229	0.232	0.253	0.235
	iTransformer	MSE	0.156	0.156	0.154	0.177	0.161

Table 6: TimeXer-Gate comparison with forecasting baselines (continued)

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ETTm1	$k=1$	0.029/0.127	0.045/0.161	0.059/0.186	0.081/0.218	0.053/0.173
	$k=2$	0.029/0.126	0.044/0.161	0.058/0.186	0.080/0.217	0.053/0.173
	$k=3$	0.028/0.125	0.044/0.160	0.058/0.186	0.080/0.217	0.052/0.172
	$k=4$	0.028/0.125	0.044/0.159	0.058/0.185	0.080/0.217	0.052/0.172
	$k=5$	0.028/0.125	0.043/0.158	0.058/0.185	0.079/0.217	0.052/0.172
ETTm2	$k=1$	0.069/0.189	0.107/0.242	0.135/0.278	0.183/0.333	0.123/0.260

Appendix

Table 7: Random-subset grid results

Dataset	Setting	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ETTm1	$k=1$	0.029/0.127	0.045/0.161	0.059/0.186	0.081/0.218	0.053/0.173
	$k=2$	0.029/0.127	0.044/0.160	0.059/0.187	0.081/0.218	0.053/0.173
	$k=3$	0.029/0.127	0.044/0.160	0.058/0.187	0.080/0.218	0.053/0.173
	$k=4$	0.029/0.127	0.044/0.160	0.058/0.186	0.080/0.218	0.053/0.173
	$k=5$	0.028/0.126	0.044/0.159	0.058/0.186	0.079/0.217	0.052/0.172
ETTm2	$k=1$	0.067/0.186	0.105/0.239	0.134/0.278	0.182/0.331	0.122/0.259

Appendix

Table 8: Low-VIF subset grid results

Dataset	Setting	$H_{1}$	$H_{2}$	$H_{3}$	$H_{4}$	AVG
ETTm1	$\lambda=0$	0.029/0.126	0.045/0.161	0.058/0.187	0.080/0.218	0.053/0.173
	$\lambda=10^{-3}$	0.028/0.126	0.045/0.161	0.060/0.189	0.080/0.218	0.053/0.173
	$\lambda=10^{-2}$	0.028/0.125	0.044/0.160	0.059/0.186	0.080/0.217	0.053/0.172
	$\lambda=10^{-1}$	0.028/0.125	0.043/0.159	0.058/0.186	0.080/0.217	0.053/0.172
ETTm2	$\lambda=0$	0.066/0.185	0.104/0.239	0.134/0.278	0.183/0.333	0.122/0.259
	$\lambda=10^{-3}$	0.067/0.186	0.104/0.238	0.133/0.276	0.181/0.331	0.121/0.258

Appendix

Table 9: UsageGate penalty grid results

Explore similar work

Hopformer: Homogeneity-Pursuit Transformer for Time Series Forecasting
Jul 24, 2026Wan Zhang, Qinjie Lin, Chan Lee +3Multivariate Time Series Forecasting Time Series Forecasting
The AMSS Center of Forecasting Science, Chinese Academy of Sciences, Beijing, China · Department of Computer Science, Northwestern University, Evanston, IL · Department of Statistics and Data Science, Northwestern University, Evanston, IL +1
Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
May 6, 2026Alper YıldırımTransformer Interpretability Time Series Forecasting
Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting
Sep 14, 2026Tamanna Kumavat, Georg Brunner, Kyriakos FlourisTime Series Forecasting LLM-Based Time Series Forecasting
University of Zurich, Switzerland · Computer Vision Laboratory, ETH Zurich, Switzerland · MRC Biostatistics Unit, University of Cambridge, United Kingdom

Jul 24, 2026, stat.MLJ/K move · Enter open · S save

Hopformer: Homogeneity-Pursuit Transformer for Time Series Forecasting

Wan Zhang, Qinjie Lin, Chan Lee, Weijian Li+2

The AMSS Center of Forecasting Science, Chinese Academy of Sciences, Beijing, China · Department of Computer Science, Northwestern University, Evanston, IL · Department of Statistics and Data Science, Northwestern University, Evanston, IL+1

Multivariate Time Series Forecasting Time Series Forecasting Time Series Transformers

Open paper AskPDF arXiv

Forecasting multiple time-series with high-dimensional covariates presents a core challenge: unifying common temporal patterns while retaining meaningful series-specific information. We introduce Hopformer (Homogeneity-Pursuit Transformer), a two-stage framework that addresses this challenge. In the first stage, we perform a Sparsity Pattern Aggregation (SPA) scheme extracting a common low-variance trend that incorporates the covariates. This acts as a homogenization layer. In the second stage, a LoRA-fine-tuned Transformer models the remaining complex dependencies in the residual. Our method is theoretically grounded. We prove that SPA achieves a near-optimal bias-variance trade-off via an oracle inequality. We also provide generalization bounds for the second stage under dependent time series data. Hopformer sets a new state of the art, improving MASE by an average of 6.56% across synthetic and real-world forecasting benchmarks.

Abstract

Explore similar work

Hopformer: Homogeneity-Pursuit Transformer for Time Series Forecasting

Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting

Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting