RankFormer: Rethinking Time Series Transformers through Representation Capacity Analysis and Progressive Decoding

Peiwang Tang; Cheng HE; Xijie Liang; Defu Lian

cs.LGOct 27, 2026

RankFormer: Rethinking Time Series Transformers through Representation Capacity Analysis and Progressive Decoding

Authors: Peiwang Tang, Cheng HE, Xijie Liang, Defu Lian

Organizations: University of Science and Technology of China, Hefei, Anhui, China · State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China · Shanghai Black Wing Asset Management, Shanghai, China

Abstract

Multivariate time series forecasting plays a critical role in Web and cyber-physical systems, where large-scale services and IoT infrastructures generate complex temporal data. Transformer-based models have achieved strong empirical performance in this domain, yet the principles governing the expressive capacity of their learned representations remain insufficiently understood. Moreover, many existing models rely on simplified decoding strategies, including channel-independent modeling and direct projection heads, which limit their ability to capture complex multivariate dynamics. To address these challenges, we propose RankFormer, a novel encoder-decoder architecture for multivariate time series forecasting. We establish a rank-based analysis framework to investigate how tokenization, embedding, and attention designs affect the representation capacity of time series Transformers. Our analysis reveals that patchification increases the potential representation capacity by aggregating multiple temporal observations into each token, while subsequent encoding operations determine how much of this capacity is realized. Guided by these insights, we develop a rank-enhancing encoder with Hourglass Attention to improve rank stability during information interaction. Furthermore, we introduce the Interactive Progressive Decoder (IPD), which improves future trajectory modeling through global-guided cross-channel interaction and progressive refinement. Extensive experiments on real-world long-term forecasting benchmarks demonstrate that RankFormer achieves competitive performance against state-of-the-art methods. Code repository: https://github.com/TangPeiwang/Rankformer.

Figures & tables

Panel of Figure 1 — Figure 1. Empirical validation of rank stability and Hourglass Attention (HgA). (a) Rapid singular value decay shows bidirectional attention induces deep-layer rank collapse, while causal masking preserves representational diversity. (b) Causal masking restricts early tokens (starvation). In contrast, HgA uses a symmetric prefix as a global hub, enabling efficient context retrieval while maintaining rank stability.

Figure 2. Overall structure of RankFormer. The model preprocesses multivariate time series via Instance Norm and CI, utilizing Patch Embedding and a Multi-Head Hourglass Attention encoder to generate a base prediction. An Interactive Progressive Decoder then refines these features. It trains via one-step residual learning using randomized partial ground truth, while inference employs a $T$ -step progressive refinement from a zero-residual state, concluding with Instance Denorm.

Figure 3. Internal architecture of the Interactive Progressive Decoder.

Models	RankFormer		SEER		SRSNet		DUET		xPatch		Amplifier		FredFormer		iTransformer		PatchTST		DLinear
Models	(ours)		(2026)		(2025)		(2025)		(2025)		(2025)		(2024)		(2024)		(2023)		(2023)
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.410	0.421	0.423	0.424	0.442	0.433	0.443	0.436	0.444	0.429	0.449	0.437	0.448	0.435	0.454	0.448	0.469	0.455	0.461	0.458
ETTh2	0.354	0.381	0.356	0.384	0.376	0.402	0.372	0.398	0.369	0.392	0.389	0.411	0.378	0.403	0.383	0.407	0.387	0.407	0.563	0.519
ETTm1	0.367	0.377	0.377	0.381	0.385	0.395	0.390	0.393	0.387	0.383	0.383	0.396	0.385	0.398	0.407	0.410	0.387	0.400	0.404	0.408
ETTm2	0.268	0.311	0.276	0.317	0.284	0.329	0.280	0.324	0.280	0.319	0.280	0.326	0.281	0.324	0.288	0.332	0.281	0.326	0.354	0.402

Table 1. Results of the Long-Term Time Series Forecasting. All baseline models were configured consistently with SEER. The reported results represent the average performance over four distinct forecasting horizons: {96, 192, 336, 720}.

Figure 4. Predictive benefits of additional embedding rank capacity. Lines show three-seed means, small dots show individual seeds, and dashed curves show the MSE lower bound for rank-at-most-five predictions.

Dataset	Unconstrained		Rank $\leq 1$		Rank $\leq 4$		Rank $\leq 8$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.503	0.469	0.549	0.511	0.509	0.473	0.504	0.472
ETTh2	0.380	0.414	0.385	0.417	0.398	0.421	0.383	0.414
ETTm1	0.369	0.390	0.385	0.402	0.372	0.393	0.372	0.392
ETTm2	0.271	0.326	0.273	0.327	0.274	0.327	0.270	0.324
Weather	0.228	0.263	0.231	0.266	0.231	0.266	0.229	0.265

Table 2. Forecasting performance under different rank constraints on position-enriched embeddings. The best results are in bold.

Figure 5. Effect of Network Depth on Forecasting with Different Pure Attention Mechanisms.

Method	Metric	ETTh1	ETTh2	ETTm1	Weather	Exchange
BiA	MSE	0.710	0.418	0.648	0.279	0.375
BiA	MAE	0.583	0.433	0.530	0.303	0.414
HgA	MSE	0.510	0.407	0.421	0.273	0.371
HgA	MAE	0.480	0.428	0.422	0.293	0.412

Table 3. Performance comparison of six-layer pure-attention models. The best results are in bold.

Dataset	Unconstrained		$\mathrm{rank}(A)\leq 1$		$\mathrm{rank}(A)\leq 3$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.510	0.481	+12.44	+7.36	+0.23	+0.04
ETTh2	0.406	0.427	+0.93	-0.55	+1.88	+0.72
ETTm1	0.422	0.421	+37.72	+19.04	+7.27	+3.89
ETTm2	0.293	0.338	+6.92	+4.07	+3.60	+2.02
Weather	0.273	0.293	+6.83	+6.70	+4.47	+3.93

Table 4. Training-Time Attention Rank Constraints in Six-Layer Pure HgA. For the rank-constrained columns, values are percentage changes relative to the Unconstrained model.

Figure 6. Forecasting Comparison of HgA and Causal Attention under Different Token Orders.

Figure 7. Rank distributions of embedding matrices with patch sizes

P=1

(ETTm2) and

P=16

(Weather).

Figure 8. Rank distributions of attention matrices on Weather and ETTm2.

Figure 9. Hyperparameter sensitivity of refinement steps (

T

Figure 10. Reductions in level, trend, and remaining-shape errors from

T=1

T=5

\tau=96

Figure 11. Online inference latency of iterative refinement and offline-folded inference.

Dataset	Linear Head		Refinement Only		Full IPD
Dataset	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.426	0.430	0.412	0.421	0.405	0.416
ETTh2	0.349	0.385	0.344	0.384	0.342	0.384
ETTm1	0.349	0.368	0.342	0.367	0.339	0.367
ETTm2	0.254	0.307	0.253	0.308	0.252	0.307
Weather	0.228	0.254	0.230	0.257	0.226	0.252

Table 5. Comparison of IPD component configurations.

Figure 12. Cross-channel conditioning analysis on Weather.

Figure 13. Rank expansion via patchification.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14. Rank Distribution of Embedding Matrices (Patch Size = 16).

Panel of Figure 15 — Figure 15. PCA visualization of token representation evolution across deep layers in a pure attention network. The color gradient indicates the token index within the sequence.

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 302 papers

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting
Sep 17, 2026Abraham Ezema, Chijioke Eze, Ferdinanda Ponci +1Multivariate Time Series Forecasting Long-Term Time Series Forecasting
Institute for Automation of Complex Power Systems, RWTH Aachen University, 52064 Aachen, Germany · RWTH Aachen University, 52064 Aachen, Germany · institute for Monitoring and Distributed Control for Power Systems Teaching and Research Area, RWTH Aachen University, 52064 Aachen, Germany +1
Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting
Jan 18, 2026Saurish Nagrath, Saroj Kumar Panigrahy, Soubhagya Sankar BarpandaMultivariate Time Series Forecasting Time Series Transformers
School of Computer Science and Engineering, VIT-AP University, Amaravati, Andhra Pradesh, 522241, India
IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting
Mar 25, 2026Aymane Harkati, Moncef Garouani, Olivier Teste +2Robust Time Series Forecasting Time Series Forecasting
CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Sep 17, 2026J/K move · Enter open · S save

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

Abraham Ezema, Chijioke Eze, Ferdinanda Ponci, Antonello Monti

Institute for Automation of Complex Power Systems, RWTH Aachen University, 52064 Aachen, Germany · RWTH Aachen University, 52064 Aachen, Germany · institute for Monitoring and Distributed Control for Power Systems Teaching and Research Area, RWTH Aachen University, 52064 Aachen, Germany+1

Multivariate Time Series Forecasting Long-Term Time Series Forecasting Explainable Time Series Forecasting Time Series Forecasting Attention Mechanisms Time Series Transformers

Open paper AskPDF arXiv

Long-term multivariate time series plays a significant role in many application areas such as power systems, trading, etc. However, their accurate prediction is quite difficult for conventional forecasting methods as they often exhibit high dimensionality and complex relationships. Recent works show that transformer-based approaches are quite effective for long-term forecasting thanks to their attention mechanism. However, in the presence of complex high-dimensional inputs, they show evidence of oversmoothing, limited capacity, and opacity. To this end, this paper introduces SETTer, a transformer-based model that addresses these challenges by incorporating novel techniques for decoupled self-attention and hybrid masking. The proposed techniques enable SETTer to effectively capture the dominant short- and long-term patterns across the temporal and channel dimensions. In addition, we enrich the model layers with simple explainable structures that indicate the discriminative pattern of SETTer. We show that with a single-layer transformer architecture, SETTer can effectively model long-term dependencies in the presence of varying data complexities. Extensive experiments on real-word benchmark datasets for long-term multivariate time series forecasting demonstrate that SETTer outperforms state-of-the-art models in 88% of the scenarios.

Abstract

Multivariate Time Series Forecasting

Explore similar work

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting