RankFormer: Rethinking Time Series Transformers through Representation Capacity Analysis and Progressive Decoding

Peiwang Tang; Cheng HE; Xijie Liang; Defu Lian

cs.LGOct 27, 2026

RankFormer: Rethinking Time Series Transformers through Representation Capacity Analysis and Progressive Decoding

Authors: Peiwang Tang, Cheng HE, Xijie Liang, Defu Lian

Organizations: University of Science and Technology of China, Hefei, Anhui, China · State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China · Shanghai Black Wing Asset Management, Shanghai, China

Abstract

Multivariate time series forecasting plays a critical role in Web and cyber-physical systems, where large-scale services and IoT infrastructures generate complex temporal data. Transformer-based models have achieved strong empirical performance in this domain, yet the principles governing the expressive capacity of their learned representations remain insufficiently understood. Moreover, many existing models rely on simplified decoding strategies, including channel-independent modeling and direct projection heads, which limit their ability to capture complex multivariate dynamics. To address these challenges, we propose RankFormer, a novel encoder-decoder architecture for multivariate time series forecasting. We establish a rank-based analysis framework to investigate how tokenization, embedding, and attention designs affect the representation capacity of time series Transformers. Our analysis reveals that patchification increases the potential representation capacity by aggregating multiple temporal observations into each token, while subsequent encoding operations determine how much of this capacity is realized. Guided by these insights, we develop a rank-enhancing encoder with Hourglass Attention to improve rank stability during information interaction. Furthermore, we introduce the Interactive Progressive Decoder (IPD), which improves future trajectory modeling through global-guided cross-channel interaction and progressive refinement. Extensive experiments on real-world long-term forecasting benchmarks demonstrate that RankFormer achieves competitive performance against state-of-the-art methods. Code repository: https://github.com/TangPeiwang/Rankformer.

Figures & tables

Panel of Figure 1 — Figure 1. Empirical validation of rank stability and Hourglass Attention (HgA). (a) Rapid singular value decay shows bidirectional attention induces deep-layer rank collapse, while causal masking preserves representational diversity. (b) Causal masking restricts early tokens (starvation). In contrast, HgA uses a symmetric prefix as a global hub, enabling efficient context retrieval while maintaining rank stability.

Figure 2. Overall structure of RankFormer. The model preprocesses multivariate time series via Instance Norm and CI, utilizing Patch Embedding and a Multi-Head Hourglass Attention encoder to generate a base prediction. An Interactive Progressive Decoder then refines these features. It trains via one-step residual learning using randomized partial ground truth, while inference employs a $T$ -step progressive refinement from a zero-residual state, concluding with Instance Denorm.

Figure 3. Internal architecture of the Interactive Progressive Decoder.

Models	RankFormer		SEER		SRSNet		DUET		xPatch		Amplifier		FredFormer		iTransformer		PatchTST		DLinear
Models	(ours)		(2026)		(2025)		(2025)		(2025)		(2025)		(2024)		(2024)		(2023)		(2023)
Metrics	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.410	0.421	0.423	0.424	0.442	0.433	0.443	0.436	0.444	0.429	0.449	0.437	0.448	0.435	0.454	0.448	0.469	0.455	0.461	0.458
ETTh2	0.354	0.381	0.356	0.384	0.376	0.402	0.372	0.398	0.369	0.392	0.389	0.411	0.378	0.403	0.383	0.407	0.387	0.407	0.563	0.519
ETTm1	0.367	0.377	0.377	0.381	0.385	0.395	0.390	0.393	0.387	0.383	0.383	0.396	0.385	0.398	0.407	0.410	0.387	0.400	0.404	0.408
ETTm2	0.268	0.311	0.276	0.317	0.284	0.329	0.280	0.324	0.280	0.319	0.280	0.326	0.281	0.324	0.288	0.332	0.281	0.326	0.354	0.402

Table 1. Results of the Long-Term Time Series Forecasting. All baseline models were configured consistently with SEER. The reported results represent the average performance over four distinct forecasting horizons: {96, 192, 336, 720}.

Figure 4. Predictive benefits of additional embedding rank capacity. Lines show three-seed means, small dots show individual seeds, and dashed curves show the MSE lower bound for rank-at-most-five predictions.

Dataset	Unconstrained		Rank $\leq 1$		Rank $\leq 4$		Rank $\leq 8$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.503	0.469	0.549	0.511	0.509	0.473	0.504	0.472
ETTh2	0.380	0.414	0.385	0.417	0.398	0.421	0.383	0.414
ETTm1	0.369	0.390	0.385	0.402	0.372	0.393	0.372	0.392
ETTm2	0.271	0.326	0.273	0.327	0.274	0.327	0.270	0.324
Weather	0.228	0.263	0.231	0.266	0.231	0.266	0.229	0.265

Table 2. Forecasting performance under different rank constraints on position-enriched embeddings. The best results are in bold.

Figure 5. Effect of Network Depth on Forecasting with Different Pure Attention Mechanisms.

Method	Metric	ETTh1	ETTh2	ETTm1	Weather	Exchange
BiA	MSE	0.710	0.418	0.648	0.279	0.375
BiA	MAE	0.583	0.433	0.530	0.303	0.414
HgA	MSE	0.510	0.407	0.421	0.273	0.371
HgA	MAE	0.480	0.428	0.422	0.293	0.412

Table 3. Performance comparison of six-layer pure-attention models. The best results are in bold.

Dataset	Unconstrained		$\mathrm{rank}(A)\leq 1$		$\mathrm{rank}(A)\leq 3$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.510	0.481	+12.44	+7.36	+0.23	+0.04
ETTh2	0.406	0.427	+0.93	-0.55	+1.88	+0.72
ETTm1	0.422	0.421	+37.72	+19.04	+7.27	+3.89
ETTm2	0.293	0.338	+6.92	+4.07	+3.60	+2.02
Weather	0.273	0.293	+6.83	+6.70	+4.47	+3.93

Table 4. Training-Time Attention Rank Constraints in Six-Layer Pure HgA. For the rank-constrained columns, values are percentage changes relative to the Unconstrained model.

Figure 6. Forecasting Comparison of HgA and Causal Attention under Different Token Orders.

Figure 7. Rank distributions of embedding matrices with patch sizes

P=1

(ETTm2) and

P=16

(Weather).

Figure 8. Rank distributions of attention matrices on Weather and ETTm2.

Figure 9. Hyperparameter sensitivity of refinement steps (

T

Figure 10. Reductions in level, trend, and remaining-shape errors from

T=1

T=5

\tau=96

Figure 11. Online inference latency of iterative refinement and offline-folded inference.

Dataset	Linear Head		Refinement Only		Full IPD
Dataset	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.426	0.430	0.412	0.421	0.405	0.416
ETTh2	0.349	0.385	0.344	0.384	0.342	0.384
ETTm1	0.349	0.368	0.342	0.367	0.339	0.367
ETTm2	0.254	0.307	0.253	0.308	0.252	0.307
Weather	0.228	0.254	0.230	0.257	0.226	0.252

Table 5. Comparison of IPD component configurations.

Figure 12. Cross-channel conditioning analysis on Weather.

Figure 13. Rank expansion via patchification.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14. Rank Distribution of Embedding Matrices (Patch Size = 16).

Panel of Figure 15 — Figure 15. PCA visualization of token representation evolution across deep layers in a pure attention network. The color gradient indicates the token index within the sequence.

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 302 papers

Sep 17, 2026

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

Long-term multivariate time series plays a significant role in many application areas such as power systems, trading, etc. However, their accurate prediction is quite difficult for conventional forecasting methods as they often exhibit high dimensionality and complex relationships. Recent works show that transformer-based approaches are quite effective for long-term forecasting thanks to their attention mechanism. However, in the presence of complex high-dimensional inputs, they show evidence of oversmoothing, limited capacity, and opacity. To this end, this paper introduces SETTer, a transformer-based model that addresses these challenges by incorporating novel techniques for decoupled self-attention and hybrid masking. The proposed techniques enable SETTer to effectively capture the dominant short- and long-term patterns across the temporal and channel dimensions. In addition, we enrich the model layers with simple explainable structures that indicate the discriminative pattern of SETTer. We show that with a single-layer transformer architecture, SETTer can effectively model long-term dependencies in the presence of varying data complexities. Extensive experiments on real-word benchmark datasets for long-term multivariate time series forecasting demonstrate that SETTer outperforms state-of-the-art models in 88% of the scenarios.

Abraham Ezema, Chijioke Eze, Ferdinanda Ponci +1

Institute for Automation of Complex Power Systems, RWTH Aachen University, 52064 Aachen, Germany · RWTH Aachen University, 52064 Aachen, Germany · institute for Monitoring and Distributed Control for Power Systems Teaching and Research Area, RWTH Aachen University, 52064 Aachen, Germany +1

Multivariate Time Series Forecasting Long-Term Time Series Forecasting Explainable Time Series Forecasting Time Series Forecasting Attention Mechanisms Time Series Transformers

Ask PDF

Jan 18, 2026

Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and structure of input representations derived from raw multivariate time-series data, particularly as sequence length and data scale increase. This paper proposes a two-stage forecasting framework that explicitly separates local temporal representation learning from global dependency modelling. In the proposed approach, a convolutional neural network operates on fixed-length temporal patches to extract short-range temporal dynamics and non-linear feature interactions, producing compact patch-level token embeddings. Token-level self-attention is applied during representation learning to refine these embeddings, after which a Transformer encoder models inter-patch temporal dependencies to generate forecasts. The method is evaluated on a synthetic multivariate time-series dataset with controlled static and dynamic factors, using an extended sequence length and a larger number of samples. Experimental results demonstrate that the proposed framework consistently outperforms a convolutional baseline under increased temporal context and remains competitive with a strong patch-based Transformer model. These findings indicate that structured patch-level tokenization provides a scalable and effective representation for multivariate time-series forecasting, particularly when longer input sequences are considered.

Saurish Nagrath, Saroj Kumar Panigrahy, Soubhagya Sankar Barpanda

School of Computer Science and Engineering, VIT-AP University, Amaravati, Andhra Pradesh, 522241, India

Multivariate Time Series Forecasting Time Series Transformers Time Series Tokenization Time Series Representation Learning

Ask PDF

Mar 25, 2026

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Accurate forecasting of multivariate time series remains challenging due to the need to capture both short-term fluctuations and long-range temporal dependencies. Transformer-based models have emerged as a powerful approach, but their performance depends critically on the representation of temporal data. Traditional point-wise representations preserve individual time-step information, enabling fine-grained modeling, yet they tend to be computationally expensive and less effective at modeling broader contextual dependencies, limiting their scalability to long sequences. Patch-wise representations aggregate consecutive steps into compact tokens to improve efficiency and model local temporal dynamics, but they often discard fine-grained temporal details that are critical for accurate predictions in volatile or complex time series. We propose IPatch, a multi-resolution Transformer architecture that integrates both point-wise and patch-wise tokens, modeling temporal information at multiple resolutions. Experiments on 7 benchmark datasets demonstrate that IPatch consistently improves forecasting accuracy, robustness to noise, and generalization across various prediction horizons compared to single-representation baselines.

Aymane Harkati, Moncef Garouani, Olivier Teste +2

CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Robust Time Series Forecasting Time Series Forecasting Multivariate Time Series Forecasting Time Series Transformers Multiscale Time-Series Modeling

Ask PDF

Abstract

Multivariate Time Series Forecasting

Explore similar work

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting