Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

Saurish Nagrath; Saroj Kumar Panigrahy; Soubhagya Sankar Barpanda

cs.LGJan 18, 2026

Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

Authors: Saurish Nagrath, Saroj Kumar Panigrahy, Soubhagya Sankar Barpanda

Organizations: School of Computer Science and Engineering, VIT-AP University, Amaravati, Andhra Pradesh, 522241, India

Abstract

Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and structure of input representations derived from raw multivariate time-series data, particularly as sequence length and data scale increase. This paper proposes a two-stage forecasting framework that explicitly separates local temporal representation learning from global dependency modelling. In the proposed approach, a convolutional neural network operates on fixed-length temporal patches to extract short-range temporal dynamics and non-linear feature interactions, producing compact patch-level token embeddings. Token-level self-attention is applied during representation learning to refine these embeddings, after which a Transformer encoder models inter-patch temporal dependencies to generate forecasts. The method is evaluated on a synthetic multivariate time-series dataset with controlled static and dynamic factors, using an extended sequence length and a larger number of samples. Experimental results demonstrate that the proposed framework consistently outperforms a convolutional baseline under increased temporal context and remains competitive with a strong patch-based Transformer model. These findings indicate that structured patch-level tokenization provides a scalable and effective representation for multivariate time-series forecasting, particularly when longer input sequences are considered.

Figures & tables

Fig. 1: Overview of the proposed CNN–Transformer architecture for multivariate time-series forecasting.

$t$	$d_{t}^{(1)}$	$d_{t}^{(2)}$	$s_{1}$	$s_{2}$	$s_{3}$	$s_{4}$
1	-5.36	0.36	2	16.43	3	3
2	-4.52	6.41	2	16.43	3	3
3	-2.87	3.73	2	16.43	3	3
4	-1.38	2.48	2	16.43	3	3
5	-2.02	6.28	2	16.43	3	3
6	-1.76	2.45	2	16.43	3	3

TABLE I: Illustrative example of a single synthetic multivariate time-series sample (first 6 time steps). Static features remain constant across time.

Parameter	Value
Number of input features	6
Sequence length ( $T$ )	160
Patch size ( $P$ )	8
Number of patches ( $\lfloor T/P\rfloor$ )	20
Forecasting granularity	Patch-level

TABLE II: Key hyperparameter settings used in all experiments.

Model	Architecture	Layers	Attention Heads	Learning Rate	Epochs	Batch Size
Proposed	CNN Patch Encoder	–	–	$1\times 10^{-3}$	2000	32
Proposed	Transformer Encoder	4	16	$1\times 10^{-3}$	300	32
TCN	Temporal Conv. Network	4	–	$1\times 10^{-3}$	300	32
PatchTST	Patch-based Transformer	4	16	$1\times 10^{-3}$	300	32

TABLE III: Hyperparameter settings for the proposed model and baseline methods.

Horizon	Model	MSE	MAE	RMSE
$h=1$ (8 steps)	Proposed CNN–Attention–Transformer	0.076000	0.205000	0.275681
	TCN	0.238000	0.355000	0.487852
	PatchTST	0.095000	0.218000	0.308221
$h=3$ (24 steps)	Proposed CNN–Attention–Transformer	0.058860	0.190912	0.242611
	TCN	0.105902	0.218461	0.325426
	PatchTST	0.101054	0.242811	0.317890

TABLE IV: Forecasting performance on the synthetic dataset. Lower values indicate better performance.

Model	MSE	MAE	RMSE
Proposed model	0.090665	0.142513	0.301105
TCN	0.145150	0.195749	0.380985
PatchTST	0.128451	0.187312	0.358400

TABLE V: Test-set performance on Electricity Load Diagrams 2011–2014.

Configuration	MSE	MAE	RMSE
Full model	0.097670	0.222652	0.312522
No dense connections	1.304764	0.781693	1.142263
Mean pooling (no attention pool)	2.029864	0.810744	1.424733
No token attention	0.546755	0.445993	0.739429

TABLE VI: Ablation results on the synthetic dataset. Lower values indicate better performance.

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 289 papers

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting
Mar 25, 2026Aymane Harkati, Moncef Garouani, Olivier Teste +2Robust Time Series Forecasting Time Series Forecasting
CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France
TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting
Mar 11, 2026Sravan Kumar Ankireddy, Nikita Seleznev, Nam H. Nguyen +4Time Series Forecasting Time Series Transformers
University of Texas at Austin, Austin, TX, USA · Capital One, USA · University of Maryland, College Park, MD, USA
Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured Partitioning
Aug 20, 2026Zhiquan Huang, Jiazhe Wang, Linjing Xue +3Multivariate Time Series Forecasting Time Series Forecasting
School of Information Engineering, Henan University of Science and Technology, Luoyang, 471023, Henan, China · School of Software, Henan University of Science and Technology, Luoyang, 471000, Henan, China

Mar 25, 2026J/K move · Enter open · S save

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Aymane Harkati, Moncef Garouani, Olivier Teste, Julien Aligon+1

CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Robust Time Series Forecasting Time Series Forecasting Multivariate Time Series Forecasting Time Series Transformers Multiscale Time-Series Modeling

Open paper AskPDF arXiv

Accurate forecasting of multivariate time series remains challenging due to the need to capture both short-term fluctuations and long-range temporal dependencies. Transformer-based models have emerged as a powerful approach, but their performance depends critically on the representation of temporal data. Traditional point-wise representations preserve individual time-step information, enabling fine-grained modeling, yet they tend to be computationally expensive and less effective at modeling broader contextual dependencies, limiting their scalability to long sequences. Patch-wise representations aggregate consecutive steps into compact tokens to improve efficiency and model local temporal dynamics, but they often discard fine-grained temporal details that are critical for accurate predictions in volatile or complex time series. We propose IPatch, a multi-resolution Transformer architecture that integrates both point-wise and patch-wise tokens, modeling temporal information at multiple resolutions. Experiments on 7 benchmark datasets demonstrate that IPatch consistently improves forecasting accuracy, robustness to noise, and generalization across various prediction horizons compared to single-representation baselines.

Abstract

Multivariate Time Series Forecasting

Explore similar work

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting

Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured Partitioning