cs.LGJan 18, 2026

Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

Authors: Saurish Nagrath, Saroj Kumar Panigrahy, Soubhagya Sankar Barpanda

Organizations: School of Computer Science and Engineering, VIT-AP University, Amaravati, Andhra Pradesh, 522241, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and structure of input representations derived from raw multivariate time-series data, particularly as sequence length and data scale increase. This paper proposes a two-stage forecasting framework that explicitly separates local temporal representation learning from global dependency modelling. In the proposed approach, a convolutional neural network operates on fixed-length temporal patches to extract short-range temporal dynamics and non-linear feature interactions, producing compact patch-level token embeddings. Token-level self-attention is applied during representation learning to refine these embeddings, after which a Transformer encoder models inter-patch temporal dependencies to generate forecasts. The method is evaluated on a synthetic multivariate time-series dataset with controlled static and dynamic factors, using an extended sequence length and a larger number of samples. Experimental results demonstrate that the proposed framework consistently outperforms a convolutional baseline under increased temporal context and remains competitive with a strong patch-based Transformer model. These findings indicate that structured patch-level tokenization provides a scalable and effective representation for multivariate time-series forecasting, particularly when longer input sequences are considered.

Figures & tables

Fig. 1: Overview of the proposed CNN–Transformer architecture for multivariate time-series forecasting.
ttdt(1)d_{t}^{(1)}dt(2)d_{t}^{(2)}s1s_{1}s2s_{2}s3s_{3}s4s_{4}
1-5.360.36216.4333
2-4.526.41216.4333
3-2.873.73216.4333
4-1.382.48216.4333
5-2.026.28216.4333
6-1.762.45216.4333
TABLE I: Illustrative example of a single synthetic multivariate time-series sample (first 6 time steps). Static features remain constant across time.
ParameterValue
Number of input features6
Sequence length ( TT )160
Patch size ( PP )8
Number of patches ( ⌊T/P⌋\lfloor T/P\rfloor )20
Forecasting granularityPatch-level
TABLE II: Key hyperparameter settings used in all experiments.
ModelArchitectureLayersAttention HeadsLearning RateEpochsBatch Size
ProposedCNN Patch Encoder––1×10−31\times 10^{-3}200032
Transformer Encoder4161×10−31\times 10^{-3}30032
TCNTemporal Conv. Network4–1×10−31\times 10^{-3}30032
PatchTSTPatch-based Transformer4161×10−31\times 10^{-3}30032
TABLE III: Hyperparameter settings for the proposed model and baseline methods.
HorizonModelMSEMAERMSE
h=1h=1 (8 steps)Proposed CNN–Attention–Transformer0.0760000.2050000.275681
TCN0.2380000.3550000.487852
PatchTST0.0950000.2180000.308221
h=3h=3 (24 steps)Proposed CNN–Attention–Transformer0.0588600.1909120.242611
TCN0.1059020.2184610.325426
PatchTST0.1010540.2428110.317890
TABLE IV: Forecasting performance on the synthetic dataset. Lower values indicate better performance.
ModelMSEMAERMSE
Proposed model0.0906650.1425130.301105
TCN0.1451500.1957490.380985
PatchTST0.1284510.1873120.358400
TABLE V: Test-set performance on Electricity Load Diagrams 2011–2014.
ConfigurationMSEMAERMSE
Full model0.0976700.2226520.312522
No dense connections1.3047640.7816931.142263
Mean pooling (no attention pool)2.0298640.8107441.424733
No token attention0.5467550.4459930.739429
TABLE VI: Ablation results on the synthetic dataset. Lower values indicate better performance.
More in this topic

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 289 papers

Explore similar work

CardsList
Mar 25, 2026

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Accurate forecasting of multivariate time series remains challenging due to the need to capture both short-term fluctuations and long-range temporal dependencies. Transformer-based models have emerged as a powerful approach, but their performance depends critically on the representation of temporal data. Traditional point-wise representations preserve individual time-step information, enabling fine-grained modeling, yet they tend to be computationally expensive and less effective at modeling broader contextual dependencies, limiting their scalability to long sequences. Patch-wise representations aggregate consecutive steps into compact tokens to improve efficiency and model local temporal dynamics, but they often discard fine-grained temporal details that are critical for accurate predictions in volatile or complex time series. We propose IPatch, a multi-resolution Transformer architecture that integrates both point-wise and patch-wise tokens, modeling temporal information at multiple resolutions. Experiments on 7 benchmark datasets demonstrate that IPatch consistently improves forecasting accuracy, robustness to noise, and generalization across various prediction horizons compared to single-representation baselines.

Aymane Harkati, Moncef Garouani, Olivier Teste +2

CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Robust Time Series ForecastingTime Series ForecastingMultivariate Time Series ForecastingTime Series TransformersMultiscale Time-Series Modeling
AskPDF
Mar 11, 2026

TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting

Transformer-based time series foundation models face a fundamental trade-off in choice of tokenization: point-wise embeddings preserve temporal fidelity but scale poorly with sequence length, whereas fixed-length patching improves efficiency by imposing uniform boundaries that may disrupt natural transitions and blur informative local dynamics. In order to address these limitations, we introduce TimeSqueeze, a dynamic patching mechanism that adaptively selects patch boundaries within each sequence based on local signal complexity. TimeSqueeze first applies a lightweight state-space encoder to extract full-resolution point-wise features, then performs content-aware segmentation by allocating short patches to information-dense regions and long patches to smooth or redundant segments. This variable-resolution compression preserves critical temporal structure while substantially reducing the token sequence presented to the Transformer backbone. Specifically for large-scale pretraining, TimeSqueeze attains up to 20x faster convergence and 8x higher data efficiency compared to equivalent point-token baselines. Experiments across long-horizon forecasting benchmarks show that TimeSqueeze consistently outperforms comparable architectures that use either point-wise tokenization or fixed-size patching.

Sravan Kumar Ankireddy, Nikita Seleznev, Nam H. Nguyen +4

University of Texas at Austin, Austin, TX, USA · Capital One, USA · University of Maryland, College Park, MD, USA

Time Series ForecastingTime Series TransformersTime-Series Foundation Models
AskPDF
Aug 20, 2026

Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured Partitioning

Multivariate time series forecasting (MTSF) is a fundamental task in many real world applications. Existing patch based forecasting methods generally fall into three categories: fixed partitioning, multi-scale partitioning, and extendable partitioning. Fixed partitioning often breaks meaningful temporal boundaries, multi-scale partitioning may introduce redundant representations across scales, and extendable partitioning improves flexibility but still lacks an explicit mechanism for organizing semantic structure and modeling interactions among heterogeneous temporal patterns. To address these limitations, we propose SCPaT, a Transformer based framework built on semantic structured partitioning. SCPaT first decomposes input sequences into semantically consistent units through adaptive semantic unit generation, then constructs a dynamic semantic graph to model directed dependencies among these units and organize them into higher order semantic blocks. Based on these structured representations, an importance aware routing mechanism adaptively dispatches different semantic blocks to different experts for customized modeling. Extensive experiments on 12 real world datasets demonstrate the effectiveness of SCPaT.

Zhiquan Huang, Jiazhe Wang, Linjing Xue +3

School of Information Engineering, Henan University of Science and Technology, Luoyang, 471023, Henan, China · School of Software, Henan University of Science and Technology, Luoyang, 471000, Henan, China

Multivariate Time Series ForecastingTime Series ForecastingTime Series Transformers
AskPDF