Learning from Points and Patches: A Pre-trained Hybrid Attention Transformer for Time Series Analysis

Peiwang Tang; Cheng HE; Yupeng Ye; Weitai Zhang; Qinghua Sun; Defu Lian

cs.LGOct 27, 2026

Learning from Points and Patches: A Pre-trained Hybrid Attention Transformer for Time Series Analysis

Authors: Peiwang Tang, Cheng HE, Yupeng Ye, Weitai Zhang, Qinghua Sun, Defu Lian

Organizations: University of Science and Technology of China, Hefei, Anhui, China · State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China · Shanghai Black Wing Asset Management, Shanghai, China · iFLYTEK Research, Hefei, Anhui, China

Abstract

Time-series data generated by Web services and connected systems support applications such as user activity analysis, service monitoring, and resource management. However, learning transferable representations from heterogeneous time series remains challenging due to diverse temporal patterns, variable sequence lengths, and limited high-quality annotations. Despite the success of pre-training in natural language processing and computer vision, effective time-series pre-training requires strategies tailored to temporal data. We propose TimeHAT (Hybrid Attention Transformer), a general-purpose pre-training framework for time-series analysis that addresses two complementary questions: what to learn and how to learn. For what to learn, we introduce a Unified Point-Patch Masking Strategy (UPPMS) that integrates point-level and patch-level supervision within a unified framework for contextual reconstruction and causal extrapolation, enabling the model to capture both fine-grained temporal variations and broader temporal patterns. For how to learn, we develop a Hybrid Attention Encoder (HAE) that combines prefix-constrained bidirectional attention in initial layers with directional hybrid attention in deeper layers, aiming to preserve effective contextual interactions while mitigating rank degeneration. We evaluate TimeHAT across real-world datasets and four downstream tasks: classification, long-term forecasting, imputation, and anomaly detection. Experiments cover fine-tuning, few-shot, and zero-shot transfer settings, demonstrating competitive performance across diverse time-series analysis tasks. Code is available at https://github.com/TangPeiwang/TimeHAT.

Figures & tables

Figure 1. (a) Patch-level reconstruction can miss fine-grained temporal details; (b) Deep bidirectional attention risks representation homogenization; (c) Causal attention cannot access two-sided context.

Figure 2. Comparison of Point-Level, Patch-Level, and Unified Point-Patch Masking Strategies for Time Series Data.

Figure 3. Overall architecture of TimeHAT. The model employs channel-independence for multivariate inputs. Sequences undergo a Unified Point-Patch Masking Strategy, patch tokenization, and MLP embedding. A two-stage encoder then captures temporal dynamics, utilizing Bidirectional Attention in early layers for global context, and Hybrid Attention in later layers for causal constraints. Finally, an MLP projection head reconstructs the output by modeling both patch- and point-level interactions.

Methods	TimeHAT		MOMENT		GPT4TS		TimesNet		PatchTST		Timer-XL		iTransformer		CrossLinear		TimeMixer		TimeMixer++
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh 1	0.402	0.421	0.418	0.436	0.427	0.426	0.458	0.450	0.413	0.430	0.409	0.430	0.454	0.447	0.431	0.433	0.447	0.440	0.419	0.432
ETTh 2	0.322	0.368	0.352	0.395	0.346	0.394	0.414	0.427	0.330	0.379	0.352	0.402	0.383	0.407	0.368	0.396	0.364	0.395	0.339	0.380
ETTm 1	0.336	0.366	0.344	0.379	0.352	0.383	0.400	0.406	0.351	0.387	0.359	0.382	0.407	0.410	0.370	0.393	0.369	0.378	0.369	0.378
ETTm 2	0.252	0.309	0.259	0.318	0.266	0.326	0.291	0.333	0.255	0.315	0.271	0.322	0.288	0.332	0.272	0.320	0.275	0.323	0.269	0.320
Weather	0.221	0.255	0.228	0.270	0.237	0.270	0.259	0.287	0.225	0.264	0.240	0.273	0.258	0.279	0.238	0.269	0.240	0.271	0.226	0.262

Table 1. Long-term forecasting task. Results are averaged over four prediction horizons: {96, 192, 336, 720}. Other models all adhere to the core research conclusions proposed in the original papers. The best results are represented in bold .

Methods	TimeHAT		GPT4TS		TimesNet		PatchTST		ETSformer		LightTS		DLinear		FEDformer		Non-Stationary		Autoformer
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTm 1	0.021	0.095	0.028	0.105	0.027	0.107	0.047	0.140	0.120	0.253	0.104	0.218	0.093	0.206	0.062	0.177	0.036	0.126	0.051	0.150
ETTm 2	0.015	0.076	0.021	0.084	0.022	0.088	0.029	0.102	0.208	0.327	0.046	0.151	0.096	0.208	0.101	0.215	0.026	0.099	0.029	0.105
ETTh 1	0.052	0.151	0.069	0.173	0.078	0.187	0.115	0.224	0.202	0.329	0.284	0.373	0.201	0.305	0.17	0.246	0.094	0.201	0.103	0.314
ETTh 2	0.041	0.134	0.048	0.141	0.049	0.146	0.065	0.163	0.367	0.436	0.119	0.250	0.142	0.259	0.163	0.279	0.053	0.152	0.055	0.156
Electricity	0.066	0.182	0.090	0.207	0.092	0.210	0.072	0.183	0.214	0.339	0.131	0.262	0.132	0.260	0.130	0.259	0.100	0.218	0.101	0.225

Table 2. Imputation task. We randomly mask the input time series at {12.5%, 25%, 37.5%, 50%}. The results are averaged from 4 different mask ratios. The best performance is shown in Bold .

Methods	SMD	MSL	SMAP	SWaT	PSM	Average
TimeHAT	86.95	89.46	76.36	94.64	97.71	89.02
TIMESBERT	86.04	88.07	75.69	93.95	98.27	88.40
GPT4TS	86.89	82.45	72.88	94.23	97.13	86.72
TimesNet	84.61	81.84	69.39	93.02	97.34	85.24
Pyraformer	83.04	84.86	71.09	91.78	82.08	82.57
Anomaly Transformer	85.49	83.31	71.18	83.10	79.40	80.50

Table 3. Anomaly detection task. We calculate the F1-score (as %) for each dataset. Bold represents the best.

Figure 4. Model comparison in classification.

Dataset	TimeHAT	GTM	Timer	Moirai	MOMENT
ETTh1	0.399	0.407	0.438	0.441	0.674
ETTm1	0.558	0.593	0.690	0.562	0.670
Weather	0.170	0.172	0.181	0.195	0.255
Electricity	0.176	0.187	0.192	0.212	0.744
Traffic	0.447	0.542	0.458	0.616	1.293

Table 4. Zero-shot capability (MSE).

Panel of Figure 5 — Figure 5. Few-shot capability of TimeHAT compared to baselines.

Figure 6. Relative error changes of recursive versus direct forecasting across pre-training schemes on ETTh1 and ETTm2. Recon: Contextual Reconstruction; Tail: Causal Extrapolation; Joint: joint objectives; Full: full UPPMS.

Variant	ETTh2		ETTm1		ETTm2		Weather
Variant	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
LPR	+2.05	+0.79	-0.39	+0.01	+0.80	-0.16	-0.31	-0.25
LPA	+1.55	+0.45	+0.02	-0.09	-0.45	-0.79	+0.17	-0.08
PO	+1.79	+0.31	-0.37	-0.20	+0.40	-0.26	-0.70	-0.88
ZO	-1.00	-0.46	+0.36	+0.27	+1.06	+0.27	+0.10	+0.07
FTL	-0.92	-0.61	+0.40	+0.07	+1.52	-1.08	-0.07	-0.22

Table 5. Relative changes in forecasting errors under UPPMS ablations (%). Positive values indicate higher errors than the full method. LPR: w/o last-patch recovery; LPA: w/o left-padding augmentation; PO: patch-only masking; ZO: zero-only replacement; FTL: fixed tail length.

Dataset	Full		Tail		Joint		Random
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.384	0.409	0.390	0.413	0.386	0.410	0.387	0.410
ETTh2	0.300	0.353	0.321	0.364	0.307	0.356	0.308	0.356
ETTm1	0.294	0.355	0.297	0.353	0.301	0.360	0.298	0.357
ETTm2	0.177	0.266	0.179	0.269	0.178	0.268	0.177	0.268
Weather	0.152	0.202	0.153	0.203	0.154	0.204	0.153	0.203

Table 6. Forecasting performance of different pre-training objectives. Full: full UPPMS; Tail: Causal Extrapolation; Joint: joint objectives; Random: Contextual Reconstruction.

Dataset	Metric	$0$	$0.25$	$0.40$	$0.50$	$0.75$	$1$
ETTh1	MSE	0.403	0.394	0.396	0.395	0.396	0.402
ETTh1	MAE	0.412	0.411	0.412	0.411	0.412	0.416
ETTm2	MSE	0.176	0.176	0.173	0.173	0.178	0.177
ETTm2	MAE	0.257	0.258	0.258	0.257	0.258	0.263
Weather	MSE	0.150	0.150	0.149	0.149	0.149	0.152
Weather	MAE	0.195	0.195	0.193	0.193	0.192	0.195

Table 7. Forecasting performance under different tail-task probabilities.

Figure 7. Forecasting MSE of different attention mechanisms across prediction horizons on ETTh1 and ETTm1.

Figure 8. Forecasting MSE of different attention mechanisms across network depths on ETTh2 and ETTm2.

Figure 9. Forecasting MSE with different numbers of prefix-constrained BiA layers on ETTh2 and Weather.

Config	ETTh1		ETTm1		ETTm2		Weather
Config	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MMMMMM	0.507	0.492	0.437	0.440	0.278	0.337	0.242	0.280
BBBMMM	0.598	0.541	0.486	0.464	0.287	0.345	0.241	0.280
BBBBBB	0.721	0.596	0.622	0.527	0.310	0.364	0.252	0.292
MMMBBB	0.538	0.508	0.444	0.444	0.280	0.339	0.249	0.285
BMBMBM	0.683	0.574	0.442	0.443	0.286	0.343	0.241	0.280

Table 8. Forecasting performance of different BiA–HA layer configurations. B and M denote BiA and HA, respectively.

Figure 10. Sensitivity to readout strategies across attention mechanisms.

Figure 11. Performance of Different Proportions of Causal Attention in Hybrid Attention.

Attention Rule	ETTh1		ETTh2		ETTm1		ETTm2
Attention Rule	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Default	0.386	0.410	0.307	0.355	0.298	0.357	0.177	0.268
w/o TSR	0.389	0.413	0.310	0.359	0.304	0.362	0.183	0.267
Hist $\rightarrow$ F	0.387	0.410	0.308	0.356	0.301	0.359	0.179	0.267
w/o DS	0.388	0.413	0.310	0.359	0.300	0.359	0.179	0.267

Table 9. Downstream forecasting performance under different pre-training attention rules. TSR: task-specific rules; Hist

\rightarrow

F: historical queries can attend to forecast tokens; DS: direction switching for forecast queries.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dim	Input Length	Prediction Length	Time Points	Dataset Splitting	Frequency
ETTm1 ETTm2	7	336	{96,192,336,720}	69,680	6:2:2	15 mins
ETTh1 ETTh2	7	336	{96,192,336,720}	17,420	6:2:2	Hourly
Electricity	321	336	{96,192,336,720}	26,304	7:1:2	Hourly
Traffic	862	336	{96,192,336,720}	17,544	7:1:2	Hourly
Weather	21	336	{96,192,336,720}	52,696	7:1:2	10 mins
Exchange	8	336	{96,192,336,720}	7,588	7:1:2	Daily

Appendix

Table 10. Long-term forecasting dataset descriptions. The Dataset Splitting is organized in (Train,Validation,Test).

Tasks	Dataset	Dim	Series Length	Dataset Size
Imputation	ETTm1,ETTm2 (Electricity)	7	96	(34465,11521,11521)
	ETTh1,ETTh2 (Electricity)	7	96	(8545,2881,2881)
	Electricity	321	96	(18317,2633,5261)
	Weather	21	96	(36792,5271,10540)
Classification (UEA)	EthanolConcentration (Alcohol Industry)	3	1751	(261,0,263)
	FaceDetection (Face)	144	62	(5890,0,3524)

Appendix

Table 11. Other dataset descriptions. The dataset size is organized in (Train,Validation,Test).

Dataset	$P=1$						$P=16$
	Scratch		Patch		Point		Scratch		Patch		Point
	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.465	0.466	0.465	0.468	0.466	0.469	0.458	0.453	0.447	0.445	0.444	0.444
ETTh2	0.406	0.422	0.405	0.424	0.401	0.422	0.386	0.410	0.385	0.410	0.387	0.410
ETTm1	0.403	0.415	0.408	0.419	0.403	0.417	0.393	0.406	0.383	0.402	0.383	0.401
ETTm2	0.294	0.339	0.306	0.349	0.298	0.343	0.290	0.336	0.291	0.337	0.289	0.334

Appendix

Table 12. Forecasting performance with embedding and encoder transfer.

P

denotes the embedding patch size. Scratch: training from scratch; Patch: patch-level masking; Point: point-level masking.

Dataset	Patch $P=1$		Patch $P=16$		Point $P=1$		Point $P=16$
Dataset	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.450	0.449	0.448	0.445	0.456	0.452	0.447	0.445
ETTh2	0.389	0.411	0.385	0.410	0.387	0.411	0.387	0.410
ETTm1	0.392	0.405	0.386	0.403	0.393	0.407	0.387	0.404
ETTm2	0.288	0.334	0.288	0.333	0.288	0.334	0.286	0.332
Weather	0.254	0.277	0.254	0.277	0.254	0.277	0.253	0.276

Appendix

Table 13. Forecasting performance with encoder-only transfer. All downstream models use a newly initialized

P=16

embedding and prediction head.

P

denotes the embedding patch size. Patch and Point denote patch-level and point-level masking, respectively.

Figure 12. Qualitative reconstruction examples under block and point masking on five datasets. Each row shows the same selected sequence and vertical scale.

Self-Supervised Time Series Representation Learning

Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.

See all 144 papers

Sep 19, 2026

A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting

Time series foundation models (TSFMs) have recently delivered impressive zero-shot performance across diverse forecasting tasks. However, real-world decision-making frequently relies on irregular multivariate time series (IMTS), where inconsistent inter-observation intervals and asynchronous sampling across variables coexist with informative missingness. Existing TSFMs handle such inputs either through imputation that injects spurious values or through index-based positional encodings that ignore continuous time. There is still a gap in the foundation model that follows the original IMTS patterns. In this paper, we propose a hybrid attention model that learns a unified time-aware patch representation for IMTS forecasting. We first design a time-aware patch encoding that maps a variable number of intra-patch timestamps into a fixed-size embedding, producing a uniform format for irregular patches without resorting to imputation. We then introduce a time bias attention mechanism that calibrates inter-patch temporal misalignment and asynchronous cross-channel dependencies as auxiliary attention offset. Finally, on top of a decoder-only Transformer backbone, we adopt a hybrid causal mask that preserves a bidirectional full view over the historical context while keeping the forecast horizon strictly autoregressive. To support large-scale pretraining under irregular settings, we also curate VersaTSA, an archive of

30

B observations that retains the native sampling sparsity of its sources. Experiments on three IMTS benchmarks and a standard regular-MTS benchmark show that our model achieves state-of-the-art zero-shot performance on IMTS and remains competitive when transferred to regular forecasting.

Li Lin, Zhihao Lin, Qi Zhang +3

Southeast University, China · Nanyang Technological University, Singapore · Timecho Ltd., China

Irregular Time-Series Modeling Multivariate Time Series Forecasting Time Series Forecasting Hybrid Attention Time-Series Foundation Models

Ask PDF

Mar 25, 2026

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Accurate forecasting of multivariate time series remains challenging due to the need to capture both short-term fluctuations and long-range temporal dependencies. Transformer-based models have emerged as a powerful approach, but their performance depends critically on the representation of temporal data. Traditional point-wise representations preserve individual time-step information, enabling fine-grained modeling, yet they tend to be computationally expensive and less effective at modeling broader contextual dependencies, limiting their scalability to long sequences. Patch-wise representations aggregate consecutive steps into compact tokens to improve efficiency and model local temporal dynamics, but they often discard fine-grained temporal details that are critical for accurate predictions in volatile or complex time series. We propose IPatch, a multi-resolution Transformer architecture that integrates both point-wise and patch-wise tokens, modeling temporal information at multiple resolutions. Experiments on 7 benchmark datasets demonstrate that IPatch consistently improves forecasting accuracy, robustness to noise, and generalization across various prediction horizons compared to single-representation baselines.

Aymane Harkati, Moncef Garouani, Olivier Teste +2

CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Robust Time Series Forecasting Time Series Forecasting Multivariate Time Series Forecasting Time Series Transformers Multiscale Time-Series Modeling

Ask PDF

Sep 12, 2026

Tabby: An Open Pretraining Recipe for Time Series Foundation Models

In this report, we release Tabby, a long context probabilistic time series foundation model, together with a complete and open recipe of how it was built. Tabby adopts an encoder-only patch Transformer architecture and concentrates the contributions on the data and the training procedure. The pretraining corpus combines an extended real-world collection, GIFT-Eval-Pretrain+ and BLAST, with synthetic data from KernelSynth and CauKerV2, an online generator that composes temporal dynamics through randomly sampled structural causal models. Training couples a progressive convergence schedule, which yields reusable intermediate checkpoints, with a deep quantile supervision objective for intermediate layers. The resulting 145M parameter backbone supports contexts of up to 8,192 observations and serves forecasting, classification, and anomaly detection, while a prompt-tuning module further improves in-distribution forecasting performance with the pretrained weights frozen. Tabby achieves competitive zero-shot forecasting performance on GIFT-Eval and the out-of-distribution TIME benchmark, while the same pretrained backbone also supports classification on the UCR Archive and zero-shot anomaly detection on TSB-AD-U. We release training pipeline and model as open source at huawei-noah/trustworthyAI.

Shifeng Xie, Bahaeddine Abdessalem, Zehao Xiao +9

Huawei Noah’s Ark Lab, Paris, France · LIPADE, Université Paris Cité, Paris, France · École Polytechnique, France +3

Time-Series Foundation Models Time Series Transformers Probabilistic Time Series Forecasting Zero-Shot Time Series Forecasting

Ask PDF

Abstract

Self-Supervised Time Series Representation Learning

Explore similar work

A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Tabby: An Open Pretraining Recipe for Time Series Foundation Models