Entropy Guided Dynamic Patch Segmentation for Time Series Transformers
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Institute for Infocomm Research, A*STAR, Singapore · Department of Computer Science, Khalifa University, UAE · Information Systems Technology and Design, Singapore University of Technology and Design, Singapore
Abstract
Patch-based transformers have emerged as efficient and improved long-horizon modeling architectures for time series modeling. Yet, existing approaches rely on temporally-agnostic patch construction, where arbitrary starting positions and fixed lengths fracture temporal coherence by splitting natural transitions across boundaries. This naive segmentation often disrupts short-term dependencies and weakens representation learning. We propose a novel Entropy-Guided Dynamic Patch Encoder (EntroPE), as a temporally informed framework that dynamically detects transition points via conditional entropy and dynamically places patch boundaries. This preserves temporal structure while retaining the computational benefits of patching. EntroPE consists of two key modules, namely an Entropy-based Dynamic Patcher (EDP) that applies information-theoretic criteria to locate natural temporal shifts and determine patch boundaries, and an Adaptive Patch Encoder (APE) that employs pooling and cross-attention to capture intra-patch dependencies and produce fixed-size latent representations. Extensive experiments on long-term forecasting, classification, and anomaly detection demonstrate that the proposed method improves both accuracy and efficiency, establishing entropy-guided dynamic patching as a promising new paradigm for time series modeling. Code is available at https://github.com/Sachithx/EntroPE.
Figures & tables
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Solar | Electricity | Traffic | Overall | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| EntroPE | 0.427 | 0.431 | 0.369 | 0.402 | 0.379 | 0.394 | 0.285 | 0.332 | 0.247 | 0.277 | 0.239 | 0.275 | 0.193 | 0.284 | 0.498 | 0.298 | 0.330 | 0.336 |
| TimeMosaic | 0.437 | 0.426 | 0.375 | 0.393 | 0.389 | 0.384 | 0.283 | 0.319 | 0.257 | 0.273 | 0.268 | 0.269 | 0.199 | 0.276 | 0.485 | 0.290 | 0.336 | 0.329 |
| CALF | 0.443 | 0.435 | 0.379 | 0.402 | 0.403 | 0.397 | 0.285 | 0.324 | 0.257 | 0.277 | 0.242 | 0.243 | 0.192 | 0.271 | 0.467 | 0.285 | 0.333 | 0.329 |
| iTransformer | 0.457 | 0.449 | 0.396 | 0.414 | 0.421 | 0.419 | 0.293 | 0.335 | 0.261 | 0.281 | 0.244 | 0.268 | 0.182 | 0.271 | 0.442 | 0.289 | 0.337 | 0.341 |
| TimeMixer | 0.483 | 0.454 | 0.397 | 0.413 | 0.389 | 0.401 | 0.282 | 0.326 | 0.250 | 0.278 | 0.250 | 0.287 | 0.186 | 0.276 | 0.495 | 0.310 | 0.341 | 0.343 |
| Dataset | EntroPE | TSLANet | GPT4TS | TimesNet | ROCKET | CrossF. | PatchTST | MLP | TS-TCC |
|---|---|---|---|---|---|---|---|---|---|
| EthanolConcentration | 29.10 | 30.42 | 25.48 | 27.73 | 42.58 | 34.98 | 28.90 | 33.46 | 32.32 |
| FaceDetection | 69.10 | 66.77 | 65.58 | 67.47 | 64.70 | 66.17 | 68.96 | 67.42 | 63.05 |
| Handwriting | 57.40 | 57.88 | 34.56 | 26.18 | 48.47 | 26.24 | 26.00 | 22.47 | 47.76 |
| Heartbeat | 80.00 | 77.56 | 36.59 | 74.48 | 69.76 | 76.59 | 76.59 | 73.17 | 77.07 |
| JapaneseVowels | 99.20 | 99.19 | 98.11 | 97.83 | 95.68 | 98.92 | 98.65 | 97.84 | 97.30 |
| PEMS-SF | 88.60 | 83.82 | 87.28 | 88.13 | 75.10 | 82.08 | 88.44 | 82.08 | 86.71 |
| Dataset | EntroPE | TSLANet | GPT4TS | TimesNet | PatchTST | ETSformer | FEDformer | LightTS | DLinear |
|---|---|---|---|---|---|---|---|---|---|
| SMD | 87.10 | 87.91 | 86.89 | 84.61 | 84.62 | 83.13 | 85.08 | 82.53 | 77.10 |
| MSL | 85.30 | 83.32 | 82.45 | 81.84 | 78.70 | 85.03 | 78.57 | 78.95 | 84.88 |
| SMAP | 77.93 | 75.96 | 72.88 | 69.39 | 68.82 | 69.50 | 70.76 | 69.21 | 69.26 |
| SWaT | 94.30 | 92.80 | 94.23 | 93.02 | 85.72 | 84.91 | 93.19 | 93.33 | 87.52 |
| PSM | 97.70 | 97.73 | 97.13 | 97.34 | 96.08 | 91.76 | 97.23 | 97.15 | 93.55 |
| Average | 88.47 | 87.54 | 86.72 | 85.24 | 82.79 | 82.87 | 84.97 | 84.23 | 82.46 |
| Dataset | Full | EDP | EDP APE | EDP APE FD | |||
|---|---|---|---|---|---|---|---|
| Dyn. | Stat(1) | Stat(8) | Stat(16) | Pool+FD | Pool+Flat | ||
| ETTh1 | 336 | 0.429 | 0.425 | 0.441 | 0.444 | 0.438 | 0.519 |
| 720 | 0.439 | 0.460 | 0.469 | 0.477 | 0.461 | 0.527 | |
| ETTh2 | 336 | 0.355 | 0.428 | 0.435 | 0.439 | 0.439 | 0.462 |
| 720 | 0.397 | 0.448 | 0.470 | 0.463 | 0.466 | 0.456 | |
| ETTm1 | 336 | 0.393 | 0.401 | 0.409 | 0.415 | 0.402 | 0.427 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Dim | Reported split sizes (train, val., test) | Frequency | Domain |
|---|---|---|---|---|
| ETTh1 | 7 | (8545, 2881, 2881) | Hourly | Energy Infrastructure |
| ETTh2 | 7 | (8545, 2881, 2881) | Hourly | Energy Infrastructure |
| ETTm1 | 7 | (34465, 11521, 11521) | 15 min | Energy Infrastructure |
| ETTm2 | 7 | (34465, 11521, 11521) | 15 min | Energy Infrastructure |
| Weather | 21 | (36792, 5271, 10540) | 10 min | Meteorological Monitoring |
| Electricity (ECL) | 321 | (18317, 2633, 5261) | Hourly | Electricity Consumption |
| Dataset | Dim | Reported split sizes (train, val., test) | Length | Domain |
|---|---|---|---|---|
| SMD | 38 | (566724, 141681, 708420) | 100 | Server Machine |
| MSL | 55 | (44653, 11664, 73729) | 100 | Spacecraft |
| SMAP | 25 | (108146, 27037, 427617) | 100 | Spacecraft |
| SWaT | 51 | (396000, 99000, 449919) | 100 | Infrastructure |
| PSM | 25 | (105984, 26497, 87841) | 100 | Server Machine |
| Model | Category | Key Features / Innovations |
| TimeMosaic [ 8 ] | Transformer | patch granularity based on temporal heterogeneity and motif reuse. |
| iTransformer [ 25 ] | Transformer | Inverted attention across feature and time dimensions. |
| PatchTST [ 27 ] | Transformer | Patch-based embedding with channel-independence. |
| FEDformer [ 52 ] | Transformer (freq.-decomp.) | Fourier/Wavelet enhanced decomposition. |
| TimesNet [ 44 ] | CNN | Temporal 2D variation modeling with convolutional layers. |
| TSLANet [ 11 ] | CNN + Lightweight | Lightweight and pretraining module for enhanced feature representation |
| Hyperparameter | Value |
| Number of layers ( ) | 2 |
| Number of heads ( ) | 4 |
| Embedding dimension ( ) | 16 |
| Dropout | 0.1 |
| Bias | False |
| Vocabulary size ( ) | 256 |
| Hyperparameter | Value |
|---|---|
| Input | 96 |
| Global transformer layers | 2 |
| Patch encoder / fusion decoder layers | 1 / 1 |
| Attention heads | 4 |
| FFN dimension ( ) | 256 |
| Entropy vocabulary size | 256 |
| Dataset | Size | Batch | LR | Dropout | Mono. | ||
|---|---|---|---|---|---|---|---|
| ETTh1 | 96 | M | 64 | 0.95 | 0.1 | 0 | |
| 192 | S | 128 | 0.75 | 0.1 | 1 | ||
| 336 | S | 64 | 0.85 | 0.1 | 0 | ||
| 720 | S | 32 | 0.85 | 0.2 | 0 | ||
| ETTh2 | 96 | M | 32 | 0.85 | 0.3 | 1 | |
| 192 | L | 128 | 0.95 | 0.2 | 0 |
| Models | Horizon | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Solar | ECL | Traffic | All Dataset | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | ||
| FilterTS | 96 | 0.388 | 0.396 | 0.292 | 0.341 | 0.325 | 0.362 | 0.174 | 0.256 | 0.173 | 0.217 | 0.230 | 0.272 | 0.174 | 0.261 | 0.466 | 0.319 | 0.278 | 0.303 |
| 192 | 0.436 | 0.423 | 0.378 | 0.392 | 0.367 | 0.384 | 0.240 | 0.299 | 0.223 | 0.260 | 0.268 | 0.296 | 0.184 | 0.270 | 0.472 | 0.318 | 0.321 | 0.330 | |
| 336 | 0.479 | 0.446 | 0.423 | 0.429 | 0.400 | 0.405 | 0.303 | 0.340 | 0.279 | 0.300 | 0.298 | 0.314 | 0.199 | 0.285 | 0.488 | 0.325 | 0.359 | 0.356 | |
| 720 | 0.482 | 0.472 | 0.429 | 0.444 | 0.464 | 0.441 | 0.409 | 0.400 | 0.356 | 0.350 | 0.299 | 0.314 | 0.241 | 0.319 | 0.520 | 0.343 | 0.400 | 0.385 | |
| Avg. | 0.446 | 0.434 | 0.381 | 0.402 | 0.389 | 0.398 | 0.282 | 0.324 | 0.258 | 0.282 | 0.274 | 0.299 | 0.200 | 0.284 | 0.487 | 0.326 | 0.339 | 0.344 | |
| Dataset | L T | S-MSE | S-MAE | M-MSE | M-MAE | L-MSE | L-MAE | TimeMosaic |
|---|---|---|---|---|---|---|---|---|
| (MSE/MAE) | ||||||||
| M4-Monthly | 36 18 | .200/.199 | .195/.197 | .196/.199 | .193/.196 | .196/.195 | .192/.192 | .198/.194 |
| ETTh1 | 512 96 | .392/.420 | .417/.437 | .408/.422 | .427/.438 | .387/.443 | .410/.452 | .389/.418 |
| ETTh2 | 512 96 | .312/.318 | .371/.380 | .294/.324 | .357/.384 | .322/.334 | .375/.380 | .314/.318 |
| ETTm1 | 512 96 | .310/.326 | .357/.370 | .324/.337 | .364/.381 | .305/.313 | .353/.361 | .320/.362 |
| ETTm2 | 512 96 | .182/.188 | .270/.276 | .178/.194 | .271/.277 | .179/.194 | .267/.279 | .179/.279 |
| Dataset | L T | S-MSE | S-MAE | M-MSE | M-MAE | L-MSE | L-MAE |
|---|---|---|---|---|---|---|---|
| M4-Monthly | 36 18 | -0.313 | +0.770 | +1.336 | +1.317 | -0.465 | -0.346 |
| ETTh1 | 512 96 | +6.845 | +4.467 | +3.201 | +2.490 | +12.654 | +9.340 |
| ETTh2 | 512 96 | +1.945 | +2.530 | +9.462 | +6.998 | +3.591 | +1.264 |
| ETTm1 | 512 96 | +5.074 | +3.594 | +3.783 | +4.561 | +2.432 | +2.164 |
| ETTm2 | 512 96 | +3.276 | +2.272 | +8.108 | +1.976 | +7.700 | +4.310 |
| Weather | 512 96 | +7.360 | +6.868 | +4.876 | +2.968 | +2.086 | +2.229 |
| Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | ECL | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| PatchTST | 0.516 | 0.484 | 0.391 | 0.411 | 0.406 | 0.407 | 0.290 | 0.334 | 0.265 | 0.285 | 0.216 | 0.318 |
| EntroPE | 0.416 | 0.425 | 0.366 | 0.387 | 0.378 | 0.391 | 0.286 | 0.335 | 0.242 | 0.273 | 0.182 | 0.271 |
| % Improvement | 19% | 12% | 6% | 6% | 7% | 4% | 1% | 0% | 9% | 4% | 16% | 15% |
| Dataset | Metric | TimeMosaic | CALF | iTransformer | HDMixer | EntroPE |
|---|---|---|---|---|---|---|
| ETTh1 | MSE | 0.437 0.004 | 0.443 0.002 | 0.457 0.003 | 0.452 0.003 | 0.427 0.004 |
| MAE | 0.426 0.003 | 0.435 0.001 | 0.449 0.002 | 0.438 0.003 | 0.431 0.002 | |
| ETTh2 | MSE | 0.375 0.002 | 0.379 0.003 | 0.396 0.005 | 0.383 0.005 | 0.369 0.005 |
| MAE | 0.393 0.001 | 0.402 0.001 | 0.414 0.003 | 0.405 0.001 | 0.402 0.003 | |
| ETTm1 | MSE | 0.389 0.003 | 0.403 0.001 | 0.421 0.004 | 0.398 0.002 | 0.379 0.003 |
| MAE | 0.384 0.001 | 0.397 0.000 | 0.419 0.003 | 0.402 0.001 | 0.394 0.001 |
| Dataset | Horizon | MSE | MAE |
|---|---|---|---|
| ETTh1 | 96 | 0.385 0.004 | 0.407 0.002 |
| 192 | 0.430 0.001 | 0.433 0.001 | |
| 336 | 0.460 0.004 | 0.446 0.003 | |
| 720 | 0.468 0.013 | 0.469 0.007 | |
| ETTh2 | 96 | 0.333 0.005 | 0.376 0.004 |
| 192 | 0.409 0.007 | 0.421 0.005 |
| Models | Parameters (M) | MACs (G) | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Solar | ECL | Traffic | Avg. | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Solar | ECL | Traffic | Avg. | |
| FilterTS | 0.546 | 0.565 | 0.703 | 0.369 | 0.246 | 0.438 | 1.032 | 4.909 | 1.101 | 0.005 | 0.006 | 0.007 | 0.003 | 0.006 | 0.038 | 0.273 | 1.178 | 0.190 |
| HDMixer | 0.072 | 0.071 | 0.072 | 0.072 | 0.074 | 0.185 | 0.692 | 4.536 | 0.722 | 0.001 | 0.001 | 0.001 | 0.001 | 0.003 | 0.036 | 0.153 | 0.948 | 0.143 |
| TimeBase | 0.001 | 0.001 | 0.002 | 0.002 | 0.015 | 0.101 | 0.001 | 0.001 | 0.015 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.004 | 0.001 |
| TimeMosaic | 10.033 | 7.669 | 6.091 | 7.669 | 10.033 | 11.609 | 8.457 | 5.310 | 8.359 | 2.642 | 1.892 | 1.330 | 1.798 | 7.925 | 60.872 | 95.359 | 129.305 | 37.640 |
| TSLANet | 0.172 | 0.172 | 0.620 | 0.342 | 0.235 | 0.235 | 0.715 | 0.715 | 0.401 | 0.003 | 0.003 | 0.023 | 0.004 | 0.009 | 0.058 | 0.868 | 2.330 | 0.412 |
| EntroPE-M | TimeMosaic | PatchTST | LangTime [ 28 ] | |
|---|---|---|---|---|
| 176.68K | 6.40M ( ) | 408.98K ( ) | 494M+ ( ) | |
| 952.83K | 9.75M ( ) | 1.19M ( ) | 494M+ ( ) | |
| 4.10M | 22.49M ( ) | 4.33M ( ) | 494M+ ( ) |
| Metric/Dataset | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | ECL |
|---|---|---|---|---|---|---|
| Epoch (dynamic) | 3.2 | 3.5 | 17.2 | 20.8 | 54.5 | 181 |
| Epoch (static) | 3.1 | 3.4 | 17.0 | 20.5 | 54.1 | 180 |
| Patching time | 0.1 | 0.1 | 0.2 | 0.3 | 0.4 | 1 |
| Overhead % | 3% | 3% | 1% | 1% | 1% | 1% |
| Threshold | ETTh1 | ETTh2 | ETTm1 | ETTm2 |
|---|---|---|---|---|
| 0.55 | 215.68 / 2.37 [1pt] [1–9] | 220.61 / 2.32 [1pt] [1–7] | 224.01 / 2.29 [1pt] [1–7] | 221.53 / 2.31 [1pt] [1–7] |
| 0.60 | 191.67 / 2.67 [1pt] [1–10] | 195.40 / 2.62 [1pt] [1–9] | 198.32 / 2.58 [1pt] [1–9] | 195.95 / 2.61 [1pt] [1–9] |
| 0.65 | 164.67 / 3.11 [1pt] [1–12] | 169.12 / 3.03 [1pt] [1–11] | 172.46 / 2.97 [1pt] [1–12] | 170.65 / 3.00 [1pt] [1–11] |
| 0.70 | 140.28 / 3.65 [1pt] [1–14] | 144.52 / 3.54 [1pt] [1–14] | 147.55 / 3.47 [1pt] [1–16] | 145.77 / 3.51 [1pt] [1–14] |
| 0.75 | 115.67 / 4.43 [1pt] [1–17] | 119.25 / 4.29 [1pt] [1–18] | 121.64 / 4.21 [1pt] [1–22] | 120.42 / 4.25 [1pt] [1–19] |
| 0.80 | 92.52 / 5.53 [1pt] [1–23] | 94.64 / 5.41 [1pt] [1–23] | 97.57 / 5.25 [1pt] [1–30] | 96.68 / 5.30 [1pt] [1–27] |
| Threshold | Solar | M4-Monthly | Weather |
|---|---|---|---|
| 0.55 | 224.56 / 2.28 [1pt] [1–3] | 220.44 / 2.32 [1pt] [1–6] | 226.58 / 2.26 [1pt] [1–5] |
| 0.60 | 203.32 / 2.52 [1pt] [1–2] | 197.83 / 2.59 [1pt] [1–10] | 201.61 / 2.54 [1pt] [1–6] |
| 0.65 | 178.42 / 2.87 [1pt] [1–4] | 171.53 / 2.98 [1pt] [1–16] | 175.55 / 2.92 [1pt] [1–8] |
| 0.70 | 153.54 / 3.33 [1pt] [1–6] | 147.35 / 3.47 [1pt] [1–22] | 151.11 / 3.39 [1pt] [1–11] |
| 0.75 | 126.00 / 4.06 [1pt] [1–9] | 122.10 / 4.19 [1pt] [1–25] | 125.30 / 4.09 [1pt] [1–17] |
| 0.80 | 98.97 / 5.17 [1pt] [1–18] | 97.84 / 5.23 [1pt] [1–27] | 100.18 / 5.11 [1pt] [1–26] |
| SNR | Boundary Jaccard | # Boundaries |
|---|---|---|
| Clean | 1.000 | 22.35 0.00 |
| 40 dB | 0.626 | 22.61 1.52 |
| 20 dB | 0.528 | 22.47 1.67 |
| 10 dB | 0.419 | 22.27 1.71 |
| 5 dB | 0.314 | 21.82 2.10 |
| 2 dB | 0.281 | 21.94 1.89 |