Decomposing Irregular Time Series into Interpretable Temporal Patterns with Mixture-of-Experts

Jenna L. Ballard; Zijie Pan; Shivani Shekhar; Yiming Bian; Yun-Shiuan Chuang; Avinash Thangali; Uma Kona; Linsey Pang

cs.LGOct 30, 2026

Decomposing Irregular Time Series into Interpretable Temporal Patterns with Mixture-of-Experts

Authors: Jenna L. Ballard, Zijie Pan, Shivani Shekhar, Yiming Bian, Yun-Shiuan Chuang, Avinash Thangali, Uma Kona, Linsey Pang,

Organizations: University of Pennsylvania · PayPal AI

Abstract

Irregularly-spaced sequential data, such as transaction logs, encode multi-scale temporal structure that is critical for prediction but difficult to recover explicitly with standard sequence models, which often trade interpretability for performance. We introduce MoTIF, an interpretable framework using mixture-of-experts (MoE) attention to decompose irregular time series into temporal patterns spanning minute-to-year scales. Each expert is trained via temporal alignment that induces attention specialization to a designated cyclical or within-range pattern, with label-free routing supervision that learns to activate the correct expert from patterns present in the data. On synthetic data with known temporal structure, MoTIF recovers correct routing well above chance (0.389/0.829 for range/cyclical patterns vs. 0.091 random) and outperforms all baselines, including a standard transformer and an unspecialized MoE, on next-value prediction. On a public credit card transaction dataset, MoTIF achieves higher macro-F1 than all baselines for next-amount prediction. Its learned routing reveals structure a black-box model cannot: transactions are governed primarily by intraday and short-interval rhythms, while fraudulent transactions show bursty, calendar-agnostic timing versus the steadier periodicity of legitimate ones. On downstream fraud detection, MoTIF matches standard transformer and MoE baselines and clearly outperforms recent sequential recommendation models used as embedding baselines. These results show that decomposing temporal structure into interpretable, expert-specific patterns need not sacrifice predictive accuracy, offering a practical route to sequence models that are both accurate and auditable in high-stakes settings. Code is available at https://anonymous.4open.science/r/MoTIF-82E4.

Figures & tables

Figure 1: Method Schematic. MoTIF consists of (1) a feature embedding layer, (2) a transformer backbone with temporal pattern-aware MoE-based attention query and label-free routing supervision, and (3) task heads. Continuous and categorical features are embedded via an MLP and learned lookup tables, respectively, and concatenated into a token embedding. These embeddings pass through a transformer whose self-attention query is computed by an MoE layer, where specialized experts induce attention patterns aligned with cyclical or elapsed-range temporal structure from minute to year scales. Label-free routing supervision trains the model to route sequences to experts matching their constituent patterns, based on next-value prediction performance and detected autocorrelations. Final representations feed into task heads for next-value prediction, and their generalizability is evaluated via DCN and GBT probes for downstream fraud detection.

Metric	MoTIF (ours)	FAME (no specialization)
alignment diag CE (own kernel, $\downarrow$ )	4.009 ± 0.010	12.046 ± 0.470
alignment off-diag CE	6.859 ± 0.106	12.072 ± 0.457
alignment gap (off-diag minus diag $\uparrow$ )	2.850 ± 0.116	0.026 ± 0.014
routing recovery - range mean	0.389 ± 0.004	0.092 ± 0.022
routing recovery - cyclical mean	0.829 ± 0.050	0.089 ± 0.032

Table 1: Temporal attention alignment and routing recovery on synthetic data. Top half : Expert specialization - alignment diag/off-diag CE refers to the matrix of cross-entropy (CE) between target and actual attention distributions for all pairs of experts, where lower CE indicates more agreement. Bottom half : Routing recovery - how often the correct expert is assigned the top routing weight.

Method	MAE	RMSE	F1 macro	Accuracy	Within 1
DuoRec	0.3612 $\pm$ 0.0014	0.7813 $\pm$ 0.0023	0.5552 $\pm$ 0.0190	0.7313 $\pm$ 0.0008	0.9327 $\pm$ 0.0004
SASRec	0.3574 $\pm$ 0.0025	0.7784 $\pm$ 0.0036	0.5969 $\pm$ 0.0035	0.7332 $\pm$ 0.0015	0.9329 $\pm$ 0.0007
HSTU	0.3627 $\pm$ 0.0003	0.7835 $\pm$ 0.0010	0.5795 $\pm$ 0.0056	0.7305 $\pm$ 0.0002	0.9322 $\pm$ 0.0002
PATHER	0.3576 $\pm$ 0.0003	0.7690 $\pm$ 0.0009	0.6421 $\pm$ 0.0023	0.7345 $\pm$ 0.0001	0.9355 $\pm$ 0.0002
Transformer	0.3609 $\pm$ 0.0010	0.7818 $\pm$ 0.0020	0.6108 $\pm$ 0.0082	0.7320 $\pm$ 0.0006	0.9325 $\pm$ 0.0005
FAME	0.3600 $\pm$ 0.0007	0.7811 $\pm$ 0.0013	0.6349 $\pm$ 0.0156	0.7330 $\pm$ 0.0004	0.9323 $\pm$ 0.0002

Table 2: Next value prediction performance comparison on synthetic data. Continuous metrics (MAE, RMSE) on value bucket index and classification metrics on the next value bucket (F1 macro, accuracy, within 1). Bold indicates top metric.

Figure 2: Routing-based interpretation of temporal patterns in CCT. (A) Expert prevalence: proportion of top-1 routing per expert, at the transaction level (light) and sequence-dominant level (dark). (B) Validation: correlation between routing weight and raw-data statistics, independent of the router; solid bar reflects next-amount predictability (the model’s actual routing signal), hatched bars are controls (next category, arrival timing). (C) Routing by fraud cohort: mean routing weight at the labeled transaction, non-fraud (light blue) vs. fraud (red). (D) Fraud predictability from routing alone: GBT probe performance using transaction-level (dark) vs. sequence-level (light) routing weights; dotted line indicates chance.

Next Amount Prediction
Method	MAE $\downarrow$	RMSE $\downarrow$	F1 macro $\uparrow$	Accuracy $\uparrow$	Within 1 $\uparrow$
DuoRec	0.6863 $\pm$ 0.0019	1.3316 $\pm$ 0.0031	0.7122 $\pm$ 0.0036	0.6208 $\pm$ 0.0006	0.8384 $\pm$ 0.0007
SASRec	0.9419 $\pm$ 0.0022	1.6180 $\pm$ 0.0011	0.5565 $\pm$ 0.0024	0.5179 $\pm$ 0.0015	0.7679 $\pm$ 0.0007
HSTU	0.8460 $\pm$ 0.0088	1.5073 $\pm$ 0.0104	0.6089 $\pm$ 0.0051	0.5490 $\pm$ 0.0036	0.7972 $\pm$ 0.0023
PANTHER	0.7126 $\pm$ 0.0023	1.3195 $\pm$ 0.0031	0.6270 $\pm$ 0.0026	0.5849 $\pm$ 0.0014	0.8377 $\pm$ 0.0009
Transformer	0.8177 $\pm$ 0.0038	1.4840 $\pm$ 0.0052	0.6638 $\pm$ 0.0004	0.5679 $\pm$ 0.0015	0.8015 $\pm$ 0.0008

Table 3: Next amount prediction on CCT dataset. Both continuous metrics (MAE, RMSE) on amount bucket index and classification metrics on the next amount bucket (F1 macro, accuracy, within 1). Bold indicates top metric, underline indicates ours is second best.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Feature	Non-Fraud	Fraud
Age	30.57 $\pm$ 14.44	52.64 $\pm$ 15.59
Gender (female/male)	329/328	687/656
Per Capita Income - By Zipcode	$23,084.63$ \pm10,804.45	$23,169.96$ \pm11,573.71
Yearly Income - By Person	$46,611.70$ \pm22,003.43	$45,277.64$ \pm23,456.91
Total Debt	$73,498.22$ \pm50,976.28	$58,921.12$ \pm52,222.65
FICO Score	702.90 $\pm$ 69.33	713.08 $\pm$ 65.94

Appendix

Table 4: Demographic features by fraud label. Reported values are mean

\pm

standard deviation for age, income, debt, and FICO score, separated by fraud label. For gender, we report female/male counts per fraud label. Individuals are labeled as fraudsters if they have at least one fraudulent transaction.

Figure 3: Attention specialization cross-CE and routing confusion matrices on synthetic data. Comparing MoTIF (ours, left ) against FAME (unspecialized attention experts, right ) (A) Layer-1 expert attention versus target temporal kernels cross-entropy, seed-averaged; gold indicates own kernel (diagonal); lower CE indicates closer match. (B) Layer-1 routing confusion matrix: which expert wins each generating pattern, row-normalized top-expert fractions, seed-averaged; gold indicates the generating pattern’s own expert (diagonal); crimson indicates the most-routed expert.

Metric	MoTIF (ours)	FAME (no specialization)
alignment diag CE (own kernel, $\downarrow$ )	5.105 ± 0.009	8.756 ± 0.004
alignment off-diag CE	6.857 ± 0.012	8.798 ± 0.091
alignment gap (off-diag minus diag $\uparrow$ )	1.752 ± 0.007	0.041 ± .095
# experts whose own kernel is best (/11)	5.330 ± 0.470	1.670 ± 0.470
mean own-kernel rank (0=perfect, 5=random)	0.910 ± 0.070	4.730 ± 0.790
token-dispatch entropy ratio $\uparrow$	0.954 ± 0.003	0.991 ± 0.005

Appendix

Table 5: Temporal attention alignment and routing diversity on CCT dataset. Top half : Expert specialization. Alignment diag/off-diag CE refers to the matrix of cross-entropy (CE) between target and actual attention distributions for all pairs of experts - lower CE indicates more agreement. Bottom half : Routing diversity versus collapse at both token (token-dispatch) and sequence (across-seq) levels. Higher entropy ratio indicates more uniform routing over experts. Across-seq top-expert share measures how often the primary routed expert is selected.

Figure 4: Individual sequence routing patterns. Heatmaps indicating routing weight for each expert across the transaction sequence for fraud (left) and non-fraud (right). Gold dashed line marks the labeled transaction (anchor); bar plots show routing weights at the anchor, with black dots indicating cohort mean. Range-mass is the sum of routing weights over the six range experts.

Next Item Category Prediction
Method	Accuracy $\uparrow$	F1 weighted $\uparrow$
DuoRec	0.4703 $\pm$ 0.0008	0.4521 $\pm$ 0.0010
SASRec	0.3728 $\pm$ 0.0007	0.3402 $\pm$ 0.0011
HSTU	0.3905 $\pm$ 0.0009	0.3575 $\pm$ 0.0020
PANTHER	0.4038 $\pm$ 0.0017	0.3699 $\pm$ 0.0022
Transformer	0.4355 $\pm$ 0.0010	0.4127 $\pm$ 0.0012

Appendix

Table 6: Next item category prediction on CCT dataset. Non-ordinal classification only. Bold indicates top metric, underline indicates ours is second best.

Method	AUROC	AUPRC	Precision@ $\bm{\tau}^{*}$	Recall@ $\bm{\tau}^{*}$	F1@ $\bm{\tau}^{*}$
DuoRec	0.9554 ± .0017	0.8853 ± .0064	0.7711 ± .0150	0.8792 ± .0224	0.8212 ± .0042
SASRec	0.9547 ± .0020	0.8826 ± .0078	0.7662 ± .0020	0.8854 ± .0077	0.8215 ± .0044
HSTU	0.9538 ± .0027	0.8776 ± .0128	0.7727 ± .0115	0.8778 ± .0164	0.8217 ± .0022
PANTHER	0.9541 ± .0014	0.8847 ± .0052	0.7651 ± .0145	0.8839 ± .0160	0.8199 ± .0031
Transformer	0.9685 ± .0015	0.9176 ± .0042	0.8106 ± .0225	0.8953 ± .0081	0.8506 ± .0092
FAME	0.9706 ± .0027	0.9271 ± .0080	0.8166 ± .0013	0.8918 ± .0067	0.8525 ± .0036

Appendix

Table 7: CCT fraud prediction, full DCN head. Transaction-level fraud prediction from sequence embeddings and handcrafted features.

\tau^{*}

is validation-tuned for precision, recall, and F1. Bold: best; underline: second-best.

Method	AUROC	AUPRC	Precision@ $\bm{\tau}^{*}$	Recall@ $\bm{\tau}^{*}$	F1@ $\bm{\tau}^{*}$
DuoRec	0.7866 ± .0008	0.5702 ± .0025	0.4773 ± .0031	0.7839 ± .0116	0.5933 ± .0010
SASRec	0.7866 ± .0020	0.5716 ± .0043	0.4844 ± .0061	0.7833 ± .0200	0.5984 ± .0016
HSTU	0.7845 ± .0008	0.5667 ± .0012	0.4808 ± .0090	0.7874 ± .0304	0.5967 ± .0052
PANTHER	0.7855 ± .0032	0.5704 ± .0046	0.4789 ± .0101	0.7942 ± .0141	0.5973 ± .0050
Transformer	0.8309 ± .0036	0.6934 ± .0064	0.5530 ± .0136	0.7491 ± .0056	0.6362 ± .0108
FAME	0.8334 ± .0050	0.7047 ± .0053	0.5570 ± .0202	0.7275 ± .0254	0.6301 ± .0048

Appendix

Table 8: Embedding-only GBT fraud prediction probe. Predicting transaction-level fraud from sequence embedding only to assess representation quality.

\tau^{*}

is a threshold optimally tuned on the validation set (nested-CV) for precision, recall, and F1.Bold indicates top metric, underline indicates ours is second best.

Irregular Time-Series Modeling

Irregular time-series modeling learns from observations that arrive at uneven times, across variables, or with missing values.

See all 106 papers

AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
May 24, 2026Rui Wang, Renhao Xue, Ray Razi +2Sparse Mixture-of-Experts Time Series Forecasting
Amazon Web Services
Fuzzy-MoE: Interpretable Regime-Conditioned Expert Routing for Non-Stationary Multivariate Time Series Forecasting
Aug 21, 2026Lan Guo, Jie Xiao, Zhao Su +5Multivariate Time Series Forecasting Non-Stationary Time Series Forecasting
School of Information Science and Engineering, Lanzhou University · School of Computing and Information Technology, University of Wollongong · Department of Data Science and AI, Monash University
AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE
Jun 2, 2026Tao Xie, Zexi Tan, Haoyi Xiao +5Explainable Time Series Forecasting Multivariate Time Series Classification
Guangdong University of Technology · Guangzhou, China · Shenzhen University +7

May 24, 2026J/K move · Enter open · S save

AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting

Rui Wang, Renhao Xue, Ray Razi, Huan Song+1

Amazon Web Services

Sparse Mixture-of-Experts Time Series Forecasting Expert Routing Time-Series Foundation Models Time Series Modeling

Open paper AskPDF arXiv

Time series forecasting models are increasingly scaled through large Transformer backbones, yet most existing approaches process all series through a shared dense computation path despite substantial heterogeneity in temporal structure. Mixture-of-Experts (MoE) offers a natural alternative by enabling conditional computation, but standard MoE routing leaves expert specialization weakly identified and often unstable during downstream adaptation. We propose AME-TS, a structure-guided sparse time series foundation model that aligns expert routing with interpretable temporal structure. AME-TS first uses a lightweight regime predictor to estimate series-level descriptors, including forecastability, seasonality, trend, and sparsity, and maps them to a soft structural prior over experts. This series-level prior guides token-level routing during training, encouraging structure-aligned specialization. On the GIFT-Eval benchmark, AME-TS delivers a strong accuracy-efficiency tradeoff across model scales: it substantially outperforms existing time series foundation models at small model scales and remains competitive with the strongest models at larger scales, while activating substantially fewer parameters through sparse routing. We further show that AME-TS learns more interpretable routing geometry and substantially more stable expert specialization than standard MoE during fine-tuning on the M5 dataset. These results suggest that structure-aware routing is an effective and reliable way to realize the benefits of sparse expert models for time series forecasting.

Abstract

Irregular Time-Series Modeling

Explore similar work

AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting

Fuzzy-MoE: Interpretable Regime-Conditioned Expert Routing for Non-Stationary Multivariate Time Series Forecasting

AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE