Irregularly-spaced sequential data, such as transaction logs, encode multi-scale temporal structure that is critical for prediction but difficult to recover explicitly with standard sequence models, which often trade interpretability for performance. We introduce MoTIF, an interpretable framework using mixture-of-experts (MoE) attention to decompose irregular time series into temporal patterns spanning minute-to-year scales. Each expert is trained via temporal alignment that induces attention specialization to a designated cyclical or within-range pattern, with label-free routing supervision that learns to activate the correct expert from patterns present in the data. On synthetic data with known temporal structure, MoTIF recovers correct routing well above chance (0.389/0.829 for range/cyclical patterns vs. 0.091 random) and outperforms all baselines, including a standard transformer and an unspecialized MoE, on next-value prediction. On a public credit card transaction dataset, MoTIF achieves higher macro-F1 than all baselines for next-amount prediction. Its learned routing reveals structure a black-box model cannot: transactions are governed primarily by intraday and short-interval rhythms, while fraudulent transactions show bursty, calendar-agnostic timing versus the steadier periodicity of legitimate ones. On downstream fraud detection, MoTIF matches standard transformer and MoE baselines and clearly outperforms recent sequential recommendation models used as embedding baselines. These results show that decomposing temporal structure into interpretable, expert-specific patterns need not sacrifice predictive accuracy, offering a practical route to sequence models that are both accurate and auditable in high-stakes settings. Code is available at https://anonymous.4open.science/r/MoTIF-82E4.
Figures & tables
Figure 1: Method Schematic. MoTIF consists of (1) a feature embedding layer, (2) a transformer backbone with temporal pattern-aware MoE-based attention query and label-free routing supervision, and (3) task heads. Continuous and categorical features are embedded via an MLP and learned lookup tables, respectively, and concatenated into a token embedding. These embeddings pass through a transformer whose self-attention query is computed by an MoE layer, where specialized experts induce attention patterns aligned with cyclical or elapsed-range temporal structure from minute to year scales. Label-free routing supervision trains the model to route sequences to experts matching their constituent patterns, based on next-value prediction performance and detected autocorrelations. Final representations feed into task heads for next-value prediction, and their generalizability is evaluated via DCN and GBT probes for downstream fraud detection.
Metric
MoTIF (ours)
FAME (no specialization)
alignment diag CE (own kernel, ↓ )
4.009 ± 0.010
12.046 ± 0.470
alignment off-diag CE
6.859 ± 0.106
12.072 ± 0.457
alignment gap (off-diag minus diag ↑ )
2.850 ± 0.116
0.026 ± 0.014
routing recovery - range mean
0.389 ± 0.004
0.092 ± 0.022
routing recovery - cyclical mean
0.829 ± 0.050
0.089 ± 0.032
Table 1: Temporal attention alignment and routing recovery on synthetic data. Top half : Expert specialization - alignment diag/off-diag CE refers to the matrix of cross-entropy (CE) between target and actual attention distributions for all pairs of experts, where lower CE indicates more agreement. Bottom half : Routing recovery - how often the correct expert is assigned the top routing weight.
Method
MAE
RMSE
F1 macro
Accuracy
Within 1
DuoRec
0.3612 ± 0.0014
0.7813 ± 0.0023
0.5552 ± 0.0190
0.7313 ± 0.0008
0.9327 ± 0.0004
SASRec
0.3574 ± 0.0025
0.7784 ± 0.0036
0.5969 ± 0.0035
0.7332 ± 0.0015
0.9329 ± 0.0007
HSTU
0.3627 ± 0.0003
0.7835 ± 0.0010
0.5795 ± 0.0056
0.7305 ± 0.0002
0.9322 ± 0.0002
PATHER
0.3576 ± 0.0003
0.7690 ± 0.0009
0.6421 ± 0.0023
0.7345 ± 0.0001
0.9355 ± 0.0002
Transformer
0.3609 ± 0.0010
0.7818 ± 0.0020
0.6108 ± 0.0082
0.7320 ± 0.0006
0.9325 ± 0.0005
FAME
0.3600 ± 0.0007
0.7811 ± 0.0013
0.6349 ± 0.0156
0.7330 ± 0.0004
0.9323 ± 0.0002
Table 2: Next value prediction performance comparison on synthetic data. Continuous metrics (MAE, RMSE) on value bucket index and classification metrics on the next value bucket (F1 macro, accuracy, within 1). Bold indicates top metric.
Figure 2: Routing-based interpretation of temporal patterns in CCT. (A) Expert prevalence: proportion of top-1 routing per expert, at the transaction level (light) and sequence-dominant level (dark). (B) Validation: correlation between routing weight and raw-data statistics, independent of the router; solid bar reflects next-amount predictability (the model’s actual routing signal), hatched bars are controls (next category, arrival timing). (C) Routing by fraud cohort: mean routing weight at the labeled transaction, non-fraud (light blue) vs. fraud (red). (D) Fraud predictability from routing alone: GBT probe performance using transaction-level (dark) vs. sequence-level (light) routing weights; dotted line indicates chance.
Next Amount Prediction
Method
MAE ↓
RMSE ↓
F1 macro ↑
Accuracy ↑
Within 1 ↑
DuoRec
0.6863 ± 0.0019
1.3316 ± 0.0031
0.7122 ± 0.0036
0.6208 ± 0.0006
0.8384 ± 0.0007
SASRec
0.9419 ± 0.0022
1.6180 ± 0.0011
0.5565 ± 0.0024
0.5179 ± 0.0015
0.7679 ± 0.0007
HSTU
0.8460 ± 0.0088
1.5073 ± 0.0104
0.6089 ± 0.0051
0.5490 ± 0.0036
0.7972 ± 0.0023
PANTHER
0.7126 ± 0.0023
1.3195 ± 0.0031
0.6270 ± 0.0026
0.5849 ± 0.0014
0.8377 ± 0.0009
Transformer
0.8177 ± 0.0038
1.4840 ± 0.0052
0.6638 ± 0.0004
0.5679 ± 0.0015
0.8015 ± 0.0008
Table 3: Next amount prediction on CCT dataset. Both continuous metrics (MAE, RMSE) on amount bucket index and classification metrics on the next amount bucket (F1 macro, accuracy, within 1). Bold indicates top metric, underline indicates ours is second best.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Feature
Non-Fraud
Fraud
Age
30.57 ± 14.44
52.64 ± 15.59
Gender (female/male)
329/328
687/656
Per Capita Income - By Zipcode
23,084.63\pm10,804.45
23,169.96\pm11,573.71
Yearly Income - By Person
46,611.70\pm22,003.43
45,277.64\pm23,456.91
Total Debt
73,498.22\pm50,976.28
58,921.12\pm52,222.65
FICO Score
702.90 ± 69.33
713.08 ± 65.94
Appendix
Table 4: Demographic features by fraud label. Reported values are mean ± standard deviation for age, income, debt, and FICO score, separated by fraud label. For gender, we report female/male counts per fraud label. Individuals are labeled as fraudsters if they have at least one fraudulent transaction.
Figure 3: Attention specialization cross-CE and routing confusion matrices on synthetic data. Comparing MoTIF (ours, left ) against FAME (unspecialized attention experts, right ) (A) Layer-1 expert attention versus target temporal kernels cross-entropy, seed-averaged; gold indicates own kernel (diagonal); lower CE indicates closer match. (B) Layer-1 routing confusion matrix: which expert wins each generating pattern, row-normalized top-expert fractions, seed-averaged; gold indicates the generating pattern’s own expert (diagonal); crimson indicates the most-routed expert.
Metric
MoTIF (ours)
FAME (no specialization)
alignment diag CE (own kernel, ↓ )
5.105 ± 0.009
8.756 ± 0.004
alignment off-diag CE
6.857 ± 0.012
8.798 ± 0.091
alignment gap (off-diag minus diag ↑ )
1.752 ± 0.007
0.041 ± .095
# experts whose own kernel is best (/11)
5.330 ± 0.470
1.670 ± 0.470
mean own-kernel rank (0=perfect, 5=random)
0.910 ± 0.070
4.730 ± 0.790
token-dispatch entropy ratio ↑
0.954 ± 0.003
0.991 ± 0.005
Appendix
Table 5: Temporal attention alignment and routing diversity on CCT dataset. Top half : Expert specialization. Alignment diag/off-diag CE refers to the matrix of cross-entropy (CE) between target and actual attention distributions for all pairs of experts - lower CE indicates more agreement. Bottom half : Routing diversity versus collapse at both token (token-dispatch) and sequence (across-seq) levels. Higher entropy ratio indicates more uniform routing over experts. Across-seq top-expert share measures how often the primary routed expert is selected.
Figure 4: Individual sequence routing patterns. Heatmaps indicating routing weight for each expert across the transaction sequence for fraud (left) and non-fraud (right). Gold dashed line marks the labeled transaction (anchor); bar plots show routing weights at the anchor, with black dots indicating cohort mean. Range-mass is the sum of routing weights over the six range experts.
Next Item Category Prediction
Method
Accuracy ↑
F1 weighted ↑
DuoRec
0.4703 ± 0.0008
0.4521 ± 0.0010
SASRec
0.3728 ± 0.0007
0.3402 ± 0.0011
HSTU
0.3905 ± 0.0009
0.3575 ± 0.0020
PANTHER
0.4038 ± 0.0017
0.3699 ± 0.0022
Transformer
0.4355 ± 0.0010
0.4127 ± 0.0012
Appendix
Table 6: Next item category prediction on CCT dataset. Non-ordinal classification only. Bold indicates top metric, underline indicates ours is second best.
Method
AUROC
AUPRC
Precision@ τ∗
Recall@ τ∗
F1@ τ∗
DuoRec
0.9554 ± .0017
0.8853 ± .0064
0.7711 ± .0150
0.8792 ± .0224
0.8212 ± .0042
SASRec
0.9547 ± .0020
0.8826 ± .0078
0.7662 ± .0020
0.8854 ± .0077
0.8215 ± .0044
HSTU
0.9538 ± .0027
0.8776 ± .0128
0.7727 ± .0115
0.8778 ± .0164
0.8217 ± .0022
PANTHER
0.9541 ± .0014
0.8847 ± .0052
0.7651 ± .0145
0.8839 ± .0160
0.8199 ± .0031
Transformer
0.9685 ± .0015
0.9176 ± .0042
0.8106 ± .0225
0.8953 ± .0081
0.8506 ± .0092
FAME
0.9706 ± .0027
0.9271 ± .0080
0.8166 ± .0013
0.8918 ± .0067
0.8525 ± .0036
Appendix
Table 7: CCT fraud prediction, full DCN head. Transaction-level fraud prediction from sequence embeddings and handcrafted features. τ∗ is validation-tuned for precision, recall, and F1. Bold: best; underline: second-best.
Method
AUROC
AUPRC
Precision@ τ∗
Recall@ τ∗
F1@ τ∗
DuoRec
0.7866 ± .0008
0.5702 ± .0025
0.4773 ± .0031
0.7839 ± .0116
0.5933 ± .0010
SASRec
0.7866 ± .0020
0.5716 ± .0043
0.4844 ± .0061
0.7833 ± .0200
0.5984 ± .0016
HSTU
0.7845 ± .0008
0.5667 ± .0012
0.4808 ± .0090
0.7874 ± .0304
0.5967 ± .0052
PANTHER
0.7855 ± .0032
0.5704 ± .0046
0.4789 ± .0101
0.7942 ± .0141
0.5973 ± .0050
Transformer
0.8309 ± .0036
0.6934 ± .0064
0.5530 ± .0136
0.7491 ± .0056
0.6362 ± .0108
FAME
0.8334 ± .0050
0.7047 ± .0053
0.5570 ± .0202
0.7275 ± .0254
0.6301 ± .0048
Appendix
Table 8: Embedding-only GBT fraud prediction probe. Predicting transaction-level fraud from sequence embedding only to assess representation quality. τ∗ is a threshold optimally tuned on the validation set (nested-CV) for precision, recall, and F1.Bold indicates top metric, underline indicates ours is second best.
Time series forecasting models are increasingly scaled through large Transformer backbones, yet most existing approaches process all series through a shared dense computation path despite substantial heterogeneity in temporal structure. Mixture-of-Experts (MoE) offers a natural alternative by enabling conditional computation, but standard MoE routing leaves expert specialization weakly identified and often unstable during downstream adaptation. We propose AME-TS, a structure-guided sparse time series foundation model that aligns expert routing with interpretable temporal structure. AME-TS first uses a lightweight regime predictor to estimate series-level descriptors, including forecastability, seasonality, trend, and sparsity, and maps them to a soft structural prior over experts. This series-level prior guides token-level routing during training, encouraging structure-aligned specialization. On the GIFT-Eval benchmark, AME-TS delivers a strong accuracy-efficiency tradeoff across model scales: it substantially outperforms existing time series foundation models at small model scales and remains competitive with the strongest models at larger scales, while activating substantially fewer parameters through sparse routing. We further show that AME-TS learns more interpretable routing geometry and substantially more stable expert specialization than standard MoE during fine-tuning on the M5 dataset. These results suggest that structure-aware routing is an effective and reliable way to realize the benefits of sparse expert models for time series forecasting.
In non-stationary multivariate time series, different variables and samples often exhibit heterogeneous latent dynamic states, while existing deep forecasting models usually compress them into a unified end-to-end mapping, leading to suboptimal modeling of time-varying dynamics and limited interpretability regarding which forecasting mechanism is activated under different latent states. To overcome these limitations, we reformulate time series forecasting as a unified framework of latent temporal state identification and interpretable expert routing, and propose Fuzzy-MoE, a fuzzy logic-based dynamic Mixture-of-Experts model. Fuzzy-MoE consists of multiple parallel expert mapping networks and a dual-view fuzzy router. By jointly exploiting local convolutional dynamics and global segmented statistics, the router infers latent temporal states and computes expert activation strengths through learnable Gaussian membership functions, enabling explicit IF-THEN rule-based expert selection. This fine-grained routing strategy allows different variables within the same sequence to activate different experts, effectively capturing heterogeneous temporal dynamics while improving model interpretability. Experimental results on multiple public time series benchmark datasets show that Fuzzy-MoE significantly outperforms mainstream forecasting methods in forecasting accuracy. Moreover, fuzzy memberships and rule activations provide interpretable routing diagnostics, demonstrating the effectiveness of the proposed framework in both forecasting performance and mechanism transparency. Unlike traditional MoE models that use black-box routing, Fuzzy-MoE`s routing is based on clear, interpretable fuzzy rules. This makes the expert selection transparent and traceable.
Lan Guo, Jie Xiao, Zhao Su +5
School of Information Science and Engineering, Lanzhou University · School of Computing and Information Technology, University of Wollongong · Department of Data Science and AI, Monash University
Multivariate time series classification (MTSC) is pivotal in high-stakes domains, such as clinical diagnosis and industrial fault detection, where safe deployment necessitates transparent decision-making. However, isolating the temporal segments that drive model predictions is challenging because discriminative signals in real-world time series are typically sparse, heterogeneous, and heavily obscured by background noise. This paper, therefore, proposes AnchorMoE, an interpretable-by-construction classification framework. Built upon a Mixture-of-Experts (MoE) architecture, AnchorMoE encodes multi-view representations of local patches and routes them to specialized experts, ensuring that the final prediction is formulated as an exact additive decomposition over the input segments, facilitating ante-hoc transparency rather than relying on post-hoc estimations. To maintain the reliability of this decomposition under sparse signal distributions, we introduce a geometric orthogonality constraint that penalizes representational redundancy, compelling distinct experts to specialize in heterogeneous predictive patterns. Furthermore, an uncertainty-aware reliability gate is designed to dynamically calibrate the contribution of each segment, effectively suppressing residual background noise. Extensive experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance while faithfully grounding its decisions in the raw time series.
Tao Xie, Zexi Tan, Haoyi Xiao +5
Guangdong University of Technology · Guangzhou, China · Shenzhen University +7